From lyuben at stoev.eu Thu Dec 2 12:05:52 2021 From: lyuben at stoev.eu (Lyuben Stoev) Date: Thu, 2 Dec 2021 14:05:52 +0200 Subject: nginx KTLS and HTTP/2 performance degradation Message-ID: <2d37d96e-ff1b-7113-434f-8d04751bb85e@stoev.eu> Hello, ??? I have tested the nginx with the patch https://hg.nginx.org/nginx/rev/65946a191197 (SSL: SSL_sendfile() support with kernel TLS.) following the nginx blog article https://www.nginx.com/blog/improving-nginx-performance-with-kernel-tls/ And it sort of works, but I have bad performance when making HTTP/2 requests. If I made a HTTP/1.1 request there is 30-35% increase in performance as the Nginx blog article stated, but when I changed the request to use HTTP/2 the request was 40% slower than an ordinary nginx without KTLS enabled. Does anyone have such perfomance degradation with nginx KTLS and HTTP/2? I am using generic setup - Ubuntu 20.04.3 LTS and kernels 5.8.0-63-generic (the same results are with 5.4.0-91-generic). The nginx vritual host is the same as in the Nginx blog article with exception of adding http2 to the listen! OpenSSL 3.0.0 and nginx 1.21.4 are used. The KTLS seems to work, because the strace and debug logs show it. Just the sstrange thing is when using HTTP2, the sendfile syscalls look: ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 ??? sendfile(39, 131, [1418218] => [1426410], 8192) = 8192 ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 ??? sendfile(39, 131, [1426410] => [1434602], 8192) = 8192 ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 ??? sendfile(39, 131, [1434602] => [1442794], 8192) = 8192 ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 ??? sendfile(39, 131, [1442794] => [1450986], 8192) = 8192 ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 ??? sendfile(39, 131, [1450986] => [1459178], 8192) = 8192 ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 ??? sendfile(39, 131, [1459178] => [1467370], 8192) = 8192 It is always 8K and there are thousands of sendfile syscalls.... And when the HTTP/1.1 is used the sendfile syscall changes the window: ??? sendfile(32, 33, [586170368] => [586694656], 487571456) = 524288 ??? sendfile(32, 33, [586694656], 487047168) = -1 EAGAIN (Resource temporarily unavailable) ??? epoll_wait(8, [{EPOLLOUT, {u32=1602391248, u64=94426458393808}}], 512, 59711) = 1 ??? sendfile(32, 33, [586694656] => [594673664], 487047168) = 7979008 ??? sendfile(32, 33, [594673664] => [595001344], 479068160) = 327680 ??? sendfile(32, 33, [595001344], 478740480) = -1 EAGAIN (Resource temporarily unavailable) ??? epoll_wait(8, [{EPOLLOUT, {u32=1602391248, u64=94426458393808}}], 512, 60000) = 1 ??? sendfile(32, 33, [595001344] => [604028928], 478740480) = 9027584 ??? sendfile(32, 33, [604028928] => [604356608], 469712896) = 327680 ??? sendfile(32, 33, [604356608] => [604749824], 469385216) = 393216 ??? sendfile(32, 33, [604749824] => [605192192], 468992000) = 442368 Does anyone receive similar results? root at srv-tests:~# curl --http1.1 --tls13-ciphers TLS_AES_256_GCM_SHA384 https://mytests.local/1G --output /dev/null ? % Total??? % Received % Xferd? Average Speed?? Time??? Time Time? Current ???????????????????????????????? Dload? Upload?? Total?? Spent Left? Speed 100 1024M? 100 1024M??? 0???? 0? 1170M????? 0 --:--:-- --:--:-- --:--:-- 1168M root at srv-tests:~# curl --http1.1 --tls13-ciphers TLS_AES_256_GCM_SHA384 https://mytests.local/1G --output /dev/null ? % Total??? % Received % Xferd? Average Speed?? Time??? Time Time? Current ???????????????????????????????? Dload? Upload?? Total?? Spent Left? Speed 100 1024M? 100 1024M??? 0???? 0? 1197M????? 0 --:--:-- --:--:-- --:--:-- 1196M root at srv-tests:~# curl --http2 --tls13-ciphers TLS_AES_256_GCM_SHA384 https://mytests.local/1G --output /dev/null ? % Total??? % Received % Xferd? Average Speed?? Time??? Time Time? Current ???????????????????????????????? Dload? Upload?? Total?? Spent Left? Speed 100 1024M? 100 1024M??? 0???? 0?? 340M????? 0? 0:00:03? 0:00:03 --:--:--? 340M root at srv-tests:~# curl --http2 --tls13-ciphers TLS_AES_256_GCM_SHA384 https://mytests.local/1G --output /dev/null ? % Total??? % Received % Xferd? Average Speed?? Time??? Time Time? Current ???????????????????????????????? Dload? Upload?? Total?? Spent Left? Speed 100 1024M? 100 1024M??? 0???? 0?? 338M????? 0? 0:00:03? 0:00:03 --:--:--? 338M srv-prod ~ # h2load -n 1 -c 1 -t 1 https://mytests.local/1G starting benchmark... spawning thread #0: 1 total client(s). 1 total requests TLS Protocol: TLSv1.3 Cipher: TLS_AES_256_GCM_SHA384 Server Temp Key: X25519 253 bits Application protocol: h2 progress: 100% done finished in 7.64s, 0.13 req/s, 134.11MB/s requests: 1 total, 1 started, 1 done, 1 succeeded, 0 failed, 0 errored, 0 timeout status codes: 1 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 1.00GB (1074922031) total, 492B (492) headers (space savings 21.66%), 1.00GB (1073741824) data ???????????????????? min???????? max???????? mean sd??????? +/- sd time for request:????? 7.63s?????? 7.63s?????? 7.63s???????? 0us 100.00% time for connect:??? 13.46ms???? 13.46ms???? 13.46ms???????? 0us 100.00% time to 1st byte:??? 15.05ms???? 15.05ms???? 15.05ms???????? 0us 100.00% req/s?????????? :?????? 0.13??????? 0.13??????? 0.13??????? 0.00 100.00% srv-prod ~ # h2load --h1 -n 1 -c 1 -t 1 https://mytests.local/1G starting benchmark... spawning thread #0: 1 total client(s). 1 total requests TLS Protocol: TLSv1.3 Cipher: TLS_AES_256_GCM_SHA384 Server Temp Key: X25519 253 bits Application protocol: http/1.1 progress: 100% done finished in 2.65s, 0.38 req/s, 386.41MB/s requests: 1 total, 1 started, 1 done, 1 succeeded, 0 failed, 0 errored, 0 timeout status codes: 1 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 1.00GB (1073742546) total, 631B (631) headers (space savings 0.00%), 1.00GB (1073741824) data ???????????????????? min???????? max???????? mean sd??????? +/- sd time for request:????? 2.64s?????? 2.64s?????? 2.64s???????? 0us 100.00% time for connect:??? 13.72ms???? 13.72ms???? 13.72ms???????? 0us 100.00% time to 1st byte:??? 14.86ms???? 14.86ms???? 14.86ms???????? 0us 100.00% req/s?????????? :?????? 0.38??????? 0.38??????? 0.38??????? 0.00 100.00% With regards, Lyuben Stoev From mdounin at mdounin.ru Thu Dec 2 13:07:29 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 2 Dec 2021 16:07:29 +0300 Subject: nginx KTLS and HTTP/2 performance degradation In-Reply-To: <2d37d96e-ff1b-7113-434f-8d04751bb85e@stoev.eu> References: <2d37d96e-ff1b-7113-434f-8d04751bb85e@stoev.eu> Message-ID: Hello! On Thu, Dec 02, 2021 at 02:05:52PM +0200, Lyuben Stoev wrote: > Hello, > ??? I have tested the nginx with the patch > https://hg.nginx.org/nginx/rev/65946a191197 (SSL: SSL_sendfile() support > with kernel TLS.) following the nginx blog article > https://www.nginx.com/blog/improving-nginx-performance-with-kernel-tls/ > And it sort of works, but I have bad performance when making HTTP/2 > requests. If I made a HTTP/1.1 request there is 30-35% increase in > performance as the Nginx blog article stated, but when I changed the > request to use HTTP/2 the request was 40% slower than an ordinary nginx > without KTLS enabled. Does anyone have such perfomance degradation with > nginx KTLS and HTTP/2? I am using generic setup - Ubuntu 20.04.3 LTS and > kernels 5.8.0-63-generic (the same results are with 5.4.0-91-generic). > The nginx vritual host is the same as in the Nginx blog article with > exception of adding http2 to the listen! OpenSSL 3.0.0 and nginx 1.21.4 > are used. > The KTLS seems to work, because the strace and debug logs show it. Just > the sstrange thing is when using HTTP2, the sendfile syscalls look: > ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 > ??? sendfile(39, 131, [1418218] => [1426410], 8192) = 8192 > ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 > ??? sendfile(39, 131, [1426410] => [1434602], 8192) = 8192 > ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 > ??? sendfile(39, 131, [1434602] => [1442794], 8192) = 8192 > ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 > ??? sendfile(39, 131, [1442794] => [1450986], 8192) = 8192 > ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 > ??? sendfile(39, 131, [1450986] => [1459178], 8192) = 8192 > ??? write(39, "\0 \0\0\0\0\0\0\1", 9)?????? = 9 > ??? sendfile(39, 131, [1459178] => [1467370], 8192) = 8192 > > It is always 8K and there are thousands of sendfile syscalls.... That's expected, because of HTTP/2 framing. Unfortunately, HTTP/2 isn't designed to work with sendfile(), and sending large files over HTTP/2 require a lot of sendfile() syscalls. In general, for HTTP/2 it is better to keep sendfile() disabled. -- Maxim Dounin http://mdounin.ru/ From vadimjunk at gmail.com Fri Dec 3 00:31:50 2021 From: vadimjunk at gmail.com (Vadim Fedorenko) Date: Fri, 3 Dec 2021 00:31:50 +0000 Subject: nginx KTLS and HTTP/2 performance degradation In-Reply-To: References: <2d37d96e-ff1b-7113-434f-8d04751bb85e@stoev.eu> Message-ID: Hello! I would say that current implementation of Kernel TLS in OpenSSL will give huge overhead because of additional syscall for every frame and it's header, it doesn't matter if it's sendfile or not. Without sendfile it's actually 5% worse in my tests. That's why it's better to disable Kernel TLS for HTTP/2 requests in Nginx + OpenSSL. The only solution for this would be implementation of sendmsg()/sendmmsg() in OpenSSL and support for such implementation in Nginx together with mmap() for files. This solution would have the same performance as sendfile() from kernel perspective. ??, 2 ???. 2021 ?. ? 13:08, Maxim Dounin : > Hello! > > On Thu, Dec 02, 2021 at 02:05:52PM +0200, Lyuben Stoev wrote: > > > Hello, > > I have tested the nginx with the patch > > https://hg.nginx.org/nginx/rev/65946a191197 (SSL: SSL_sendfile() > support > > with kernel TLS.) following the nginx blog article > > https://www.nginx.com/blog/improving-nginx-performance-with-kernel-tls/ > > And it sort of works, but I have bad performance when making HTTP/2 > > requests. If I made a HTTP/1.1 request there is 30-35% increase in > > performance as the Nginx blog article stated, but when I changed the > > request to use HTTP/2 the request was 40% slower than an ordinary nginx > > without KTLS enabled. Does anyone have such perfomance degradation with > > nginx KTLS and HTTP/2? I am using generic setup - Ubuntu 20.04.3 LTS and > > kernels 5.8.0-63-generic (the same results are with 5.4.0-91-generic). > > The nginx vritual host is the same as in the Nginx blog article with > > exception of adding http2 to the listen! OpenSSL 3.0.0 and nginx 1.21.4 > > are used. > > The KTLS seems to work, because the strace and debug logs show it. Just > > the sstrange thing is when using HTTP2, the sendfile syscalls look: > > write(39, "\0 \0\0\0\0\0\0\1", 9) = 9 > > sendfile(39, 131, [1418218] => [1426410], 8192) = 8192 > > write(39, "\0 \0\0\0\0\0\0\1", 9) = 9 > > sendfile(39, 131, [1426410] => [1434602], 8192) = 8192 > > write(39, "\0 \0\0\0\0\0\0\1", 9) = 9 > > sendfile(39, 131, [1434602] => [1442794], 8192) = 8192 > > write(39, "\0 \0\0\0\0\0\0\1", 9) = 9 > > sendfile(39, 131, [1442794] => [1450986], 8192) = 8192 > > write(39, "\0 \0\0\0\0\0\0\1", 9) = 9 > > sendfile(39, 131, [1450986] => [1459178], 8192) = 8192 > > write(39, "\0 \0\0\0\0\0\0\1", 9) = 9 > > sendfile(39, 131, [1459178] => [1467370], 8192) = 8192 > > > > It is always 8K and there are thousands of sendfile syscalls.... > > That's expected, because of HTTP/2 framing. Unfortunately, HTTP/2 > isn't designed to work with sendfile(), and sending large files > over HTTP/2 require a lot of sendfile() syscalls. In general, for > HTTP/2 it is better to keep sendfile() disabled. > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Fri Dec 3 04:10:22 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 3 Dec 2021 07:10:22 +0300 Subject: nginx KTLS and HTTP/2 performance degradation In-Reply-To: References: <2d37d96e-ff1b-7113-434f-8d04751bb85e@stoev.eu> Message-ID: Hello! On Fri, Dec 03, 2021 at 12:31:50AM +0000, Vadim Fedorenko wrote: > I would say that current implementation of Kernel TLS in OpenSSL will give > huge overhead because of additional syscall for every frame and it's header, > it doesn't matter if it's sendfile or not. Without sendfile it's actually > 5% worse > in my tests. That's why it's better to disable Kernel TLS for HTTP/2 > requests > in Nginx + OpenSSL. Without SSL_sendfile(), kernel TLS might make sense if TLS offloading is supported by a NIC, freeing some CPU power. With SSL_sendfile(), it is beneficial even without any specialized hardware. Just not with HTTP/2. > The only solution for this would be implementation of sendmsg()/sendmmsg() > in OpenSSL and support for such implementation in Nginx together with mmap() > for files. This solution would have the same performance as sendfile() from > kernel perspective. On FreeBSD sendfile() is much more than mmap() and send[m]msg(). Also I tend to think that mmap() is a very risky mechanism for sending files, and shouldn't be used by a general-purpose server such as nginx, as it kills the server on disk errors. Rather, solution for HTTP/2 would be to implement in-kernel HTTP/2 framing along with in-kernel TLS. Or a more general sendfile() implementation, such as sendfilev() on Solaris. Not sure it worth the effort though. -- Maxim Dounin http://mdounin.ru/ From xeioex at nginx.com Fri Dec 3 13:57:16 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 03 Dec 2021 13:57:16 +0000 Subject: [njs] Added memory exception if alloc fails for njs_vm_function_alloc. Message-ID: details: https://hg.nginx.org/njs/rev/b27244641dbf branches: changeset: 1756:b27244641dbf user: Dmitry Volyntsev date: Fri Dec 03 13:55:11 2021 +0000 description: Added memory exception if alloc fails for njs_vm_function_alloc. diffstat: src/njs_function.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 748eca39acb3 -r b27244641dbf src/njs_function.c --- a/src/njs_function.c Tue Nov 30 14:55:57 2021 +0000 +++ b/src/njs_function.c Fri Dec 03 13:55:11 2021 +0000 @@ -72,6 +72,7 @@ njs_vm_function_alloc(njs_vm_t *vm, njs_ function = njs_mp_zalloc(vm->mem_pool, sizeof(njs_function_t)); if (njs_slow_path(function == NULL)) { + njs_memory_error(vm); return NULL; } From xeioex at nginx.com Fri Dec 3 13:57:19 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 03 Dec 2021 13:57:19 +0000 Subject: [njs] Tests: added promise support for unit tests subrequest method. Message-ID: details: https://hg.nginx.org/njs/rev/cd87a113829d branches: changeset: 1757:cd87a113829d user: Dmitry Volyntsev date: Fri Dec 03 13:55:12 2021 +0000 description: Tests: added promise support for unit tests subrequest method. diffstat: src/test/njs_externals_test.c | 46 +++++++++++++++++++++++++++++++++--------- src/test/njs_externals_test.h | 1 + src/test/njs_unit_test.c | 17 ++++++++++++--- 3 files changed, 50 insertions(+), 14 deletions(-) diffs (134 lines): diff -r b27244641dbf -r cd87a113829d src/test/njs_externals_test.c --- a/src/test/njs_externals_test.c Fri Dec 03 13:55:11 2021 +0000 +++ b/src/test/njs_externals_test.c Fri Dec 03 13:55:12 2021 +0000 @@ -378,9 +378,27 @@ njs_unit_test_r_method(njs_vm_t *vm, njs static njs_int_t +njs_unit_test_promise_trampoline(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t unused) +{ + njs_function_t *callback; + + callback = njs_value_function(njs_argument(args, 1)); + + if (callback != NULL) { + return njs_vm_call(vm, callback, njs_argument(args, 2), 1); + } + + return NJS_OK; +} + + +static njs_int_t njs_unit_test_r_subrequest(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { + njs_int_t ret; + njs_value_t retval, *argument, *select; njs_vm_event_t vm_event; njs_function_t *callback; njs_external_ev_t *ev; @@ -393,9 +411,19 @@ njs_unit_test_r_subrequest(njs_vm_t *vm, return NJS_ERROR; } - callback = njs_value_function(njs_arg(args, nargs, 1)); + ev = njs_mp_alloc(vm->mem_pool, sizeof(njs_external_ev_t)); + if (ev == NULL) { + njs_memory_error(vm); + return NJS_ERROR; + } + + ret = njs_vm_promise_create(vm, &retval, &ev->callbacks[0]); + if (ret != NJS_OK) { + return NJS_ERROR; + } + + callback = njs_vm_function_alloc(vm, njs_unit_test_promise_trampoline); if (callback == NULL) { - njs_type_error(vm, "argument is not callable"); return NJS_ERROR; } @@ -405,22 +433,20 @@ njs_unit_test_r_subrequest(njs_vm_t *vm, return NJS_ERROR; } - ev = njs_mp_alloc(vm->mem_pool, sizeof(njs_external_ev_t)); - if (ev == NULL) { - njs_memory_error(vm); - return NJS_ERROR; - } + argument = njs_arg(args, nargs, 1); + select = njs_arg(args, nargs, 2); ev->vm_event = vm_event; ev->data = r; - ev->nargs = 1; - njs_value_assign(&ev->args[0], njs_argument(args, 0)); + ev->nargs = 2; + njs_value_assign(&ev->args[0], &ev->callbacks[!!njs_bool(select)]); + njs_value_assign(&ev->args[1], argument); env = vm->external; njs_queue_insert_tail(&env->events, &ev->link); - njs_set_undefined(&vm->retval); + njs_vm_retval_set(vm, njs_value_arg(&retval)); return NJS_OK; } diff -r b27244641dbf -r cd87a113829d src/test/njs_externals_test.h --- a/src/test/njs_externals_test.h Fri Dec 03 13:55:11 2021 +0000 +++ b/src/test/njs_externals_test.h Fri Dec 03 13:55:12 2021 +0000 @@ -19,6 +19,7 @@ typedef struct { void *data; njs_uint_t nargs; njs_value_t args[3]; + njs_value_t callbacks[2]; njs_queue_link_t link; } njs_external_ev_t; diff -r b27244641dbf -r cd87a113829d src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Fri Dec 03 13:55:11 2021 +0000 +++ b/src/test/njs_unit_test.c Fri Dec 03 13:55:12 2021 +0000 @@ -21015,8 +21015,15 @@ static njs_unit_test_t njs_externals_te { njs_str("let obj = { a: 1, b: 2};" "function cb(r) { r.retval(obj.a); }" - "$r.subrequest(reply => cb(reply))"), - njs_str("1") }, + "$r.subrequest($r)" + ".then(reply => cb(reply))"), + njs_str("1") }, + + { njs_str("let obj = { a: 1, b: 2};" + "function cb(r, select) { r.retval(obj[select]); }" + "$r.subrequest('b')" + ".then(select => cb($r, select))"), + njs_str("2") }, }; @@ -21025,7 +21032,8 @@ static njs_unit_test_t njs_async_handle { njs_str("globalThis.main = (function() {" " function cb(r) { r.retval(1); }" " function handler(r) {" - " r.subrequest(reply => cb(reply));" + " r.subrequest(r)" + " .then(reply => cb(reply))" " };" " return {handler};" "})();" @@ -21036,7 +21044,8 @@ static njs_unit_test_t njs_async_handle " let obj = { a: 1, b: 2};" " function cb(r) { r.retval(obj.a); }" " function handler(r) {" - " r.subrequest(reply => cb(reply));" + " r.subrequest(r)" + " .then(reply => cb(reply))" " };" " return {handler};" "})();" From xeioex at nginx.com Fri Dec 3 13:57:20 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 03 Dec 2021 13:57:20 +0000 Subject: [njs] Fixed execution of async function in synchronous context. Message-ID: details: https://hg.nginx.org/njs/rev/ed1756875eb5 branches: changeset: 1758:ed1756875eb5 user: Dmitry Volyntsev date: Fri Dec 03 13:55:22 2021 +0000 description: Fixed execution of async function in synchronous context. The bug was introduced in 92d10cd761e2 (0.7.0). diffstat: src/njs_async.c | 6 ++++++ src/test/njs_unit_test.c | 16 ++++++++++++++++ test/js/async_promise.t.js | 20 ++++++++++++++++++++ 3 files changed, 42 insertions(+), 0 deletions(-) diffs (73 lines): diff -r cd87a113829d -r ed1756875eb5 src/njs_async.c --- a/src/njs_async.c Fri Dec 03 13:55:12 2021 +0000 +++ b/src/njs_async.c Fri Dec 03 13:55:22 2021 +0000 @@ -37,6 +37,9 @@ njs_async_function_frame_invoke(njs_vm_t ret = njs_function_call(vm, njs_function(&capability->resolve), &njs_value_undefined, retval, 1, &vm->retval); + } else if (ret == NJS_AGAIN) { + ret = NJS_OK; + } else if (ret == NJS_ERROR) { if (njs_is_memory_error(vm, &vm->retval)) { return NJS_ERROR; @@ -116,6 +119,9 @@ njs_await_fulfilled(njs_vm_t *vm, njs_va njs_async_context_free(vm, ctx); + } else if (ret == NJS_AGAIN) { + ret = NJS_OK; + } else if (ret == NJS_ERROR) { if (njs_is_memory_error(vm, &vm->retval)) { return NJS_ERROR; diff -r cd87a113829d -r ed1756875eb5 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Fri Dec 03 13:55:12 2021 +0000 +++ b/src/test/njs_unit_test.c Fri Dec 03 13:55:22 2021 +0000 @@ -21024,6 +21024,22 @@ static njs_unit_test_t njs_externals_te "$r.subrequest('b')" ".then(select => cb($r, select))"), njs_str("2") }, + + { njs_str("function pr(x) { return new Promise(resolve => {resolve(x + ':pr')}); };" + "Promise.all(['a', 'b', 'c'].map(async (v) => {" + " return await pr(v + ':async');" + "}))" + ".then(v => $r.retval(v))"), + njs_str("a:async:pr,b:async:pr,c:async:pr") }, + + { njs_str("function pr(x) { return new Promise(resolve => {resolve(x + ':pr')}); };" + "Promise.all(['a', 'b', 'c'].map(async (v) => {" + " let r = await pr(v + ':async');" + " let r2 = await pr(r + ':async2');" + " return r2 + ':r';" + "}))" + ".then(v => $r.retval(v))"), + njs_str("a:async:pr:async2:pr:r,b:async:pr:async2:pr:r,c:async:pr:async2:pr:r") }, }; diff -r cd87a113829d -r ed1756875eb5 test/js/async_promise.t.js --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test/js/async_promise.t.js Fri Dec 03 13:55:22 2021 +0000 @@ -0,0 +1,20 @@ +/*--- +includes: [compareArray.js] +flags: [async] +---*/ + +let stages = []; + +function pr(x) { return new Promise(resolve => {resolve(x)}); } + +pr(10) +.then(async (v) => { + stages.push("then before"); + let y = await pr(22); + stages.push(`then ${v} ${y}`); + return v + y; +}) +.then(v => stages.push(`then2 ${v}`)) +.catch(e => $DONOTEVALUATE()) +.then(v => assert.compareArray(stages, ['then before', 'then 10 22', 'then2 32'])) +.then($DONE, $DONE); From pluknet at nginx.com Mon Dec 6 12:37:28 2021 From: pluknet at nginx.com (Sergey Kandaurov) Date: Mon, 6 Dec 2021 15:37:28 +0300 Subject: [PATCH 1 of 3] QUIC: post stream events instead of calling their handlers In-Reply-To: <5b03ffd757804542daec.1637850049@arut-laptop> References: <5b03ffd757804542daec.1637850049@arut-laptop> Message-ID: <4A4C02F0-D125-474C-B4E7-E3A6F1BD0144@nginx.com> > On 25 Nov 2021, at 17:20, Roman Arutyunyan wrote: > > # HG changeset patch > # User Roman Arutyunyan > # Date 1637692791 -10800 > # Tue Nov 23 21:39:51 2021 +0300 > # Branch quic > # Node ID 5b03ffd757804542daec73188a509b02e6b2c596 > # Parent d041b8d6ab0b2dea150536531345fa47c696b303 > QUIC: post stream events instead of calling their handlers. > > This potentially reduces the number of handler calls. > > diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c > --- a/src/event/quic/ngx_event_quic_streams.c > +++ b/src/event/quic/ngx_event_quic_streams.c > @@ -1122,7 +1122,7 @@ ngx_quic_handle_stream_frame(ngx_connect > rev->ready = 1; > > if (rev->active) { > - rev->handler(rev); > + ngx_post_event(rev, &ngx_posted_events); > } > } > > @@ -1369,7 +1369,7 @@ ngx_quic_handle_reset_stream_frame(ngx_c > } > > if (rev->active) { > - rev->handler(rev); > + ngx_post_event(rev, &ngx_posted_events); > } > > return NGX_OK; > @@ -1438,7 +1438,7 @@ ngx_quic_handle_stop_sending_frame(ngx_c > wev = qs->connection->write; > > if (wev->active) { > - wev->handler(wev); > + ngx_post_event(wev, &ngx_posted_events); > } > > return NGX_OK; I don't like this change. While in practice this indeed tends to reduce the number of handler invocations in certain edge cases, it also entails negative effects. First, postponing processing the next part of a stream means that if the stream spans multiple packets, its handling will be deferred inadvertently until after the next packet is finished processing. Next, if a stream is the encoder stream, used to send new dynamic entries, then new request streams will be inevitably blocked. I've run intentionally on this to see what happens: looks like the blocked stream isn't woken up until connection close (which consumes active streams in ngx_quic_close_streams()). 0x6 is the encoder stream, 0x40 is the request stream. 2021/12/06 14:45:36 [debug] 17379#0: *1 quic frame rx app STREAM id:0x6 off:1 len:428 2021/12/06 14:45:36 [debug] 17379#0: *4 post event 000062F000000740 2021/12/06 14:45:36 [debug] 17379#0: *1 quic frame rx app STREAM id:0x40 len:38 fin:1 ... 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 check insert count req:23, have:0 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 block stream ... 2021/12/06 14:45:36 [debug] 17379#0: posted event 000062F000000740 2021/12/06 14:45:36 [debug] 17379#0: *4 delete posted event 000062F000000740 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 read handler 2021/12/06 14:45:36 [debug] 17379#0: *4 quic stream id:0x6 recv eof:0 buf:128 2021/12/06 14:45:36 [debug] 17379#0: *4 quic stream id:0x6 recv len:128 2021/12/06 14:45:36 [debug] 17379#0: *4 quic flow update 129 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse encoder instruction 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse prefix int 16384 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 set capacity 16384 2021/12/06 14:45:36 [debug] 17379#0: *1 malloc: 000062100005DD00:4096 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse encoder instruction 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse field inr 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse prefix int 0 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse prefix int 6 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse literal huff:1, len:6 2021/12/06 14:45:36 [debug] 17379#0: *4 malloc: 00006020000A5630:10 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse literal done "localhost" 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse field inr done static[0] "localhost" 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 ref insert static[0] "localhost" 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 static[0] lookup ":authority":"" 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 insert [0] ":authority":"localhost", size:51 2021/12/06 14:45:36 [debug] 17379#0: *1 malloc: 0000606000008D80:51 2021/12/06 14:45:36 [debug] 17379#0: *1 post event 0000614000005A78 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 new dynamic entry, blocked:1 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 unblock stream 2021/12/06 14:45:36 [debug] 17379#0: *1 post event 000062F000000608 ... 2021/12/06 14:45:36 [debug] 17379#0: posted event 0000614000005A78 2021/12/06 14:45:36 [debug] 17379#0: *1 delete posted event 0000614000005A78 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 inc insert count handler 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 send insert count increment 23 ... 2021/12/06 14:45:36 [debug] 17379#0: posted event 000062F000000608 2021/12/06 14:45:36 [debug] 17379#0: *1 delete posted event 000062F000000608 2021/12/06 14:45:36 [debug] 17379#0: *1 quic input handler [nothing] 2021/12/06 14:45:36 [debug] 17379#0: posted event 000061D00003C648 ... ... 2021/12/06 14:45:37 [debug] 17379#0: *1 quic frame rx app CONNECTION_CLOSE err:0 ft:0 2021/12/06 14:45:37 [debug] 17379#0: *1 quic immediate close drain:1 2021/12/06 14:45:37 [debug] 17379#0: *22 post event 000062F000000A18 ... 2021/12/06 14:45:37 [debug] 17379#0: posted event 000062F000000A18 2021/12/06 14:45:37 [debug] 17379#0: *22 delete posted event 000062F000000A18 2021/12/06 14:45:37 [debug] 17379#0: *22 http3 process request 2021/12/06 14:45:37 [debug] 17379#0: *1 http3 check insert count req:23, have:23 ... 2021/12/06 14:45:37 [debug] 17379#0: *22 http3 parse field representation done 2021/12/06 14:45:37 [debug] 17379#0: *22 http3 parse headers done 2021/12/06 14:45:37 [debug] 17379#0: *1 http3 send section acknowledgement 64 2021/12/06 14:45:37 [debug] 17379#0: *1 quic creating server uni stream streams:2 max:100 id:0xb 2021/12/06 14:45:37 [debug] 17379#0: *1 quic stream id:0xb create ... 2021/12/06 14:45:37 [debug] 17379#0: *22 http3 header: "user-agent: something something" 2021/12/06 14:45:37 [debug] 17379#0: *22 http request count:1 blk:0 2021/12/06 14:45:37 [debug] 17379#0: *22 http close request 2021/12/06 14:45:37 [debug] 17379#0: *22 http log handler 2021/12/06 14:45:37 [debug] 17379#0: *22 malloc: 000060D000007AF0:130 2021/12/06 14:45:37 [debug] 17379#0: *22 malloc: 000060F000019EA0:169 2021/12/06 14:45:37 [debug] 17379#0: *22 run cleanup: 000060300007D500 2021/12/06 14:45:37 [debug] 17379#0: *1 event timer add: 8: 75000:2579974395 2021/12/06 14:45:37 [debug] 17379#0: *22 close http connection: 8 2021/12/06 14:45:37 [debug] 17379#0: *1 http3 send stream cancellation 64 -- Sergey Kandaurov From maciej at xeredo.it Tue Dec 7 05:54:19 2021 From: maciej at xeredo.it (=?iso-8859-1?q?Maciej_Kr=FCger?=) Date: Tue, 07 Dec 2021 06:54:19 +0100 Subject: [PATCH] http: add autoindex_name_length Message-ID: <8c9f2abb0e4ee6302203.1638856459@localhost> # HG changeset patch # User Maciej Kr?ger # Date 1638856063 -3600 # Tue Dec 07 06:47:43 2021 +0100 # Node ID 8c9f2abb0e4ee6302203e1fa3e63ad218d6251cb # Parent a7a77549265ef46f1f0fdb3897f4beabf9e09c40 http: add autoindex_name_length This adds an option to change the name length from the hardcoded 50 characters to whatever the user desires In my case specifically it is the long filenames openwrt produces that I need displayed properly, but others might have similar usecases where having this option makes sense Can be used like so autoindex on; autoindex_name_length 100; diff -r a7a77549265e -r 8c9f2abb0e4e src/http/modules/ngx_http_autoindex_module.c --- a/src/http/modules/ngx_http_autoindex_module.c Thu Nov 25 22:02:10 2021 +0300 +++ b/src/http/modules/ngx_http_autoindex_module.c Tue Dec 07 06:47:43 2021 +0100 @@ -42,6 +42,7 @@ ngx_uint_t format; ngx_flag_t localtime; ngx_flag_t exact_size; + ngx_uint_t name_length; } ngx_http_autoindex_loc_conf_t; @@ -50,10 +51,6 @@ #define NGX_HTTP_AUTOINDEX_JSONP 2 #define NGX_HTTP_AUTOINDEX_XML 3 -#define NGX_HTTP_AUTOINDEX_PREALLOCATE 50 - -#define NGX_HTTP_AUTOINDEX_NAME_LEN 50 - static ngx_buf_t *ngx_http_autoindex_html(ngx_http_request_t *r, ngx_array_t *entries); @@ -114,6 +111,13 @@ offsetof(ngx_http_autoindex_loc_conf_t, exact_size), NULL }, + { ngx_string("autoindex_name_length"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_num_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_autoindex_loc_conf_t, name_length), + NULL }, + ngx_null_command }; @@ -186,8 +190,7 @@ return rc; } - last = ngx_http_map_uri_to_path(r, &path, &root, - NGX_HTTP_AUTOINDEX_PREALLOCATE); + last = ngx_http_map_uri_to_path(r, &path, &root, alcf->name_length); if (last == NULL) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } @@ -324,7 +327,7 @@ if (path.len + 1 + len + 1 > allocated) { allocated = path.len + 1 + len + 1 - + NGX_HTTP_AUTOINDEX_PREALLOCATE; + + alcf->name_length; filename = ngx_pnalloc(pool, allocated); if (filename == NULL) { @@ -472,6 +475,8 @@ utf8 = 0; } + alcf = ngx_http_get_module_loc_conf(r, ngx_http_autoindex_module); + escape_html = ngx_escape_html(NULL, r->uri.data, r->uri.len); len = sizeof(title) - 1 @@ -505,7 +510,7 @@ + sizeof("\">") - 1 + entry[i].name.len - entry[i].utf_len + entry[i].escape_html - + NGX_HTTP_AUTOINDEX_NAME_LEN + sizeof(">") - 2 + + alcf->name_length + sizeof(">") - 2 + sizeof("") - 1 + sizeof(" 28-Sep-1970 12:00 ") - 1 + 20 /* the file size */ @@ -541,7 +546,6 @@ b->last = ngx_cpymem(b->last, "
../" CRLF,
                          sizeof("
../" CRLF) - 1);
 
-    alcf = ngx_http_get_module_loc_conf(r, ngx_http_autoindex_module);
     tp = ngx_timeofday();
 
     for (i = 0; i < entries->nelts; i++) {
@@ -568,11 +572,11 @@
         len = entry[i].utf_len;
 
         if (entry[i].name.len != len) {
-            if (len > NGX_HTTP_AUTOINDEX_NAME_LEN) {
-                char_len = NGX_HTTP_AUTOINDEX_NAME_LEN - 3 + 1;
+            if (len > alcf->name_length) {
+                char_len = alcf->name_length - 3 + 1;
 
             } else {
-                char_len = NGX_HTTP_AUTOINDEX_NAME_LEN + 1;
+                char_len = alcf->name_length + 1;
             }
 
             last = b->last;
@@ -588,8 +592,8 @@
 
         } else {
             if (entry[i].escape_html) {
-                if (len > NGX_HTTP_AUTOINDEX_NAME_LEN) {
-                    char_len = NGX_HTTP_AUTOINDEX_NAME_LEN - 3;
+                if (len > alcf->name_length) {
+                    char_len = alcf->name_length - 3;
 
                 } else {
                     char_len = len;
@@ -601,25 +605,25 @@
 
             } else {
                 b->last = ngx_cpystrn(b->last, entry[i].name.data,
-                                      NGX_HTTP_AUTOINDEX_NAME_LEN + 1);
+                                      alcf->name_length + 1);
                 last = b->last - 3;
             }
         }
 
-        if (len > NGX_HTTP_AUTOINDEX_NAME_LEN) {
+        if (len > alcf->name_length) {
             b->last = ngx_cpymem(last, "..>", sizeof("..>") - 1);
 
         } else {
-            if (entry[i].dir && NGX_HTTP_AUTOINDEX_NAME_LEN - len > 0) {
+            if (entry[i].dir && alcf->name_length - len > 0) {
                 *b->last++ = '/';
                 len++;
             }
 
             b->last = ngx_cpymem(b->last, "", sizeof("") - 1);
 
-            if (NGX_HTTP_AUTOINDEX_NAME_LEN - len > 0) {
-                ngx_memset(b->last, ' ', NGX_HTTP_AUTOINDEX_NAME_LEN - len);
-                b->last += NGX_HTTP_AUTOINDEX_NAME_LEN - len;
+            if (alcf->name_length - len > 0) {
+                ngx_memset(b->last, ' ', alcf->name_length - len);
+                b->last += alcf->name_length - len;
             }
         }
 
@@ -1048,6 +1052,7 @@
                               NGX_HTTP_AUTOINDEX_HTML);
     ngx_conf_merge_value(conf->localtime, prev->localtime, 0);
     ngx_conf_merge_value(conf->exact_size, prev->exact_size, 1);
+    ngx_conf_merge_uint_value(conf->name_length, prev->name_length, 50);
 
     return NGX_CONF_OK;
 }

From pluknet at nginx.com  Tue Dec  7 08:05:39 2021
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Tue, 7 Dec 2021 11:05:39 +0300
Subject: [PATCH 1 of 2] HTTP/3: use parent QUIC connection as argument
 when possible
In-Reply-To: 
References: 
 
Message-ID: 


> On 18 Nov 2021, at 12:52, Roman Arutyunyan  wrote:
> 
> # HG changeset patch
> # User Roman Arutyunyan 
> # Date 1637160358 -10800
> #      Wed Nov 17 17:45:58 2021 +0300
> # Branch quic
> # Node ID b844c77ff22218a4863d1d926bcaaa0b043c8af5
> # Parent  41caf541011045612975b7bb8423a18fd424df77
> HTTP/3: use parent QUIC connection as argument when possible.
> 
> Functions in ngx_http_v3.c, ngx_http_v3_streams.c and ngx_http_v3_tables.c
> now receive parent QUIC connection as the first argument instead of QUIC stream
> connection.  It makes sense since they are not related to a QUIC stream and
> operate connection-wise.
> 
> Also, ngx_quic_open_stream() now receives parent QUIC connection instead of
> QUIC stream connection for the same reason.
> 
> Also, ngx_http_v3_finalize_connection() and ngx_http_v3_shutdown_connection()
> macros are eliminated.  Instead, ngx_quic_finalize_connection() and
> ngx_quic_shutdown_connection() are called directly with the parent QUIC
> connection.
> 
> diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> --- a/src/event/quic/ngx_event_quic_streams.c
> +++ b/src/event/quic/ngx_event_quic_streams.c
> @@ -37,11 +37,10 @@ ngx_connection_t *
> ngx_quic_open_stream(ngx_connection_t *c, ngx_uint_t bidi)
> {
>     uint64_t                id;
> -    ngx_quic_stream_t      *qs, *nqs;
> +    ngx_quic_stream_t      *qs;
>     ngx_quic_connection_t  *qc;
> 
> -    qs = c->quic;
> -    qc = ngx_quic_get_connection(qs->parent);
> +    qc = ngx_quic_get_connection(c);
> 
>     if (bidi) {
>         if (qc->streams.server_streams_bidi
> @@ -87,12 +86,12 @@ ngx_quic_open_stream(ngx_connection_t *c
>         qc->streams.server_streams_uni++;
>     }
> 
> -    nqs = ngx_quic_create_stream(qs->parent, id);
> -    if (nqs == NULL) {
> +    qs = ngx_quic_create_stream(c, id);
> +    if (qs == NULL) {
>         return NULL;
>     }
> 
> -    return nqs->connection;
> +    return qs->connection;
> }
> 
> 
> diff --git a/src/http/modules/ngx_http_quic_module.h b/src/http/modules/ngx_http_quic_module.h
> --- a/src/http/modules/ngx_http_quic_module.h
> +++ b/src/http/modules/ngx_http_quic_module.h
> @@ -19,7 +19,8 @@
> 
> 
> #define ngx_http_quic_get_connection(c)                                       \
> -    ((ngx_http_connection_t *) (c)->quic->parent->data)
> +    ((ngx_http_connection_t *) ((c)->quic ? (c)->quic->parent->data           \
> +                                          : (c)->data))
> 
> 
> ngx_int_t ngx_http_quic_init(ngx_connection_t *c);
> diff --git a/src/http/v3/ngx_http_v3.c b/src/http/v3/ngx_http_v3.c
> --- a/src/http/v3/ngx_http_v3.c
> +++ b/src/http/v3/ngx_http_v3.c
> @@ -17,13 +17,11 @@ static void ngx_http_v3_cleanup_session(
> ngx_int_t
> ngx_http_v3_init_session(ngx_connection_t *c)
> {
> -    ngx_connection_t       *pc;
>     ngx_pool_cleanup_t     *cln;
>     ngx_http_connection_t  *hc;
>     ngx_http_v3_session_t  *h3c;
> 
> -    pc = c->quic->parent;
> -    hc = pc->data;
> +    hc = c->data;
> 
>     if (hc->v3_session) {
>         return NGX_OK;
> @@ -31,7 +29,7 @@ ngx_http_v3_init_session(ngx_connection_
> 
>     ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 init session");
> 
> -    h3c = ngx_pcalloc(pc->pool, sizeof(ngx_http_v3_session_t));
> +    h3c = ngx_pcalloc(c->pool, sizeof(ngx_http_v3_session_t));
>     if (h3c == NULL) {
>         goto failed;
>     }
> @@ -42,12 +40,12 @@ ngx_http_v3_init_session(ngx_connection_
>     ngx_queue_init(&h3c->blocked);
>     ngx_queue_init(&h3c->pushing);
> 
> -    h3c->keepalive.log = pc->log;
> -    h3c->keepalive.data = pc;
> +    h3c->keepalive.log = c->log;
> +    h3c->keepalive.data = c;
>     h3c->keepalive.handler = ngx_http_v3_keepalive_handler;
>     h3c->keepalive.cancelable = 1;
> 
> -    cln = ngx_pool_cleanup_add(pc->pool, 0);
> +    cln = ngx_pool_cleanup_add(c->pool, 0);
>     if (cln == NULL) {
>         goto failed;
>     }
> @@ -63,8 +61,8 @@ failed:
> 
>     ngx_log_error(NGX_LOG_ERR, c->log, 0, "failed to create http3 session");
> 
> -    ngx_http_v3_finalize_connection(c, NGX_HTTP_V3_ERR_INTERNAL_ERROR,
> -                                    "failed to create http3 session");
> +    ngx_quic_finalize_connection(c, NGX_HTTP_V3_ERR_INTERNAL_ERROR,
> +                                 "failed to create http3 session");
>     return NGX_ERROR;
> }
> 
> @@ -106,8 +104,8 @@ ngx_http_v3_check_flood(ngx_connection_t
>     if (h3c->total_bytes / 8 > h3c->payload_bytes + 1048576) {
>         ngx_log_error(NGX_LOG_INFO, c->log, 0, "http3 flood detected");
> 
> -        ngx_http_v3_finalize_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> -                                        "HTTP/3 flood detected");
> +        ngx_quic_finalize_connection(c, NGX_HTTP_V3_ERR_NO_ERROR,
> +                                     "HTTP/3 flood detected");
>         return NGX_ERROR;
>     }
> 
> diff --git a/src/http/v3/ngx_http_v3.h b/src/http/v3/ngx_http_v3.h
> --- a/src/http/v3/ngx_http_v3.h
> +++ b/src/http/v3/ngx_http_v3.h
> @@ -84,12 +84,6 @@
>     ngx_http_get_module_srv_conf(ngx_http_quic_get_connection(c)->conf_ctx,     \
>                                  module)
> 
> -#define ngx_http_v3_finalize_connection(c, code, reason)                      \
> -    ngx_quic_finalize_connection(c->quic->parent, code, reason)
> -
> -#define ngx_http_v3_shutdown_connection(c, code, reason)                      \
> -    ngx_quic_shutdown_connection(c->quic->parent, code, reason)
> -
> #define ngx_http_v3_connection(c)                                             \
>     ((c)->quic ? ngx_http_quic_get_connection(c)->addr_conf->http3 : 0)
> 
> diff --git a/src/http/v3/ngx_http_v3_filter_module.c b/src/http/v3/ngx_http_v3_filter_module.c
> --- a/src/http/v3/ngx_http_v3_filter_module.c
> +++ b/src/http/v3/ngx_http_v3_filter_module.c
> @@ -907,7 +907,7 @@ ngx_http_v3_create_push_request(ngx_http
>     ngx_log_debug1(NGX_LOG_DEBUG_HTTP, pc->log, 0,
>                    "http3 create push request id:%uL", push_id);
> 
> -    c = ngx_http_v3_create_push_stream(pc, push_id);
> +    c = ngx_http_v3_create_push_stream(pc->quic->parent, push_id);
>     if (c == NULL) {
>         return NGX_ABORT;
>     }
> diff --git a/src/http/v3/ngx_http_v3_parse.c b/src/http/v3/ngx_http_v3_parse.c
> --- a/src/http/v3/ngx_http_v3_parse.c
> +++ b/src/http/v3/ngx_http_v3_parse.c
> @@ -353,7 +353,8 @@ ngx_http_v3_parse_headers(ngx_connection
> 
>         case sw_verify:
> 
> -            rc = ngx_http_v3_check_insert_count(c, st->prefix.insert_count);
> +            rc = ngx_http_v3_check_insert_count(c->quic->parent,
> +                                                st->prefix.insert_count);

For the record, as previously reported in
http://mailman.nginx.org/pipermail/nginx-devel/2021-December/014607.html
and discussed privately, this needs to be a stream connection, as otherwise,
when inserting a new dynamic entry that unblocks a stream, an event will be
mis-posted on the main connection, instead.  Reverting this part helps.

>             if (rc != NGX_OK) {
>                 return rc;
>             }
> @@ -392,7 +393,9 @@ done:
>     ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http3 parse headers done");
> 
>     if (st->prefix.insert_count > 0) {
> -        if (ngx_http_v3_send_ack_section(c, c->quic->id) != NGX_OK) {
> +        if (ngx_http_v3_send_ack_section(c->quic->parent, c->quic->id)
> +            != NGX_OK)
> +        {

Similarly, I doubt if this kind of changes is really necessary
throughout ngx_http_v3_parse.c.  In general, c->quic->parent is
used to finalize a connection, so I'd consider pushing this down
to ngx_quic_finalize_connection().

>             return NGX_ERROR;
>         }
>     }
> @@ -466,7 +469,7 @@ ngx_http_v3_parse_field_section_prefix(n
> 
> done:
> 
> -    rc = ngx_http_v3_decode_insert_count(c, &st->insert_count);
> +    rc = ngx_http_v3_decode_insert_count(c->quic->parent, &st->insert_count);
>     if (rc != NGX_OK) {
>         return rc;
>     }
> @@ -1102,14 +1105,16 @@ ngx_http_v3_parse_lookup(ngx_connection_
>     u_char  *p;
> 
>     if (!dynamic) {
> -        if (ngx_http_v3_lookup_static(c, index, name, value) != NGX_OK) {
> +        if (ngx_http_v3_lookup_static(c->quic->parent, index, name, value)
> +            != NGX_OK)
> +        {
>             return NGX_HTTP_V3_ERR_DECOMPRESSION_FAILED;
>         }
> 
>         return NGX_OK;
>     }
> 
> -    if (ngx_http_v3_lookup(c, index, name, value) != NGX_OK) {
> +    if (ngx_http_v3_lookup(c->quic->parent, index, name, value) != NGX_OK) {
>         return NGX_HTTP_V3_ERR_DECOMPRESSION_FAILED;
>     }
> 
> @@ -1264,7 +1269,7 @@ ngx_http_v3_parse_control(ngx_connection
>                 return rc;
>             }
> 
> -            rc = ngx_http_v3_cancel_push(c, st->vlint.value);
> +            rc = ngx_http_v3_cancel_push(c->quic->parent, st->vlint.value);
>             if (rc != NGX_OK) {
>                 return rc;
>             }
> @@ -1310,7 +1315,7 @@ ngx_http_v3_parse_control(ngx_connection
>                 return rc;
>             }
> 
> -            rc = ngx_http_v3_set_max_push_id(c, st->vlint.value);
> +            rc = ngx_http_v3_set_max_push_id(c->quic->parent, st->vlint.value);
>             if (rc != NGX_OK) {
>                 return rc;
>             }
> @@ -1334,7 +1339,7 @@ ngx_http_v3_parse_control(ngx_connection
>                 return rc;
>             }
> 
> -            rc = ngx_http_v3_goaway(c, st->vlint.value);
> +            rc = ngx_http_v3_goaway(c->quic->parent, st->vlint.value);
>             if (rc != NGX_OK) {
>                 return rc;
>             }
> @@ -1398,7 +1403,9 @@ ngx_http_v3_parse_settings(ngx_connectio
>                 return rc;
>             }
> 
> -            if (ngx_http_v3_set_param(c, st->id, st->vlint.value) != NGX_OK) {
> +            if (ngx_http_v3_set_param(c->quic->parent, st->id, st->vlint.value)
> +                != NGX_OK)
> +            {
>                 return NGX_HTTP_V3_ERR_SETTINGS_ERROR;
>             }
> 
> @@ -1493,7 +1500,7 @@ ngx_http_v3_parse_encoder(ngx_connection
>                 return rc;
>             }
> 
> -            rc = ngx_http_v3_set_capacity(c, st->pint.value);
> +            rc = ngx_http_v3_set_capacity(c->quic->parent, st->pint.value);
>             if (rc != NGX_OK) {
>                 return rc;
>             }
> @@ -1508,7 +1515,7 @@ ngx_http_v3_parse_encoder(ngx_connection
>                 return rc;
>             }
> 
> -            rc = ngx_http_v3_duplicate(c, st->pint.value);
> +            rc = ngx_http_v3_duplicate(c->quic->parent, st->pint.value);
>             if (rc != NGX_OK) {
>                 return rc;
>             }
> @@ -1613,7 +1620,8 @@ done:
>                    st->dynamic ? "dynamic" : "static",
>                    st->index, &st->value);
> 
> -    rc = ngx_http_v3_ref_insert(c, st->dynamic, st->index, &st->value);
> +    rc = ngx_http_v3_ref_insert(c->quic->parent, st->dynamic, st->index,
> +                                &st->value);
>     if (rc != NGX_OK) {
>         return rc;
>     }
> @@ -1731,7 +1739,7 @@ done:
>                    "http3 parse field iln done \"%V\":\"%V\"",
>                    &st->name, &st->value);
> 
> -    rc = ngx_http_v3_insert(c, &st->name, &st->value);
> +    rc = ngx_http_v3_insert(c->quic->parent, &st->name, &st->value);
>     if (rc != NGX_OK) {
>         return rc;
>     }
> @@ -1793,7 +1801,7 @@ ngx_http_v3_parse_decoder(ngx_connection
>                 return rc;
>             }
> 
> -            rc = ngx_http_v3_ack_section(c, st->pint.value);
> +            rc = ngx_http_v3_ack_section(c->quic->parent, st->pint.value);
>             if (rc != NGX_OK) {
>                 return rc;
>             }
> @@ -1808,7 +1816,7 @@ ngx_http_v3_parse_decoder(ngx_connection
>                 return rc;
>             }
> 
> -            rc = ngx_http_v3_cancel_stream(c, st->pint.value);
> +            rc = ngx_http_v3_cancel_stream(c->quic->parent, st->pint.value);
>             if (rc != NGX_OK) {
>                 return rc;
>             }
> @@ -1823,7 +1831,7 @@ ngx_http_v3_parse_decoder(ngx_connection
>                 return rc;
>             }
> 
> -            rc = ngx_http_v3_inc_insert_count(c, st->pint.value);
> +            rc = ngx_http_v3_inc_insert_count(c->quic->parent, st->pint.value);
>             if (rc != NGX_OK) {
>                 return rc;
>             }

[..]

-- 
Sergey Kandaurov


From sunzhiyong3210 at gmail.com  Tue Dec  7 10:05:48 2021
From: sunzhiyong3210 at gmail.com (sun edward)
Date: Tue, 7 Dec 2021 18:05:48 +0800
Subject: Congestion control questions
Message-ID: 

Hi dev team,
       I have some questions about congestion control,   what's the current
congestion control algorithm in nginx quic,  is there any way or plan to
support CUBIC or BBR in nginx quic?

thanks & regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From xeioex at nginx.com  Tue Dec  7 13:15:49 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 07 Dec 2021 13:15:49 +0000
Subject: [njs] Fixed function redeclaration in CLI when interactive mode is on.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/44f6a4ad4464
branches:  
changeset: 1759:44f6a4ad4464
user:      Dmitry Volyntsev 
date:      Tue Dec 07 13:15:19 2021 +0000
description:
Fixed function redeclaration in CLI when interactive mode is on.

The issue was introduced in 0a2a0b5a74f4 (0.6.0).

diffstat:

 src/njs_generator.c      |   6 +-----
 src/njs_parser.c         |   2 +-
 src/njs_variable.h       |  11 +++++++++++
 src/test/njs_unit_test.c |   8 ++++++++
 4 files changed, 21 insertions(+), 6 deletions(-)

diffs (67 lines):

diff -r ed1756875eb5 -r 44f6a4ad4464 src/njs_generator.c
--- a/src/njs_generator.c	Fri Dec 03 13:55:22 2021 +0000
+++ b/src/njs_generator.c	Tue Dec 07 13:15:19 2021 +0000
@@ -3583,11 +3583,7 @@ njs_generate_function_declaration(njs_vm
         return njs_generator_stack_pop(vm, generator, NULL);
     }
 
-    if (njs_is_function(&var->value)) {
-        lambda = njs_function(&var->value)->u.lambda;
-    } else {
-        lambda = var->value.data.u.lambda;
-    }
+    lambda = njs_variable_lambda(var);
 
     lex_entry = njs_lexer_entry(node->u.reference.unique_id);
     if (njs_slow_path(lex_entry == NULL)) {
diff -r ed1756875eb5 -r 44f6a4ad4464 src/njs_parser.c
--- a/src/njs_parser.c	Fri Dec 03 13:55:22 2021 +0000
+++ b/src/njs_parser.c	Tue Dec 07 13:15:19 2021 +0000
@@ -6747,7 +6747,7 @@ njs_parser_function_declaration(njs_pars
         return NJS_ERROR;
     }
 
-    node->u.value.data.u.lambda = var->value.data.u.lambda;
+    node->u.value.data.u.lambda = njs_variable_lambda(var);
 
     node->left = (njs_parser_node_t *) unique_id;
 
diff -r ed1756875eb5 -r 44f6a4ad4464 src/njs_variable.h
--- a/src/njs_variable.h	Fri Dec 03 13:55:22 2021 +0000
+++ b/src/njs_variable.h	Tue Dec 07 13:15:19 2021 +0000
@@ -93,6 +93,17 @@ njs_variable_node_alloc(njs_vm_t *vm, nj
     return node;
 }
 
+njs_inline njs_function_lambda_t *
+njs_variable_lambda(njs_variable_t * var)
+{
+    if (njs_is_function(&var->value)) {
+        /* may be set by generator in njs_generate_function_declaration(). */
+        return njs_function(&var->value)->u.lambda;
+    }
+
+    return var->value.data.u.lambda;
+}
+
 
 njs_inline void
 njs_variable_node_free(njs_vm_t *vm, njs_variable_node_t *node)
diff -r ed1756875eb5 -r 44f6a4ad4464 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Fri Dec 03 13:55:22 2021 +0000
+++ b/src/test/njs_unit_test.c	Tue Dec 07 13:15:19 2021 +0000
@@ -21408,6 +21408,14 @@ static njs_unit_test_t  njs_shell_test[]
               "Number.prototype.test" ENTER),
       njs_str("test") },
 
+    { njs_str("function f(a) {return a}" ENTER
+              "function f(a) {return a}; f(2)" ENTER),
+      njs_str("2") },
+
+    { njs_str("function f() {return 1}" ENTER
+              "function f(a) {return 1}; f(2)" ENTER),
+      njs_str("1") },
+
     { njs_str("try {(new Function('function foo(){return 1}; ()=>{}breakhere'))} catch (e) {}" ENTER
               "foo()" ENTER),
       njs_str("ReferenceError: \"foo\" is not defined\n"

From bdhess at google.com  Tue Dec  7 16:45:01 2021
From: bdhess at google.com (Bradley Hess)
Date: Tue, 7 Dec 2021 11:45:01 -0500
Subject: [PATCH] Add ENGINE_init/finish directives around
 ENGINE_load_private_key.
Message-ID: 

# HG changeset patch
# User Bradley Hess 
# Date 1638894138 18000
#      Tue Dec 07 11:22:18 2021 -0500
# Node ID c3ec7674556519a9068c4e7a9f6279bbff6c3d31
# Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
Add ENGINE_init/finish directives around ENGINE_load_private_key.

ENGINE_by_id creates a "structural reference" to an engine; for
actually performing cryptography, this ought to be a "functional
reference" per those two topics in the OpenSSL docs:
https://www.openssl.org/docs/man1.1.1/man3/ENGINE_load_private_key.html

Unlike the default OpenSSL engine, the OpenSC PKCS #11 engine cannot
load a private key if it is not first initialized.

diff -r a7a77549265e -r c3ec76745565 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c Thu Nov 25 22:02:10 2021 +0300
+++ b/src/event/ngx_event_openssl.c Tue Dec 07 11:22:18 2021 -0500
@@ -734,16 +734,24 @@
             return NULL;
         }

+        if (!ENGINE_init(engine)) {
+            *err = "ENGINE_init() failed";
+            ENGINE_free(engine);
+            return NULL;
+        }
+
         *last++ = ':';

         pkey = ENGINE_load_private_key(engine, (char *) last, 0, 0);

         if (pkey == NULL) {
             *err = "ENGINE_load_private_key() failed";
+            ENGINE_finish(engine);
             ENGINE_free(engine);
             return NULL;
         }

+        ENGINE_finish(engine);
         ENGINE_free(engine);

         return pkey;

From mdounin at mdounin.ru  Tue Dec  7 19:27:59 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 7 Dec 2021 22:27:59 +0300
Subject: [PATCH] Add ENGINE_init/finish directives around
 ENGINE_load_private_key.
In-Reply-To: 
References: 
Message-ID: 

Hello!

On Tue, Dec 07, 2021 at 11:45:01AM -0500, Bradley Hess wrote:

> # HG changeset patch
> # User Bradley Hess 
> # Date 1638894138 18000
> #      Tue Dec 07 11:22:18 2021 -0500
> # Node ID c3ec7674556519a9068c4e7a9f6279bbff6c3d31
> # Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
> Add ENGINE_init/finish directives around ENGINE_load_private_key.
> 
> ENGINE_by_id creates a "structural reference" to an engine; for
> actually performing cryptography, this ought to be a "functional
> reference" per those two topics in the OpenSSL docs:
> https://www.openssl.org/docs/man1.1.1/man3/ENGINE_load_private_key.html
> 
> Unlike the default OpenSSL engine, the OpenSC PKCS #11 engine cannot
> load a private key if it is not first initialized.
> 
> diff -r a7a77549265e -r c3ec76745565 src/event/ngx_event_openssl.c
> --- a/src/event/ngx_event_openssl.c Thu Nov 25 22:02:10 2021 +0300
> +++ b/src/event/ngx_event_openssl.c Tue Dec 07 11:22:18 2021 -0500
> @@ -734,16 +734,24 @@
>              return NULL;
>          }
> 
> +        if (!ENGINE_init(engine)) {
> +            *err = "ENGINE_init() failed";
> +            ENGINE_free(engine);
> +            return NULL;
> +        }
> +
>          *last++ = ':';
> 
>          pkey = ENGINE_load_private_key(engine, (char *) last, 0, 0);
> 
>          if (pkey == NULL) {
>              *err = "ENGINE_load_private_key() failed";
> +            ENGINE_finish(engine);
>              ENGINE_free(engine);
>              return NULL;
>          }
> 
> +        ENGINE_finish(engine);
>          ENGINE_free(engine);
> 
>          return pkey;

Previously, a similar patch was rejected, because it caused an 
immediate segfault for most affected users, see here:

http://mailman.nginx.org/pipermail/nginx-devel/2018-May/011149.html

If you think that things have changed since then, you may want to 
elaborate.

-- 
Maxim Dounin
http://mdounin.ru/

From bdhess at google.com  Tue Dec  7 20:01:40 2021
From: bdhess at google.com (Bradley Hess)
Date: Tue, 7 Dec 2021 15:01:40 -0500
Subject: [PATCH] Add ENGINE_init/finish directives around
 ENGINE_load_private_key.
In-Reply-To: 
References: 
 
Message-ID: 

Hello Maxim,

Ah, well that explains why a patch like this has never been upstreamed,
even though it exists in a bunch of places on teh interwebz.  Sorry,
I didn't do enough archeology here.

I didn't realize the `init = 1` workaround existed, so thanks for the
pointer there.  However, it would be ideal if users could use OpenSSL's
dynamic engine loading, and avoid authoring an OpenSSL config file.

>From the description in the issue you linked, it looks like the patch was
removed for OpenSSL 1.0.x compatibility.  Would you accept a patch that
supplies the init/finish directives only if the OpenSSL version >= 1.1.0?

At this point many distros have OpenSSL 1.1 and a fixed PKCS #11 engine;
for example, the patch I submitted worked smoothly with OpenSSL 1.1 and the
PKCS #11 engine available on Debian 11, and without any engine config.

Thanks a bunch,
Brad

On Tue, Dec 7, 2021 at 2:28 PM Maxim Dounin  wrote:

> Hello!
>
> On Tue, Dec 07, 2021 at 11:45:01AM -0500, Bradley Hess wrote:
>
> > # HG changeset patch
> > # User Bradley Hess 
> > # Date 1638894138 18000
> > #      Tue Dec 07 11:22:18 2021 -0500
> > # Node ID c3ec7674556519a9068c4e7a9f6279bbff6c3d31
> > # Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
> > Add ENGINE_init/finish directives around ENGINE_load_private_key.
> >
> > ENGINE_by_id creates a "structural reference" to an engine; for
> > actually performing cryptography, this ought to be a "functional
> > reference" per those two topics in the OpenSSL docs:
> > https://www.openssl.org/docs/man1.1.1/man3/ENGINE_load_private_key.html
> >
> > Unlike the default OpenSSL engine, the OpenSC PKCS #11 engine cannot
> > load a private key if it is not first initialized.
> >
> > diff -r a7a77549265e -r c3ec76745565 src/event/ngx_event_openssl.c
> > --- a/src/event/ngx_event_openssl.c Thu Nov 25 22:02:10 2021 +0300
> > +++ b/src/event/ngx_event_openssl.c Tue Dec 07 11:22:18 2021 -0500
> > @@ -734,16 +734,24 @@
> >              return NULL;
> >          }
> >
> > +        if (!ENGINE_init(engine)) {
> > +            *err = "ENGINE_init() failed";
> > +            ENGINE_free(engine);
> > +            return NULL;
> > +        }
> > +
> >          *last++ = ':';
> >
> >          pkey = ENGINE_load_private_key(engine, (char *) last, 0, 0);
> >
> >          if (pkey == NULL) {
> >              *err = "ENGINE_load_private_key() failed";
> > +            ENGINE_finish(engine);
> >              ENGINE_free(engine);
> >              return NULL;
> >          }
> >
> > +        ENGINE_finish(engine);
> >          ENGINE_free(engine);
> >
> >          return pkey;
>
> Previously, a similar patch was rejected, because it caused an
> immediate segfault for most affected users, see here:
>
> http://mailman.nginx.org/pipermail/nginx-devel/2018-May/011149.html
>
> If you think that things have changed since then, you may want to
> elaborate.
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From mdounin at mdounin.ru  Tue Dec  7 21:49:37 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 8 Dec 2021 00:49:37 +0300
Subject: [PATCH] Add ENGINE_init/finish directives around
 ENGINE_load_private_key.
In-Reply-To: 
References: 
 
 
Message-ID: 

Hello!

On Tue, Dec 07, 2021 at 03:01:40PM -0500, Bradley Hess wrote:

> Hello Maxim,
> 
> Ah, well that explains why a patch like this has never been upstreamed,
> even though it exists in a bunch of places on teh interwebz.  Sorry,
> I didn't do enough archeology here.
> 
> I didn't realize the `init = 1` workaround existed, so thanks for the
> pointer there.  However, it would be ideal if users could use OpenSSL's
> dynamic engine loading, and avoid authoring an OpenSSL config file.
> 
> From the description in the issue you linked, it looks like the patch was
> removed for OpenSSL 1.0.x compatibility.  Would you accept a patch that
> supplies the init/finish directives only if the OpenSSL version >= 1.1.0?
> 
> At this point many distros have OpenSSL 1.1 and a fixed PKCS #11 engine;
> for example, the patch I submitted worked smoothly with OpenSSL 1.1 and the
> PKCS #11 engine available on Debian 11, and without any engine config.

As outlined in the message I linked, at least Ubuntu 18.04 ships 
OpenSSL 1.1.x but an old pkcs11 engine, so the patch will result 
in segfaults even if restricted to OpenSSL 1.1.x.  As far as I 
understand, that's still the case.

Note well that engines are deprecated in OpenSSL 3.0.

-- 
Maxim Dounin
http://mdounin.ru/

From xeioex at nginx.com  Wed Dec  8 13:16:49 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 08 Dec 2021 13:16:49 +0000
Subject: [njs] Tests: improved unit tests after efdc5f18195e (0.6.2).
Message-ID: 

details:   https://hg.nginx.org/njs/rev/1fb13b1d66a1
branches:  
changeset: 1760:1fb13b1d66a1
user:      Dmitry Volyntsev 
date:      Tue Dec 07 19:42:27 2021 +0000
description:
Tests: improved unit tests after efdc5f18195e (0.6.2).

In efdc5f18195e the recursion was removed from generator,
therefore tests that are not limited by system stack should always
pass.

diffstat:

 src/test/njs_unit_test.c |  2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

diffs (19 lines):

diff -r 44f6a4ad4464 -r 1fb13b1d66a1 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Tue Dec 07 13:15:19 2021 +0000
+++ b/src/test/njs_unit_test.c	Tue Dec 07 19:42:27 2021 +0000
@@ -13048,7 +13048,6 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("(function(){}).constructor === Function"),
       njs_str("true") },
 
-#if NJS_HAVE_LARGE_STACK
     { njs_str("new Function('('.repeat(2**13));"),
       njs_str("SyntaxError: Unexpected token \"}\" in runtime:1") },
 
@@ -13091,7 +13090,6 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("var a = (new Function('return [' + '1,'.repeat(2**13) + ']'))();"
               "a.push(5); [a[2**13 - 1], a[2**13]]"),
       njs_str("1,5") },
-#endif
 
     { njs_str("var f = new Function('return 1;'); f();"),
       njs_str("1") },

From xeioex at nginx.com  Wed Dec  8 13:16:51 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 08 Dec 2021 13:16:51 +0000
Subject: [njs] Generator: fixed typo introduced in efdc5f18195e (0.6.2).
Message-ID: 

details:   https://hg.nginx.org/njs/rev/f32bf14b74e1
branches:  
changeset: 1761:f32bf14b74e1
user:      Dmitry Volyntsev 
date:      Wed Dec 08 13:14:56 2021 +0000
description:
Generator: fixed typo introduced in efdc5f18195e (0.6.2).

diffstat:

 src/njs_generator.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r 1fb13b1d66a1 -r f32bf14b74e1 src/njs_generator.c
--- a/src/njs_generator.c	Tue Dec 07 19:42:27 2021 +0000
+++ b/src/njs_generator.c	Wed Dec 08 13:14:56 2021 +0000
@@ -453,7 +453,7 @@ njs_generator_after(njs_vm_t *vm, njs_ge
 {
     njs_generator_stack_entry_t  *entry;
 
-    entry = njs_mp_alloc(vm->mem_pool, sizeof(njs_parser_stack_entry_t));
+    entry = njs_mp_alloc(vm->mem_pool, sizeof(njs_generator_stack_entry_t));
     if (njs_slow_path(entry == NULL)) {
         return NJS_ERROR;
     }

From xeioex at nginx.com  Wed Dec  8 13:16:53 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 08 Dec 2021 13:16:53 +0000
Subject: [njs] Generator: introduced njs_generator_init().
Message-ID: 

details:   https://hg.nginx.org/njs/rev/58786d97f25f
branches:  
changeset: 1762:58786d97f25f
user:      Dmitry Volyntsev 
date:      Wed Dec 08 13:14:58 2021 +0000
description:
Generator: introduced njs_generator_init().

diffstat:

 src/njs_function.c  |   7 +++++--
 src/njs_generator.c |  24 +++++++++++++++++++++---
 src/njs_generator.h |   2 ++
 src/njs_vm.c        |   6 +++++-
 4 files changed, 33 insertions(+), 6 deletions(-)

diffs (93 lines):

diff -r f32bf14b74e1 -r 58786d97f25f src/njs_function.c
--- a/src/njs_function.c	Wed Dec 08 13:14:56 2021 +0000
+++ b/src/njs_function.c	Wed Dec 08 13:14:58 2021 +0000
@@ -1220,8 +1220,11 @@ njs_function_constructor(njs_vm_t *vm, n
         return ret;
     }
 
-    njs_memzero(&generator, sizeof(njs_generator_t));
-    generator.runtime = 1;
+    ret = njs_generator_init(&generator, 0, 1);
+    if (njs_slow_path(ret != NJS_OK)) {
+        njs_internal_error(vm, "njs_generator_init() failed");
+        return NJS_ERROR;
+    }
 
     code = njs_generate_scope(vm, &generator, scope, &njs_entry_anonymous);
     if (njs_slow_path(code == NULL)) {
diff -r f32bf14b74e1 -r 58786d97f25f src/njs_generator.c
--- a/src/njs_generator.c	Wed Dec 08 13:14:56 2021 +0000
+++ b/src/njs_generator.c	Wed Dec 08 13:14:58 2021 +0000
@@ -437,6 +437,21 @@ static const njs_str_t  return_label = n
 static const njs_str_t  undef_label  = { 0xffffffff, (u_char *) "" };
 
 
+njs_int_t
+njs_generator_init(njs_generator_t *generator, njs_int_t depth,
+    njs_bool_t runtime)
+{
+    njs_memzero(generator, sizeof(njs_generator_t));
+
+    njs_queue_init(&generator->stack);
+
+    generator->depth = depth;
+    generator->runtime = runtime;
+
+    return NJS_OK;
+}
+
+
 njs_inline void
 njs_generator_next(njs_generator_t *generator, njs_generator_state_func_t state,
     njs_parser_node_t *node)
@@ -3617,6 +3632,7 @@ njs_generate_function_scope(njs_vm_t *vm
     njs_function_lambda_t *lambda, njs_parser_node_t *node,
     const njs_str_t *name)
 {
+    njs_int_t          ret;
     njs_arr_t          *arr;
     njs_bool_t         module;
     njs_uint_t         depth;
@@ -3631,9 +3647,11 @@ njs_generate_function_scope(njs_vm_t *vm
         return NJS_ERROR;
     }
 
-    njs_memzero(&generator, sizeof(njs_generator_t));
-    generator.depth = depth;
-    generator.runtime = prev->runtime;
+    ret = njs_generator_init(&generator, depth, prev->runtime);
+    if (njs_slow_path(ret != NJS_OK)) {
+        njs_internal_error(vm, "njs_generator_init() failed");
+        return NJS_ERROR;
+    }
 
     node = node->right;
 
diff -r f32bf14b74e1 -r 58786d97f25f src/njs_generator.h
--- a/src/njs_generator.h	Wed Dec 08 13:14:56 2021 +0000
+++ b/src/njs_generator.h	Wed Dec 08 13:14:58 2021 +0000
@@ -40,6 +40,8 @@ struct njs_generator_s {
 };
 
 
+njs_int_t njs_generator_init(njs_generator_t *generator, njs_int_t depth,
+    njs_bool_t runtime);
 njs_vm_code_t *njs_generate_scope(njs_vm_t *vm, njs_generator_t *generator,
     njs_parser_scope_t *scope, const njs_str_t *name);
 uint32_t njs_lookup_line(njs_vm_code_t *code, uint32_t offset);
diff -r f32bf14b74e1 -r 58786d97f25f src/njs_vm.c
--- a/src/njs_vm.c	Wed Dec 08 13:14:56 2021 +0000
+++ b/src/njs_vm.c	Wed Dec 08 13:14:58 2021 +0000
@@ -170,7 +170,11 @@ njs_vm_compile(njs_vm_t *vm, u_char **st
     *start = lexer.start;
     scope = parser.scope;
 
-    njs_memzero(&generator, sizeof(njs_generator_t));
+    ret = njs_generator_init(&generator, 0, 0);
+    if (njs_slow_path(ret != NJS_OK)) {
+        njs_internal_error(vm, "njs_generator_init() failed");
+        return NJS_ERROR;
+    }
 
     code = njs_generate_scope(vm, &generator, scope, &njs_entry_main);
     if (njs_slow_path(code == NULL)) {

From xeioex at nginx.com  Wed Dec  8 13:16:55 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 08 Dec 2021 13:16:55 +0000
Subject: [njs] Generator: removing njs_generate_children_indexes_release_pop().
Message-ID: 

details:   https://hg.nginx.org/njs/rev/00b5f28f2ec7
branches:  
changeset: 1763:00b5f28f2ec7
user:      Dmitry Volyntsev 
date:      Wed Dec 08 13:14:59 2021 +0000
description:
Generator: removing njs_generate_children_indexes_release_pop().

diffstat:

 src/njs_generator.c |  33 +++++++++++++--------------------
 1 files changed, 13 insertions(+), 20 deletions(-)

diffs (71 lines):

diff -r 58786d97f25f -r 00b5f28f2ec7 src/njs_generator.c
--- a/src/njs_generator.c	Wed Dec 08 13:14:58 2021 +0000
+++ b/src/njs_generator.c	Wed Dec 08 13:14:59 2021 +0000
@@ -348,8 +348,6 @@ static njs_index_t njs_generate_temp_ind
     njs_generator_t *generator, njs_parser_node_t *node);
 static njs_int_t njs_generate_children_indexes_release(njs_vm_t *vm,
     njs_generator_t *generator, njs_parser_node_t *node);
-static njs_int_t njs_generate_children_indexes_release_pop(njs_vm_t *vm,
-    njs_generator_t *generator, njs_parser_node_t *node);
 static njs_int_t njs_generate_node_index_release(njs_vm_t *vm,
     njs_generator_t *generator, njs_parser_node_t *node);
 static njs_int_t njs_generate_node_index_release_pop(njs_vm_t *vm,
@@ -2700,6 +2698,7 @@ static njs_int_t
 njs_generate_assignment_end(njs_vm_t *vm, njs_generator_t *generator,
     njs_parser_node_t *node)
 {
+    njs_int_t              ret;
     njs_parser_node_t      *lvalue, *expr, *object, *property;
     njs_vmcode_prop_set_t  *prop_set;
 
@@ -2733,7 +2732,12 @@ njs_generate_assignment_end(njs_vm_t *vm
     node->index = expr->index;
     node->temporary = expr->temporary;
 
-    return njs_generate_children_indexes_release_pop(vm, generator, lvalue);
+    ret = njs_generate_children_indexes_release(vm, generator, lvalue);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    return njs_generator_stack_pop(vm, generator, NULL);
 }
 
 
@@ -3573,7 +3577,12 @@ found:
 
     njs_mp_free(vm->mem_pool, generator->context);
 
-    return njs_generate_children_indexes_release_pop(vm, generator, lvalue);
+    ret = njs_generate_children_indexes_release(vm, generator, lvalue);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    return njs_generator_stack_pop(vm, generator, NULL);
 }
 
 
@@ -4938,22 +4947,6 @@ njs_generate_children_indexes_release(nj
 
 
 static njs_int_t
-njs_generate_children_indexes_release_pop(njs_vm_t *vm,
-    njs_generator_t *generator, njs_parser_node_t *node)
-{
-    njs_int_t  ret;
-
-    ret = njs_generate_node_index_release(vm, generator, node->left);
-
-    if (njs_fast_path(ret == NJS_OK)) {
-        return njs_generate_node_index_release_pop(vm, generator, node->right);
-    }
-
-    return ret;
-}
-
-
-static njs_int_t
 njs_generate_node_index_release(njs_vm_t *vm, njs_generator_t *generator,
     njs_parser_node_t *node)
 {

From vl at nginx.com  Fri Dec 10 07:38:00 2021
From: vl at nginx.com (Vladimir Homutov)
Date: Fri, 10 Dec 2021 10:38:00 +0300
Subject: [PATCH 3 of 3] QUIC: stream recv shutdown support
In-Reply-To: <20211126131133.ew64tszf5vpjxii7@Romans-MacBook-Pro.local>
References: 
 
 <20211126131133.ew64tszf5vpjxii7@Romans-MacBook-Pro.local>
Message-ID: 

On Fri, Nov 26, 2021 at 04:11:33PM +0300, Roman Arutyunyan wrote:
> On Thu, Nov 25, 2021 at 05:20:51PM +0300, Roman Arutyunyan wrote:
> > # HG changeset patch
> > # User Roman Arutyunyan 
> > # Date 1637695967 -10800
> > #      Tue Nov 23 22:32:47 2021 +0300
> > # Branch quic
> > # Node ID e1de02d829f7f85b1e2e6b289ec4c20318712321
> > # Parent  3d2354bfa1a2a257b9f73772ad0836585be85a6c
> > QUIC: stream recv shutdown support.
> >
> > Recv shutdown sends STOP_SENDING to client.  Both send and recv shutdown
> > functions are now called from stream cleanup handler.  While here, setting
> > c->read->pending_eof is moved down to fix recv shutdown in the cleanup handler.
>
> This definitely needs some improvement.  Now it's two patches.

I suggest merging both into one (also, second needs rebasing)

>
> [..]
>
> --
> Roman Arutyunyan

> # HG changeset patch
> # User Roman Arutyunyan 
> # Date 1637931593 -10800
> #      Fri Nov 26 15:59:53 2021 +0300
> # Branch quic
> # Node ID c2fa3e7689a4e286f45ccbac2288ade5966273b8
> # Parent  3d2354bfa1a2a257b9f73772ad0836585be85a6c
> QUIC: do not shutdown write part of a client uni stream.
>
> diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> --- a/src/event/quic/ngx_event_quic_streams.c
> +++ b/src/event/quic/ngx_event_quic_streams.c
> @@ -267,13 +267,20 @@ ngx_quic_shutdown_stream(ngx_connection_
>          return NGX_OK;
>      }
>
> +    qs = c->quic;
> +
> +    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> +        && (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL))
> +    {
> +        return NGX_OK;
> +    }
> +
>      wev = c->write;
>
>      if (wev->error) {
>          return NGX_OK;
>      }
>
> -    qs = c->quic;
>      pc = qs->parent;
>      qc = ngx_quic_get_connection(pc);
>

this one looks good


> # HG changeset patch
> # User Roman Arutyunyan 
> # Date 1637932014 -10800
> #      Fri Nov 26 16:06:54 2021 +0300
> # Branch quic
> # Node ID ed0cefd9fc434a7593f2f9e4b9a98ce65aaf05e9
> # Parent  c2fa3e7689a4e286f45ccbac2288ade5966273b8
> QUIC: write and full stream shutdown support.
>
> Full stream shutdown is now called from stream cleanup handler instead of
> explicitly sending frames.  The call is moved up not to be influenced by
> setting c->read->pending_eof, which was erroneously set too early.
>
> diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> --- a/src/event/quic/ngx_event_quic_streams.c
> +++ b/src/event/quic/ngx_event_quic_streams.c
> @@ -13,6 +13,8 @@
>  #define NGX_QUIC_STREAM_GONE     (void *) -1
>
>
> +static ngx_int_t ngx_quic_shutdown_stream_send(ngx_connection_t *c);
> +static ngx_int_t ngx_quic_shutdown_stream_recv(ngx_connection_t *c);
>  static ngx_quic_stream_t *ngx_quic_get_stream(ngx_connection_t *c, uint64_t id);
>  static ngx_int_t ngx_quic_reject_stream(ngx_connection_t *c, uint64_t id);
>  static void ngx_quic_init_stream_handler(ngx_event_t *ev);
> @@ -257,16 +259,31 @@ ngx_quic_reset_stream(ngx_connection_t *
>  ngx_int_t
>  ngx_quic_shutdown_stream(ngx_connection_t *c, int how)
>  {
> +    if (how == NGX_RW_SHUTDOWN || how == NGX_WRITE_SHUTDOWN) {
> +        if (ngx_quic_shutdown_stream_send(c) != NGX_OK) {
> +            return NGX_ERROR;
> +        }
> +    }
> +
> +    if (how == NGX_RW_SHUTDOWN || how == NGX_READ_SHUTDOWN) {
> +        if (ngx_quic_shutdown_stream_recv(c) != NGX_OK) {
> +            return NGX_ERROR;
> +        }
> +    }
> +
> +    return NGX_OK;
> +}
> +
> +
> +static ngx_int_t
> +ngx_quic_shutdown_stream_send(ngx_connection_t *c)
> +{
>      ngx_event_t            *wev;
>      ngx_connection_t       *pc;
>      ngx_quic_frame_t       *frame;
>      ngx_quic_stream_t      *qs;
>      ngx_quic_connection_t  *qc;
>
> -    if (how != NGX_WRITE_SHUTDOWN) {
> -        return NGX_OK;
> -    }
> -
>      qs = c->quic;
>
>      if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> @@ -290,7 +307,7 @@ ngx_quic_shutdown_stream(ngx_connection_
>      }
>
>      ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> -                   "quic stream id:0x%xL shutdown", qs->id);
> +                   "quic stream id:0x%xL send shutdown", qs->id);
>
>      frame->level = ssl_encryption_application;
>      frame->type = NGX_QUIC_FT_STREAM;
> @@ -311,6 +328,55 @@ ngx_quic_shutdown_stream(ngx_connection_
>  }
>
>
> +static ngx_int_t
> +ngx_quic_shutdown_stream_recv(ngx_connection_t *c)
> +{
> +    ngx_event_t            *rev;
> +    ngx_connection_t       *pc;
> +    ngx_quic_frame_t       *frame;
> +    ngx_quic_stream_t      *qs;
> +    ngx_quic_connection_t  *qc;
> +
> +    qs = c->quic;
> +
> +    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED)
> +        && (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL))
> +    {

maybe it's worth trying to move server/client bidi/uni tests into
ngx_quic_shutdown_stream() ? It looks like more natural place to
test which end to shut, and whether we need to do it at all.

> +        return NGX_OK;
> +    }
> +
> +    rev = c->read;
> +
> +    if (rev->pending_eof || rev->error) {
> +        return NGX_OK;
> +    }
> +
> +    pc = qs->parent;
> +    qc = ngx_quic_get_connection(pc);
> +
> +    if (qc->conf->stream_close_code == 0) {
> +        return NGX_OK;
> +    }
> +
> +    frame = ngx_quic_alloc_frame(pc);
> +    if (frame == NULL) {
> +        return NGX_ERROR;
> +    }
> +
> +    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> +                   "quic stream id:0x%xL recv shutdown", qs->id);
> +
> +    frame->level = ssl_encryption_application;
> +    frame->type = NGX_QUIC_FT_STOP_SENDING;
> +    frame->u.stop_sending.id = qs->id;
> +    frame->u.stop_sending.error_code = qc->conf->stream_close_code;
> +
> +    ngx_quic_queue_frame(qc, frame);
> +
> +    return NGX_OK;
> +}
> +
> +
>  static ngx_quic_stream_t *
>  ngx_quic_get_stream(ngx_connection_t *c, uint64_t id)
>  {
> @@ -925,30 +991,12 @@ ngx_quic_stream_cleanup_handler(void *da
>          goto done;
>      }
>
> +    (void) ngx_quic_shutdown_stream(c, NGX_RW_SHUTDOWN);
> +
>      c->read->pending_eof = 1;

I think we need to move setting pending_eof this into
ngx_quic_shutdown_stream_recv():

 - the shutdown is supposed to be called only once
 - this is the flag tested for this purpose
 - this will be symmetrical with send that sets wev->error and tests it
   on enter
 - it seems perfectly natural for recv shutdown to set some flag
   prevents reading

or probalby this need to be done in ngx_quic_shutdown_stream(), as
currently it is set without taking into account uni/bidi server/client
difference.

>
>      (void) ngx_quic_update_flow(c, qs->recv_last);
>
> -    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> -        || (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)
> -    {
> -        if (!c->read->pending_eof && !c->read->error
> -            && qc->conf->stream_close_code)
> -        {
> -            frame = ngx_quic_alloc_frame(pc);
> -            if (frame == NULL) {
> -                goto done;
> -            }
> -
> -            frame->level = ssl_encryption_application;
> -            frame->type = NGX_QUIC_FT_STOP_SENDING;
> -            frame->u.stop_sending.id = qs->id;
> -            frame->u.stop_sending.error_code = qc->conf->stream_close_code;
> -
> -            ngx_quic_queue_frame(qc, frame);
> -        }
> -    }
> -
>      if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0) {
>          frame = ngx_quic_alloc_frame(pc);
>          if (frame == NULL) {
> @@ -968,37 +1016,8 @@ ngx_quic_stream_cleanup_handler(void *da
>          }
>
>          ngx_quic_queue_frame(qc, frame);
> -
> -        if (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) {
> -            /* do not send fin for client unidirectional streams */
> -            goto done;
> -        }
>      }
>
> -    if (c->write->error) {
> -        goto done;
> -    }
> -
> -    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> -                   "quic stream id:0x%xL send fin", qs->id);
> -
> -    frame = ngx_quic_alloc_frame(pc);
> -    if (frame == NULL) {
> -        goto done;
> -    }
> -
> -    frame->level = ssl_encryption_application;
> -    frame->type = NGX_QUIC_FT_STREAM;
> -    frame->u.stream.off = 1;
> -    frame->u.stream.len = 1;
> -    frame->u.stream.fin = 1;
> -
> -    frame->u.stream.stream_id = qs->id;
> -    frame->u.stream.offset = c->sent;
> -    frame->u.stream.length = 0;
> -
> -    ngx_quic_queue_frame(qc, frame);
> -
>  done:
>
>      (void) ngx_quic_output(pc);
> diff --git a/src/os/unix/ngx_socket.h b/src/os/unix/ngx_socket.h
> --- a/src/os/unix/ngx_socket.h
> +++ b/src/os/unix/ngx_socket.h
> @@ -13,6 +13,8 @@
>
>
>  #define NGX_WRITE_SHUTDOWN SHUT_WR
> +#define NGX_READ_SHUTDOWN  SHUT_RD
> +#define NGX_RW_SHUTDOWN    SHUT_RDWR

I suggest renaming this to NGX_RDWR_SHUTDOWN  - 'RW' looks to much like
'WR' and makes confusion

>
>  typedef int  ngx_socket_t;
>

> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel


From vl at nginx.com  Fri Dec 10 07:43:40 2021
From: vl at nginx.com (Vladimir Homutov)
Date: Fri, 10 Dec 2021 10:43:40 +0300
Subject: Congestion control questions
In-Reply-To: 
References: 
Message-ID: 

On Tue, Dec 07, 2021 at 06:05:48PM +0800, sun edward wrote:
> Hi dev team,
>        I have some questions about congestion control,   what's the current
> congestion control algorithm in nginx quic,  is there any way or plan to
> support CUBIC or BBR in nginx quic?
>
> thanks & regards

Currently we have implemented minimalistic congestion control, as
described in RFC 9002 [1].
There are no exact plans for implementing more advanced schemes, but it
is quite obvious that this area needs improvements, so we will have to
do something about it in future.


[1] https://www.rfc-editor.org/rfc/rfc9002.html#name-congestion-control

From pluknet at nginx.com  Fri Dec 10 14:00:22 2021
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 10 Dec 2021 17:00:22 +0300
Subject: [PATCH 2 of 3] QUIC: simplified stream initialization
In-Reply-To: <3d2354bfa1a2a257b9f7.1637850050@arut-laptop>
References: 
 <3d2354bfa1a2a257b9f7.1637850050@arut-laptop>
Message-ID: <2D555ACA-1983-433E-BBDF-00111CCBBA8D@nginx.com>


> On 25 Nov 2021, at 17:20, Roman Arutyunyan  wrote:
> 
> # HG changeset patch
> # User Roman Arutyunyan 
> # Date 1637693300 -10800
> #      Tue Nov 23 21:48:20 2021 +0300
> # Branch quic
> # Node ID 3d2354bfa1a2a257b9f73772ad0836585be85a6c
> # Parent  5b03ffd757804542daec73188a509b02e6b2c596
> QUIC: simplified stream initialization.
> 
> After creation, a client stream is added to qc->streams.uninitialized queue.
> After initialization it's removed from the queue.  If a stream is never
> initialized, it is freed in ngx_quic_close_streams().  Stream initializer
> is now set as read event handler in stream connection.
> 
> Previously qc->streams.uninitialized was used for delayed stream

was used *solely*? with the patch, it's still used to

> initialization.
> 
> The change makes is possible not to handle separately the case of a new stream

s/is/it/

> in stream-related frame handlers.  It makes these handlers simpler since new
> streams and existing streams are now handled by the same code.
> 

The change looks good.

-- 
Sergey Kandaurov


From xeioex at nginx.com  Fri Dec 10 14:11:31 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 10 Dec 2021 14:11:31 +0000
Subject: [njs] Modules: removed "js_include" directive deprecated in 0.4.0.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/dfcbfb5e27b2
branches:  
changeset: 1764:dfcbfb5e27b2
user:      Dmitry Volyntsev 
date:      Thu Dec 09 14:38:40 2021 +0000
description:
Modules: removed "js_include" directive deprecated in 0.4.0.

diffstat:

 nginx/ngx_http_js_module.c   |  128 ++++--------------------------------------
 nginx/ngx_stream_js_module.c |  128 ++++--------------------------------------
 2 files changed, 28 insertions(+), 228 deletions(-)

diffs (442 lines):

diff -r 00b5f28f2ec7 -r dfcbfb5e27b2 nginx/ngx_http_js_module.c
--- a/nginx/ngx_http_js_module.c	Wed Dec 08 13:14:59 2021 +0000
+++ b/nginx/ngx_http_js_module.c	Thu Dec 09 14:38:40 2021 +0000
@@ -14,9 +14,6 @@
 
 typedef struct {
     njs_vm_t              *vm;
-    ngx_str_t              include;
-    u_char                *file;
-    ngx_uint_t             line;
     ngx_array_t           *imports;
     ngx_array_t           *paths;
 } ngx_http_js_main_conf_t;
@@ -216,8 +213,6 @@ static void ngx_http_js_handle_event(ngx
     njs_vm_event_t vm_event, njs_value_t *args, njs_uint_t nargs);
 
 static ngx_int_t ngx_http_js_init(ngx_conf_t *cf);
-static char *ngx_http_js_include(ngx_conf_t *cf, ngx_command_t *cmd,
-    void *conf);
 static char *ngx_http_js_import(ngx_conf_t *cf, ngx_command_t *cmd,
     void *conf);
 static char *ngx_http_js_set(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
@@ -251,13 +246,6 @@ static ngx_conf_bitmask_t  ngx_http_js_s
 
 static ngx_command_t  ngx_http_js_commands[] = {
 
-    { ngx_string("js_include"),
-      NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1,
-      ngx_http_js_include,
-      NGX_HTTP_MAIN_CONF_OFFSET,
-      offsetof(ngx_http_js_main_conf_t, include),
-      NULL },
-
     { ngx_string("js_import"),
       NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE13,
       ngx_http_js_import,
@@ -3481,7 +3469,6 @@ ngx_http_js_init_main_conf(ngx_conf_t *c
 
     size_t                 size;
     u_char                *start, *end, *p;
-    ssize_t                n;
     ngx_fd_t               fd;
     ngx_str_t             *m, file;
     njs_int_t              rc;
@@ -3489,7 +3476,6 @@ ngx_http_js_init_main_conf(ngx_conf_t *c
     ngx_uint_t             i;
     njs_value_t           *value;
     njs_vm_opt_t           options;
-    ngx_file_info_t        fi;
     ngx_pool_cleanup_t    *cln;
     njs_opaque_value_t     lvalue, exception;
     ngx_http_js_import_t  *import;
@@ -3497,42 +3483,17 @@ ngx_http_js_init_main_conf(ngx_conf_t *c
     static const njs_str_t line_number_key = njs_str("lineNumber");
     static const njs_str_t file_name_key = njs_str("fileName");
 
-    if (jmcf->include.len == 0 && jmcf->imports == NGX_CONF_UNSET_PTR) {
+    if (jmcf->imports == NGX_CONF_UNSET_PTR) {
         return NGX_CONF_OK;
     }
 
     size = 0;
     fd = NGX_INVALID_FILE;
 
-    if (jmcf->include.len != 0) {
-        file = jmcf->include;
-
-        if (ngx_conf_full_name(cf->cycle, &file, 1) != NGX_OK) {
-            return NGX_CONF_ERROR;
-        }
-
-        fd = ngx_open_file(file.data, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
-        if (fd == NGX_INVALID_FILE) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, ngx_errno,
-                          ngx_open_file_n " \"%s\" failed", file.data);
-            return NGX_CONF_ERROR;
-        }
-
-        if (ngx_fd_info(fd, &fi) == NGX_FILE_ERROR) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, ngx_errno,
-                          ngx_fd_info_n " \"%s\" failed", file.data);
-            (void) ngx_close_file(fd);
-            return NGX_CONF_ERROR;
-        }
-
-        size = ngx_file_size(&fi);
-
-    } else {
-        import = jmcf->imports->elts;
-        for (i = 0; i < jmcf->imports->nelts; i++) {
-            size += sizeof("import  from '';\n") - 1 + import[i].name.len
-                    + import[i].path.len;
-        }
+    import = jmcf->imports->elts;
+    for (i = 0; i < jmcf->imports->nelts; i++) {
+        size += sizeof("import  from '';\n") - 1 + import[i].name.len
+                + import[i].path.len;
     }
 
     start = ngx_pnalloc(cf->pool, size);
@@ -3544,41 +3505,14 @@ ngx_http_js_init_main_conf(ngx_conf_t *c
         return NGX_CONF_ERROR;
     }
 
-    if (jmcf->include.len != 0) {
-        n = ngx_read_fd(fd, start, size);
-
-        if (n == -1) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, ngx_errno,
-                          ngx_read_fd_n " \"%s\" failed", file.data);
-
-            (void) ngx_close_file(fd);
-            return NGX_CONF_ERROR;
-        }
-
-        if ((size_t) n != size) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
-                          ngx_read_fd_n " has read only %z "
-                          "of %O from \"%s\"", n, size, file.data);
-
-            (void) ngx_close_file(fd);
-            return NGX_CONF_ERROR;
-        }
-
-        if (ngx_close_file(fd) == NGX_FILE_ERROR) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, ngx_errno,
-                          ngx_close_file_n " %s failed", file.data);
-        }
-
-    } else {
-        p = start;
-        import = jmcf->imports->elts;
-        for (i = 0; i < jmcf->imports->nelts; i++) {
-            p = ngx_cpymem(p, "import ", sizeof("import ") - 1);
-            p = ngx_cpymem(p, import[i].name.data, import[i].name.len);
-            p = ngx_cpymem(p, " from '", sizeof(" from '") - 1);
-            p = ngx_cpymem(p, import[i].path.data, import[i].path.len);
-            p = ngx_cpymem(p, "';\n", sizeof("';\n") - 1);
-        }
+    p = start;
+    import = jmcf->imports->elts;
+    for (i = 0; i < jmcf->imports->nelts; i++) {
+        p = ngx_cpymem(p, "import ", sizeof("import ") - 1);
+        p = ngx_cpymem(p, import[i].name.data, import[i].name.len);
+        p = ngx_cpymem(p, " from '", sizeof(" from '") - 1);
+        p = ngx_cpymem(p, import[i].path.data, import[i].path.len);
+        p = ngx_cpymem(p, "';\n", sizeof("';\n") - 1);
     }
 
     njs_vm_opt_init(&options);
@@ -3590,12 +3524,7 @@ ngx_http_js_init_main_conf(ngx_conf_t *c
     options.argv = ngx_argv;
     options.argc = ngx_argc;
 
-    if (jmcf->include.len != 0) {
-        file = jmcf->include;
-
-    } else {
-        file = ngx_cycle->conf_prefix;
-    }
+    file = ngx_cycle->conf_prefix;
 
     options.file.start = file.data;
     options.file.length = file.len;
@@ -3665,12 +3594,6 @@ ngx_http_js_init_main_conf(ngx_conf_t *c
         njs_value_assign(&exception, njs_vm_retval(jmcf->vm));
         njs_vm_retval_string(jmcf->vm, &text);
 
-        if (jmcf->include.len != 0) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, 0, "%*s, included in %s:%ui",
-                          text.length, text.start, jmcf->file, jmcf->line);
-            return NGX_CONF_ERROR;
-        }
-
         value = njs_vm_object_prop(jmcf->vm, njs_value_arg(&exception),
                                    &file_name_key, &lvalue);
         if (value == NULL) {
@@ -3720,22 +3643,6 @@ ngx_http_js_init(ngx_conf_t *cf)
 
 
 static char *
-ngx_http_js_include(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
-{
-    ngx_http_js_main_conf_t *jmcf = conf;
-
-    if (jmcf->imports != NGX_CONF_UNSET_PTR) {
-        return "is incompatible with \"js_import\"";
-    }
-
-    jmcf->file = cf->conf_file->file.name.data;
-    jmcf->line = cf->conf_file->line;
-
-    return ngx_conf_set_str_slot(cf, cmd, conf);
-}
-
-
-static char *
 ngx_http_js_import(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
 {
     ngx_http_js_main_conf_t *jmcf = conf;
@@ -3745,10 +3652,6 @@ ngx_http_js_import(ngx_conf_t *cf, ngx_c
     ngx_str_t             *value, name, path;
     ngx_http_js_import_t  *import;
 
-    if (jmcf->include.len != 0) {
-        return "is incompatible with \"js_include\"";
-    }
-
     value = cf->args->elts;
     from = (cf->args->nelts == 4);
 
@@ -4011,9 +3914,6 @@ ngx_http_js_create_main_conf(ngx_conf_t 
      * set by ngx_pcalloc():
      *
      *     conf->vm = NULL;
-     *     conf->include = { 0, NULL };
-     *     conf->file = NULL;
-     *     conf->line = 0;
      */
 
     conf->paths = NGX_CONF_UNSET_PTR;
diff -r 00b5f28f2ec7 -r dfcbfb5e27b2 nginx/ngx_stream_js_module.c
--- a/nginx/ngx_stream_js_module.c	Wed Dec 08 13:14:59 2021 +0000
+++ b/nginx/ngx_stream_js_module.c	Thu Dec 09 14:38:40 2021 +0000
@@ -14,9 +14,6 @@
 
 typedef struct {
     njs_vm_t              *vm;
-    ngx_str_t              include;
-    u_char                *file;
-    ngx_uint_t             line;
     ngx_array_t           *imports;
     ngx_array_t           *paths;
 } ngx_stream_js_main_conf_t;
@@ -130,8 +127,6 @@ static ngx_msec_t ngx_stream_js_resolver
 static void ngx_stream_js_handle_event(ngx_stream_session_t *s,
     njs_vm_event_t vm_event, njs_value_t *args, njs_uint_t nargs);
 
-static char *ngx_stream_js_include(ngx_conf_t *cf, ngx_command_t *cmd,
-    void *conf);
 static char *ngx_stream_js_import(ngx_conf_t *cf, ngx_command_t *cmd,
     void *conf);
 static char *ngx_stream_js_set(ngx_conf_t *cf, ngx_command_t *cmd,
@@ -165,13 +160,6 @@ static ngx_conf_bitmask_t  ngx_stream_js
 
 static ngx_command_t  ngx_stream_js_commands[] = {
 
-    { ngx_string("js_include"),
-      NGX_STREAM_MAIN_CONF|NGX_CONF_TAKE1,
-      ngx_stream_js_include,
-      NGX_STREAM_MAIN_CONF_OFFSET,
-      offsetof(ngx_stream_js_main_conf_t, include),
-      NULL },
-
     { ngx_string("js_import"),
       NGX_STREAM_MAIN_CONF|NGX_CONF_TAKE13,
       ngx_stream_js_import,
@@ -1503,7 +1491,6 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
 
     size_t                   size;
     u_char                  *start, *end, *p;
-    ssize_t                  n;
     ngx_fd_t                 fd;
     ngx_str_t               *m, file;
     njs_int_t                rc;
@@ -1511,7 +1498,6 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
     ngx_uint_t               i;
     njs_value_t             *value;
     njs_vm_opt_t             options;
-    ngx_file_info_t          fi;
     ngx_pool_cleanup_t      *cln;
     njs_opaque_value_t       lvalue, exception;
     ngx_stream_js_import_t  *import;
@@ -1519,42 +1505,17 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
     static const njs_str_t line_number_key = njs_str("lineNumber");
     static const njs_str_t file_name_key = njs_str("fileName");
 
-    if (jmcf->include.len == 0 && jmcf->imports == NGX_CONF_UNSET_PTR) {
+    if (jmcf->imports == NGX_CONF_UNSET_PTR) {
         return NGX_CONF_OK;
     }
 
     size = 0;
     fd = NGX_INVALID_FILE;
 
-    if (jmcf->include.len != 0) {
-        file = jmcf->include;
-
-        if (ngx_conf_full_name(cf->cycle, &file, 1) != NGX_OK) {
-            return NGX_CONF_ERROR;
-        }
-
-        fd = ngx_open_file(file.data, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
-        if (fd == NGX_INVALID_FILE) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, ngx_errno,
-                          ngx_open_file_n " \"%s\" failed", file.data);
-            return NGX_CONF_ERROR;
-        }
-
-        if (ngx_fd_info(fd, &fi) == NGX_FILE_ERROR) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, ngx_errno,
-                          ngx_fd_info_n " \"%s\" failed", file.data);
-            (void) ngx_close_file(fd);
-            return NGX_CONF_ERROR;
-        }
-
-        size = ngx_file_size(&fi);
-
-    } else {
-        import = jmcf->imports->elts;
-        for (i = 0; i < jmcf->imports->nelts; i++) {
-            size += sizeof("import  from '';\n") - 1 + import[i].name.len
-                    + import[i].path.len;
-        }
+    import = jmcf->imports->elts;
+    for (i = 0; i < jmcf->imports->nelts; i++) {
+        size += sizeof("import  from '';\n") - 1 + import[i].name.len
+                + import[i].path.len;
     }
 
     start = ngx_pnalloc(cf->pool, size);
@@ -1566,41 +1527,14 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
         return NGX_CONF_ERROR;
     }
 
-    if (jmcf->include.len != 0) {
-        n = ngx_read_fd(fd, start, size);
-
-        if (n == -1) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, ngx_errno,
-                          ngx_read_fd_n " \"%s\" failed", file.data);
-
-            (void) ngx_close_file(fd);
-            return NGX_CONF_ERROR;
-        }
-
-        if ((size_t) n != size) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
-                          ngx_read_fd_n " has read only %z "
-                          "of %O from \"%s\"", n, size, file.data);
-
-            (void) ngx_close_file(fd);
-            return NGX_CONF_ERROR;
-        }
-
-        if (ngx_close_file(fd) == NGX_FILE_ERROR) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, ngx_errno,
-                          ngx_close_file_n " %s failed", file.data);
-        }
-
-    } else {
-        p = start;
-        import = jmcf->imports->elts;
-        for (i = 0; i < jmcf->imports->nelts; i++) {
-            p = ngx_cpymem(p, "import ", sizeof("import ") - 1);
-            p = ngx_cpymem(p, import[i].name.data, import[i].name.len);
-            p = ngx_cpymem(p, " from '", sizeof(" from '") - 1);
-            p = ngx_cpymem(p, import[i].path.data, import[i].path.len);
-            p = ngx_cpymem(p, "';\n", sizeof("';\n") - 1);
-        }
+    p = start;
+    import = jmcf->imports->elts;
+    for (i = 0; i < jmcf->imports->nelts; i++) {
+        p = ngx_cpymem(p, "import ", sizeof("import ") - 1);
+        p = ngx_cpymem(p, import[i].name.data, import[i].name.len);
+        p = ngx_cpymem(p, " from '", sizeof(" from '") - 1);
+        p = ngx_cpymem(p, import[i].path.data, import[i].path.len);
+        p = ngx_cpymem(p, "';\n", sizeof("';\n") - 1);
     }
 
     njs_vm_opt_init(&options);
@@ -1612,12 +1546,7 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
     options.argv = ngx_argv;
     options.argc = ngx_argc;
 
-    if (jmcf->include.len != 0) {
-        file = jmcf->include;
-
-    } else {
-        file = ngx_cycle->conf_prefix;
-    }
+    file = ngx_cycle->conf_prefix;
 
     options.file.start = file.data;
     options.file.length = file.len;
@@ -1687,12 +1616,6 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
         njs_value_assign(&exception, njs_vm_retval(jmcf->vm));
         njs_vm_retval_string(jmcf->vm, &text);
 
-        if (jmcf->include.len != 0) {
-            ngx_log_error(NGX_LOG_EMERG, cf->log, 0, "%*s, included in %s:%ui",
-                          text.length, text.start, jmcf->file, jmcf->line);
-            return NGX_CONF_ERROR;
-        }
-
         value = njs_vm_object_prop(jmcf->vm, njs_value_arg(&exception),
                                    &file_name_key, &lvalue);
         if (value == NULL) {
@@ -1729,22 +1652,6 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
 
 
 static char *
-ngx_stream_js_include(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
-{
-    ngx_stream_js_main_conf_t *jmcf = conf;
-
-    if (jmcf->imports != NGX_CONF_UNSET_PTR) {
-        return "is incompatible with \"js_import\"";
-    }
-
-    jmcf->file = cf->conf_file->file.name.data;
-    jmcf->line = cf->conf_file->line;
-
-    return ngx_conf_set_str_slot(cf, cmd, conf);
-}
-
-
-static char *
 ngx_stream_js_import(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
 {
     ngx_stream_js_main_conf_t *jmcf = conf;
@@ -1754,10 +1661,6 @@ ngx_stream_js_import(ngx_conf_t *cf, ngx
     ngx_str_t               *value, name, path;
     ngx_stream_js_import_t  *import;
 
-    if (jmcf->include.len != 0) {
-        return "is incompatible with \"js_include\"";
-    }
-
     value = cf->args->elts;
     from = (cf->args->nelts == 4);
 
@@ -1959,9 +1862,6 @@ ngx_stream_js_create_main_conf(ngx_conf_
      * set by ngx_pcalloc():
      *
      *     conf->vm = NULL;
-     *     conf->include = { 0, NULL };
-     *     conf->file = NULL;
-     *     conf->line = 0;
      */
 
     conf->paths = NGX_CONF_UNSET_PTR;

From zhenyatitoova at gmail.com  Sun Dec 12 12:51:50 2021
From: zhenyatitoova at gmail.com (=?iso-8859-1?q?Evgenia_Titova?=)
Date: Sun, 12 Dec 2021 15:51:50 +0300
Subject: [PATCH] on_no_ca parameter added to ssl_verify_client directive
Message-ID: 

This parameter requires the client certificate but does not require it to be signed by a trusted CA certificate.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx.patch
Type: text/x-patch
Size: 5699 bytes
Desc: not available
URL: 

From mdounin at mdounin.ru  Sun Dec 12 11:22:36 2021
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 12 Dec 2021 14:22:36 +0300
Subject: [PATCH 0 of 4] PCRE2 support
Message-ID: 

Hello!

The following patch series adds PCRE2 support.
Review and testing appreciated.

-- 
Maxim Dounin


From mdounin at mdounin.ru  Sun Dec 12 11:22:39 2021
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 12 Dec 2021 14:22:39 +0300
Subject: [PATCH 3 of 4] Configure: simplified PCRE compilation
In-Reply-To: 
References: 
Message-ID: <84fae18c1b1e9d5cfafe.1639308159@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin 
# Date 1639235686 -10800
#      Sat Dec 11 18:14:46 2021 +0300
# Node ID 84fae18c1b1e9d5cfafe10e0e9754f1d7ba628e1
# Parent  c03ed58d70b6bd37acf289621d514a7497c8815e
Configure: simplified PCRE compilation.

Removed ICC-specific PCRE optimizations which tried to link with PCRE
object files instead of the library.  Made compiler-specific code
minimal.

diff --git a/auto/lib/pcre/conf b/auto/lib/pcre/conf
--- a/auto/lib/pcre/conf
+++ b/auto/lib/pcre/conf
@@ -4,81 +4,24 @@
 
 
 if [ $PCRE != NONE ]; then
+
+    have=NGX_PCRE . auto/have
+
+    if [ "$NGX_PLATFORM" = win32 ]; then
+        have=PCRE_STATIC . auto/have
+    fi
+
     CORE_INCS="$CORE_INCS $PCRE"
+    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
 
     case "$NGX_CC_NAME" in
 
         msvc | owc | bcc)
-            have=NGX_PCRE . auto/have
-            have=PCRE_STATIC . auto/have
-            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
             LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
             CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
         ;;
 
-        icc)
-            have=NGX_PCRE . auto/have
-            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
-
-            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
-
-            echo $ngx_n "checking for PCRE library ...$ngx_c"
-
-            if [ -f $PCRE/pcre.h ]; then
-                ngx_pcre_ver=`grep PCRE_MAJOR $PCRE/pcre.h \
-                              | sed -e 's/^.*PCRE_MAJOR.* \(.*\)$/\1/'`
-
-            else if [ -f $PCRE/configure.in ]; then
-                ngx_pcre_ver=`grep PCRE_MAJOR= $PCRE/configure.in \
-                              | sed -e 's/^.*=\(.*\)$/\1/'`
-
-            else
-                ngx_pcre_ver=`grep pcre_major, $PCRE/configure.ac \
-                              | sed -e 's/^.*pcre_major,.*\[\(.*\)\].*$/\1/'`
-            fi
-            fi
-
-            echo " $ngx_pcre_ver major version found"
-
-            # to allow -ipo optimization we link with the *.o but not library
-
-            case "$ngx_pcre_ver" in
-                4|5)
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre.o"
-                ;;
-
-                6)
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
-                ;;
-
-                *)
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_newline.o"
-                ;;
-
-            esac
-        ;;
-
         *)
-            have=NGX_PCRE . auto/have
-
-            if [ "$NGX_PLATFORM" = win32 ]; then
-                have=PCRE_STATIC . auto/have
-            fi
-
-            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
             LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
             CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
         ;;


From mdounin at mdounin.ru  Sun Dec 12 11:22:38 2021
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 12 Dec 2021 14:22:38 +0300
Subject: [PATCH 2 of 4] Core: ngx_regex.c style cleanup
In-Reply-To: 
References: 
Message-ID: 

# HG changeset patch
# User Maxim Dounin 
# Date 1639235738 -10800
#      Sat Dec 11 18:15:38 2021 +0300
# Node ID c03ed58d70b6bd37acf289621d514a7497c8815e
# Parent  7b0522664bd790fda5d25df9087759953b30ac29
Core: ngx_regex.c style cleanup.

Notably, ngx_pcre_pool and ngx_pcre_studies are renamed to ngx_regex_pool
and ngx_regex_studies, respectively.

diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
--- a/src/core/ngx_regex.c
+++ b/src/core/ngx_regex.c
@@ -64,8 +64,8 @@ ngx_module_t  ngx_regex_module = {
 };
 
 
-static ngx_pool_t  *ngx_pcre_pool;
-static ngx_list_t  *ngx_pcre_studies;
+static ngx_pool_t  *ngx_regex_pool;
+static ngx_list_t  *ngx_regex_studies;
 
 
 void
@@ -79,14 +79,14 @@ ngx_regex_init(void)
 static ngx_inline void
 ngx_regex_malloc_init(ngx_pool_t *pool)
 {
-    ngx_pcre_pool = pool;
+    ngx_regex_pool = pool;
 }
 
 
 static ngx_inline void
 ngx_regex_malloc_done(void)
 {
-    ngx_pcre_pool = NULL;
+    ngx_regex_pool = NULL;
 }
 
 
@@ -112,13 +112,13 @@ ngx_regex_compile(ngx_regex_compile_t *r
            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
                               "pcre_compile() failed: %s in \"%V\"",
                                errstr, &rc->pattern)
-                      - rc->err.data;
+                         - rc->err.data;
 
         } else {
            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
                               "pcre_compile() failed: %s in \"%V\" at \"%s\"",
                                errstr, &rc->pattern, rc->pattern.data + erroff)
-                      - rc->err.data;
+                         - rc->err.data;
         }
 
         return NGX_ERROR;
@@ -133,8 +133,8 @@ ngx_regex_compile(ngx_regex_compile_t *r
 
     /* do not study at runtime */
 
-    if (ngx_pcre_studies != NULL) {
-        elt = ngx_list_push(ngx_pcre_studies);
+    if (ngx_regex_studies != NULL) {
+        elt = ngx_list_push(ngx_regex_studies);
         if (elt == NULL) {
             goto nomem;
         }
@@ -229,11 +229,8 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
 static void * ngx_libc_cdecl
 ngx_regex_malloc(size_t size)
 {
-    ngx_pool_t      *pool;
-    pool = ngx_pcre_pool;
-
-    if (pool) {
-        return ngx_palloc(pool, size);
+    if (ngx_regex_pool) {
+        return ngx_palloc(ngx_regex_pool, size);
     }
 
     return NULL;
@@ -286,10 +283,10 @@ ngx_regex_cleanup(void *data)
 
     /*
      * On configuration parsing errors ngx_regex_module_init() will not
-     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
+     * be called.  Make sure ngx_regex_studies is properly cleared anyway.
      */
 
-    ngx_pcre_studies = NULL;
+    ngx_regex_studies = NULL;
 }
 
 
@@ -357,7 +354,7 @@ ngx_regex_module_init(ngx_cycle_t *cycle
 
     ngx_regex_malloc_done();
 
-    ngx_pcre_studies = NULL;
+    ngx_regex_studies = NULL;
 
     return NGX_OK;
 }
@@ -389,7 +386,7 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
         return NULL;
     }
 
-    ngx_pcre_studies = rcf->studies;
+    ngx_regex_studies = rcf->studies;
 
     return rcf;
 }


From mdounin at mdounin.ru  Sun Dec 12 11:22:37 2021
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 12 Dec 2021 14:22:37 +0300
Subject: [PATCH 1 of 4] Core: fixed ngx_pcre_studies cleanup
In-Reply-To: 
References: 
Message-ID: <7b0522664bd790fda5d2.1639308157@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin 
# Date 1639235685 -10800
#      Sat Dec 11 18:14:45 2021 +0300
# Node ID 7b0522664bd790fda5d25df9087759953b30ac29
# Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
Core: fixed ngx_pcre_studies cleanup.

If a configuration parsing fails for some reason, ngx_regex_module_init()
is not called, and ngx_pcre_studies remained set despite the fact that
the pool it was allocated from is already freed.  This might result in
a segmentation fault during runtime regular expression compilation, such
as in SSI, for example, in the single process mode, or if a worker process
dies and respawn from a master process in such an inconsistent state.

Fix is to clear ngx_pcre_studies from the pool cleanup handler (which is
anyway used to free JIT-compiled patterns).

diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
--- a/src/core/ngx_regex.c
+++ b/src/core/ngx_regex.c
@@ -10,15 +10,14 @@
 
 
 typedef struct {
-    ngx_flag_t  pcre_jit;
+    ngx_flag_t   pcre_jit;
+    ngx_list_t  *studies;
 } ngx_regex_conf_t;
 
 
 static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
 static void ngx_libc_cdecl ngx_regex_free(void *p);
-#if (NGX_HAVE_PCRE_JIT)
-static void ngx_pcre_free_studies(void *data);
-#endif
+static void ngx_regex_cleanup(void *data);
 
 static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
 
@@ -248,18 +247,17 @@ ngx_regex_free(void *p)
 }
 
 
+static void
+ngx_regex_cleanup(void *data)
+{
 #if (NGX_HAVE_PCRE_JIT)
-
-static void
-ngx_pcre_free_studies(void *data)
-{
-    ngx_list_t *studies = data;
+    ngx_regex_conf_t *rcf = data;
 
     ngx_uint_t        i;
     ngx_list_part_t  *part;
     ngx_regex_elt_t  *elts;
 
-    part = &studies->part;
+    part = &rcf->studies->part;
     elts = part->elts;
 
     for (i = 0; /* void */ ; i++) {
@@ -274,56 +272,50 @@ ngx_pcre_free_studies(void *data)
             i = 0;
         }
 
-        if (elts[i].regex->extra != NULL) {
-            pcre_free_study(elts[i].regex->extra);
-        }
-    }
-}
-
-#endif
-
-
-static ngx_int_t
-ngx_regex_module_init(ngx_cycle_t *cycle)
-{
-    int               opt;
-    const char       *errstr;
-    ngx_uint_t        i;
-    ngx_list_part_t  *part;
-    ngx_regex_elt_t  *elts;
-
-    opt = 0;
-
-#if (NGX_HAVE_PCRE_JIT)
-    {
-    ngx_regex_conf_t    *rcf;
-    ngx_pool_cleanup_t  *cln;
-
-    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
-
-    if (rcf->pcre_jit) {
-        opt = PCRE_STUDY_JIT_COMPILE;
-
         /*
          * The PCRE JIT compiler uses mmap for its executable codes, so we
          * have to explicitly call the pcre_free_study() function to free
          * this memory.
          */
 
-        cln = ngx_pool_cleanup_add(cycle->pool, 0);
-        if (cln == NULL) {
-            return NGX_ERROR;
+        if (elts[i].regex->extra != NULL) {
+            pcre_free_study(elts[i].regex->extra);
         }
+    }
+#endif
 
-        cln->handler = ngx_pcre_free_studies;
-        cln->data = ngx_pcre_studies;
-    }
+    /*
+     * On configuration parsing errors ngx_regex_module_init() will not
+     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
+     */
+
+    ngx_pcre_studies = NULL;
+}
+
+
+static ngx_int_t
+ngx_regex_module_init(ngx_cycle_t *cycle)
+{
+    int                opt;
+    const char        *errstr;
+    ngx_uint_t         i;
+    ngx_list_part_t   *part;
+    ngx_regex_elt_t   *elts;
+    ngx_regex_conf_t  *rcf;
+
+    opt = 0;
+
+    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
+
+#if (NGX_HAVE_PCRE_JIT)
+    if (rcf->pcre_jit) {
+        opt = PCRE_STUDY_JIT_COMPILE;
     }
 #endif
 
     ngx_regex_malloc_init(cycle->pool);
 
-    part = &ngx_pcre_studies->part;
+    part = &rcf->studies->part;
     elts = part->elts;
 
     for (i = 0; /* void */ ; i++) {
@@ -374,7 +366,8 @@ ngx_regex_module_init(ngx_cycle_t *cycle
 static void *
 ngx_regex_create_conf(ngx_cycle_t *cycle)
 {
-    ngx_regex_conf_t  *rcf;
+    ngx_regex_conf_t    *rcf;
+    ngx_pool_cleanup_t  *cln;
 
     rcf = ngx_pcalloc(cycle->pool, sizeof(ngx_regex_conf_t));
     if (rcf == NULL) {
@@ -383,11 +376,21 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
 
     rcf->pcre_jit = NGX_CONF_UNSET;
 
-    ngx_pcre_studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
-    if (ngx_pcre_studies == NULL) {
+    cln = ngx_pool_cleanup_add(cycle->pool, 0);
+    if (cln == NULL) {
         return NULL;
     }
 
+    cln->handler = ngx_regex_cleanup;
+    cln->data = rcf;
+
+    rcf->studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
+    if (rcf->studies == NULL) {
+        return NULL;
+    }
+
+    ngx_pcre_studies = rcf->studies;
+
     return rcf;
 }
 


From mdounin at mdounin.ru  Sun Dec 12 11:22:40 2021
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 12 Dec 2021 14:22:40 +0300
Subject: [PATCH 4 of 4] PCRE2 library support
In-Reply-To: 
References: 
Message-ID: 

# HG changeset patch
# User Maxim Dounin 
# Date 1639235687 -10800
#      Sat Dec 11 18:14:47 2021 +0300
# Node ID dfd93beb55f7ea950efa80a580b3ffc571588d5b
# Parent  84fae18c1b1e9d5cfafe10e0e9754f1d7ba628e1
PCRE2 library support.

The PCRE2 library is now used by default if found, instead of the
original PCRE library.  If needed for some reason, this can be disabled
with the --without-pcre2 configure option.

To make it possible to specify paths to the library and include files
via --with-cc-opt / --with-ld-opt, the library is first tested without
any additional paths and options.  If this fails, the pcre2-config script
is used.

Similarly to the original PCRE library, it is now possible to build PCRE2
from sources with nginx configure, by using the --with-pcre= option.
It automatically detects if PCRE or PCRE2 sources are provided.

Note that compiling PCRE2 10.33 and later requires inttypes.h.  When
compiling on Windows with MSVC, inttypes.h is only available starting
with MSVC 2013.  In older versions some replacement needs to be provided
("echo '#include ' > pcre2-10.xx/src/inttypes.h" is good enough
for MSVC 2010).

The interface on nginx side remains unchanged.

diff --git a/auto/lib/pcre/conf b/auto/lib/pcre/conf
--- a/auto/lib/pcre/conf
+++ b/auto/lib/pcre/conf
@@ -5,29 +5,61 @@
 
 if [ $PCRE != NONE ]; then
 
-    have=NGX_PCRE . auto/have
+    if [ -f $PCRE/src/pcre2.h.generic ]; then
+
+        PCRE_LIBRARY=PCRE2
+
+        have=NGX_PCRE . auto/have
+        have=NGX_PCRE2 . auto/have
+
+        if [ "$NGX_PLATFORM" = win32 ]; then
+            have=PCRE2_STATIC . auto/have
+        fi
+
+        CORE_INCS="$CORE_INCS $PCRE/src/"
+        CORE_DEPS="$CORE_DEPS $PCRE/src/pcre2.h"
 
-    if [ "$NGX_PLATFORM" = win32 ]; then
-        have=PCRE_STATIC . auto/have
-    fi
+        case "$NGX_CC_NAME" in
+
+            msvc)
+                LINK_DEPS="$LINK_DEPS $PCRE/src/pcre2-8.lib"
+                CORE_LIBS="$CORE_LIBS $PCRE/src/pcre2-8.lib"
+            ;;
 
-    CORE_INCS="$CORE_INCS $PCRE"
-    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
+            *)
+                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre2-8.a"
+                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre2-8.a"
+            ;;
 
-    case "$NGX_CC_NAME" in
+        esac
 
-        msvc | owc | bcc)
-            LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
-            CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
-        ;;
+    else
+
+        PCRE_LIBRARY=PCRE
+
+        have=NGX_PCRE . auto/have
+
+        if [ "$NGX_PLATFORM" = win32 ]; then
+            have=PCRE_STATIC . auto/have
+        fi
+
+        CORE_INCS="$CORE_INCS $PCRE"
+        CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
 
-        *)
-            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
-            CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
-        ;;
+        case "$NGX_CC_NAME" in
+
+            msvc | owc | bcc)
+                LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
+                CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
+            ;;
 
-    esac
+            *)
+                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
+                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
+            ;;
 
+        esac
+    fi
 
     if [ $PCRE_JIT = YES ]; then
         have=NGX_HAVE_PCRE_JIT . auto/have
@@ -37,8 +69,48 @@ if [ $PCRE != NONE ]; then
 else
 
     if [ "$NGX_PLATFORM" != win32 ]; then
+        PCRE=NO
+    fi
 
-        PCRE=NO
+    if [ $PCRE = NO -a $PCRE2 != DISABLED ]; then
+
+        ngx_feature="PCRE2 library"
+        ngx_feature_name="NGX_PCRE2"
+        ngx_feature_run=no
+        ngx_feature_incs="#define PCRE2_CODE_UNIT_WIDTH 8
+                          #include "
+        ngx_feature_path=
+        ngx_feature_libs="-lpcre2-8"
+        ngx_feature_test="pcre2_code *re;
+                          re = pcre2_compile(NULL, 0, 0, NULL, NULL, NULL);
+                          if (re == NULL) return 1"
+        . auto/feature
+
+        if [ $ngx_found = no ]; then
+
+            # pcre2-config
+
+            ngx_pcre2_prefix=`pcre2-config --prefix 2>/dev/null`
+
+            if [ -n "$ngx_pcre2_prefix" ]; then
+                ngx_feature="PCRE2 library in $ngx_pcre2_prefix"
+                ngx_feature_path=`pcre2-config --cflags \
+                                  | sed -n -e 's/.*-I *\([^ ][^ ]*\).*/\1/p'`
+                ngx_feature_libs=`pcre2-config --libs8`
+                . auto/feature
+            fi
+        fi
+
+        if [ $ngx_found = yes ]; then
+            have=NGX_PCRE . auto/have
+            CORE_INCS="$CORE_INCS $ngx_feature_path"
+            CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
+            PCRE=YES
+            PCRE_LIBRARY=PCRE2
+        fi
+    fi
+
+    if [ $PCRE = NO ]; then
 
         ngx_feature="PCRE library"
         ngx_feature_name="NGX_PCRE"
@@ -114,6 +186,7 @@ else
             CORE_INCS="$CORE_INCS $ngx_feature_path"
             CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
             PCRE=YES
+            PCRE_LIBRARY=PCRE
         fi
 
         if [ $PCRE = YES ]; then
diff --git a/auto/lib/pcre/make b/auto/lib/pcre/make
--- a/auto/lib/pcre/make
+++ b/auto/lib/pcre/make
@@ -3,36 +3,138 @@
 # Copyright (C) Nginx, Inc.
 
 
-case "$NGX_CC_NAME" in
+if [ $PCRE_LIBRARY = PCRE2 ]; then
+
+    # PCRE2
+
+    if [ $NGX_CC_NAME = msvc ]; then
+
+        # With PCRE2, it is not possible to compile all sources.
+        # Since list of source files changes between versions, we
+        # test files which might not be present.
 
-    msvc)
-        ngx_makefile=makefile.msvc
-        ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
-        ngx_pcre="PCRE=\"$PCRE\""
-    ;;
+        ngx_pcre_srcs="pcre2_auto_possess.c \
+                       pcre2_chartables.c \
+                       pcre2_compile.c \
+                       pcre2_config.c \
+                       pcre2_context.c \
+                       pcre2_dfa_match.c \
+                       pcre2_error.c \
+                       pcre2_jit_compile.c \
+                       pcre2_maketables.c \
+                       pcre2_match.c \
+                       pcre2_match_data.c \
+                       pcre2_newline.c \
+                       pcre2_ord2utf.c \
+                       pcre2_pattern_info.c \
+                       pcre2_string_utils.c \
+                       pcre2_study.c \
+                       pcre2_substitute.c \
+                       pcre2_substring.c \
+                       pcre2_tables.c \
+                       pcre2_ucd.c \
+                       pcre2_valid_utf.c \
+                       pcre2_xclass.c"
+
+        ngx_pcre_test="pcre2_convert.c \
+                       pcre2_extuni.c \
+                       pcre2_find_bracket.c \
+                       pcre2_script_run.c \
+                       pcre2_serialize.c"
+
+        for ngx_src in $ngx_pcre_test
+        do
+            if [ -f $PCRE/src/$ngx_src ]; then
+                ngx_pcre_srcs="$ngx_pcre_srcs $ngx_src"
+            fi
+        done
 
-    owc)
-        ngx_makefile=makefile.owc
-        ngx_opt="CPU_OPT=\"$CPU_OPT\""
-        ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
-    ;;
+        ngx_pcre_objs=`echo $ngx_pcre_srcs \
+            | sed -e "s#\([^ ]*\.\)c#\1$ngx_objext#g"`
+
+        ngx_pcre_srcs=`echo $ngx_pcre_srcs \
+            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
+        ngx_pcre_objs=`echo $ngx_pcre_objs \
+            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
+
+        cat << END                                            >> $NGX_MAKEFILE
+
+PCRE_CFLAGS =	-O2 -Ob1 -Oi -Gs $LIBC $CPU_OPT
+PCRE_FLAGS =	-DHAVE_CONFIG_H -DPCRE2_STATIC -DPCRE2_CODE_UNIT_WIDTH=8 \\
+		-DHAVE_MEMMOVE
+
+PCRE_SRCS =	 $ngx_pcre_srcs
+PCRE_OBJS =	 $ngx_pcre_objs
+
+$PCRE/src/pcre2.h:
+	cd $PCRE/src \\
+	&& copy /y config.h.generic config.h \\
+	&& copy /y pcre2.h.generic pcre2.h \\
+	&& copy /y pcre2_chartables.c.dist pcre2_chartables.c
 
-    bcc)
-        ngx_makefile=makefile.bcc
-        ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
-        ngx_pcre=`echo \-DPCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
-    ;;
+$PCRE/src/pcre2-8.lib:	$PCRE/src/pcre2.h $NGX_MAKEFILE
+	cd $PCRE/src \\
+	&& cl -nologo -c \$(PCRE_CFLAGS) -I . \$(PCRE_FLAGS) \$(PCRE_SRCS) \\
+	&& link -lib -out:pcre2-8.lib -verbose:lib \$(PCRE_OBJS)
+
+END
+
+    else
+
+        cat << END                                            >> $NGX_MAKEFILE
+
+$PCRE/src/pcre2.h:	$PCRE/Makefile
 
-    *)
-        ngx_makefile=
-    ;;
+$PCRE/Makefile:	$NGX_MAKEFILE
+	cd $PCRE \\
+	&& if [ -f Makefile ]; then \$(MAKE) distclean; fi \\
+	&& CC="\$(CC)" CFLAGS="$PCRE_OPT" \\
+	./configure --disable-shared $PCRE_CONF_OPT
 
-esac
+$PCRE/.libs/libpcre2-8.a:	$PCRE/Makefile
+	cd $PCRE \\
+	&& \$(MAKE) libpcre2-8.la
+
+END
+
+    fi
 
 
-if [ -n "$ngx_makefile" ]; then
+else
+
+    # PCRE
+
+    case "$NGX_CC_NAME" in
+
+        msvc)
+            ngx_makefile=makefile.msvc
+            ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
+            ngx_pcre="PCRE=\"$PCRE\""
+        ;;
+
+        owc)
+            ngx_makefile=makefile.owc
+            ngx_opt="CPU_OPT=\"$CPU_OPT\""
+            ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
+        ;;
 
-    cat << END                                                >> $NGX_MAKEFILE
+        bcc)
+            ngx_makefile=makefile.bcc
+            ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
+            ngx_pcre=`echo \-DPCRE=\"$PCRE\" \
+                | sed -e "s/\//$ngx_regex_dirsep/g"`
+        ;;
+
+        *)
+            ngx_makefile=
+        ;;
+
+    esac
+
+
+    if [ -n "$ngx_makefile" ]; then
+
+        cat << END                                            >> $NGX_MAKEFILE
 
 `echo "$PCRE/pcre.lib:	$PCRE/pcre.h $NGX_MAKEFILE"			\
 	| sed -e "s/\//$ngx_regex_dirsep/g"`
@@ -43,9 +145,9 @@ if [ -n "$ngx_makefile" ]; then
 
 END
 
-else
+    else
 
-    cat << END                                                >> $NGX_MAKEFILE
+        cat << END                                            >> $NGX_MAKEFILE
 
 $PCRE/pcre.h:	$PCRE/Makefile
 
@@ -61,4 +163,6 @@ else
 
 END
 
+    fi
+
 fi
diff --git a/auto/options b/auto/options
--- a/auto/options
+++ b/auto/options
@@ -146,6 +146,7 @@ PCRE=NONE
 PCRE_OPT=
 PCRE_CONF_OPT=
 PCRE_JIT=NO
+PCRE2=YES
 
 USE_OPENSSL=NO
 OPENSSL=NONE
@@ -357,6 +358,7 @@ use the \"--with-mail_ssl_module\" optio
         --with-pcre=*)                   PCRE="$value"              ;;
         --with-pcre-opt=*)               PCRE_OPT="$value"          ;;
         --with-pcre-jit)                 PCRE_JIT=YES               ;;
+        --without-pcre2)                 PCRE2=DISABLED             ;;
 
         --with-openssl=*)                OPENSSL="$value"           ;;
         --with-openssl-opt=*)            OPENSSL_OPT="$value"       ;;
@@ -573,6 +575,7 @@ cat << END
   --with-pcre=DIR                    set path to PCRE library sources
   --with-pcre-opt=OPTIONS            set additional build options for PCRE
   --with-pcre-jit                    build PCRE with JIT compilation support
+  --without-pcre2                    do not use PCRE2 library
 
   --with-zlib=DIR                    set path to zlib library sources
   --with-zlib-opt=OPTIONS            set additional build options for zlib
diff --git a/auto/summary b/auto/summary
--- a/auto/summary
+++ b/auto/summary
@@ -16,9 +16,9 @@ if [ $USE_PCRE = DISABLED ]; then
 
 else
     case $PCRE in
-        YES)   echo "  + using system PCRE library" ;;
+        YES)   echo "  + using system $PCRE_LIBRARY library" ;;
         NONE)  echo "  + PCRE library is not used" ;;
-        *)     echo "  + using PCRE library: $PCRE" ;;
+        *)     echo "  + using $PCRE_LIBRARY library: $PCRE" ;;
     esac
 fi
 
diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
--- a/src/core/ngx_regex.c
+++ b/src/core/ngx_regex.c
@@ -15,8 +15,16 @@ typedef struct {
 } ngx_regex_conf_t;
 
 
+static ngx_inline void ngx_regex_malloc_init(ngx_pool_t *pool);
+static ngx_inline void ngx_regex_malloc_done(void);
+
+#if (NGX_PCRE2)
+static void * ngx_libc_cdecl ngx_regex_malloc(size_t size, void *data);
+static void ngx_libc_cdecl ngx_regex_free(void *p, void *data);
+#else
 static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
 static void ngx_libc_cdecl ngx_regex_free(void *p);
+#endif
 static void ngx_regex_cleanup(void *data);
 
 static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
@@ -64,15 +72,24 @@ ngx_module_t  ngx_regex_module = {
 };
 
 
-static ngx_pool_t  *ngx_regex_pool;
-static ngx_list_t  *ngx_regex_studies;
+static ngx_pool_t             *ngx_regex_pool;
+static ngx_list_t             *ngx_regex_studies;
+static ngx_uint_t              ngx_regex_direct_alloc;
+
+#if (NGX_PCRE2)
+static pcre2_compile_context  *ngx_regex_compile_context;
+static pcre2_match_data       *ngx_regex_match_data;
+static ngx_uint_t              ngx_regex_match_data_size;
+#endif
 
 
 void
 ngx_regex_init(void)
 {
+#if !(NGX_PCRE2)
     pcre_malloc = ngx_regex_malloc;
     pcre_free = ngx_regex_free;
+#endif
 }
 
 
@@ -80,6 +97,7 @@ static ngx_inline void
 ngx_regex_malloc_init(ngx_pool_t *pool)
 {
     ngx_regex_pool = pool;
+    ngx_regex_direct_alloc = (pool == NULL) ? 1 : 0;
 }
 
 
@@ -87,9 +105,146 @@ static ngx_inline void
 ngx_regex_malloc_done(void)
 {
     ngx_regex_pool = NULL;
+    ngx_regex_direct_alloc = 0;
 }
 
 
+#if (NGX_PCRE2)
+
+ngx_int_t
+ngx_regex_compile(ngx_regex_compile_t *rc)
+{
+    int                     n, errcode;
+    char                   *p;
+    u_char                  errstr[128];
+    size_t                  erroff;
+    pcre2_code             *re;
+    ngx_regex_elt_t        *elt;
+    pcre2_general_context  *gctx;
+    pcre2_compile_context  *cctx;
+
+    if (ngx_regex_compile_context == NULL) {
+        /*
+         * Allocte a compile context if not yet allocated.  This uses
+         * direct allocations from heap, so the result can be cached
+         * even at runtime.
+         */
+
+        ngx_regex_malloc_init(NULL);
+
+        gctx = pcre2_general_context_create(ngx_regex_malloc, ngx_regex_free,
+                                            NULL);
+        if (gctx == NULL) {
+            ngx_regex_malloc_done();
+            goto nomem;
+        }
+
+        cctx = pcre2_compile_context_create(gctx);
+        if (cctx == NULL) {
+            pcre2_general_context_free(gctx);
+            ngx_regex_malloc_done();
+            goto nomem;
+        }
+
+        ngx_regex_compile_context = cctx;
+
+        pcre2_general_context_free(gctx);
+        ngx_regex_malloc_done();
+    }
+
+    ngx_regex_malloc_init(rc->pool);
+
+    re = pcre2_compile(rc->pattern.data, rc->pattern.len,
+                       (uint32_t) rc->options, &errcode, &erroff,
+                       ngx_regex_compile_context);
+
+    /* ensure that there is no current pool */
+    ngx_regex_malloc_done();
+
+    if (re == NULL) {
+        pcre2_get_error_message(errcode, errstr, 128);
+
+        if ((size_t) erroff == rc->pattern.len) {
+            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                              "pcre2_compile() failed: %s in \"%V\"",
+                               errstr, &rc->pattern)
+                          - rc->err.data;
+
+        } else {
+            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                              "pcre2_compile() failed: %s in \"%V\" at \"%s\"",
+                               errstr, &rc->pattern, rc->pattern.data + erroff)
+                          - rc->err.data;
+        }
+
+        return NGX_ERROR;
+    }
+
+    rc->regex = re;
+
+    /* do not study at runtime */
+
+    if (ngx_regex_studies != NULL) {
+        elt = ngx_list_push(ngx_regex_studies);
+        if (elt == NULL) {
+            goto nomem;
+        }
+
+        elt->regex = rc->regex;
+        elt->name = rc->pattern.data;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_CAPTURECOUNT, &rc->captures);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_CAPTURECOUNT) failed: %d";
+        goto failed;
+    }
+
+    if (rc->captures == 0) {
+        return NGX_OK;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_NAMECOUNT, &rc->named_captures);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMECOUNT) failed: %d";
+        goto failed;
+    }
+
+    if (rc->named_captures == 0) {
+        return NGX_OK;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_NAMEENTRYSIZE, &rc->name_size);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMEENTRYSIZE) failed: %d";
+        goto failed;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_NAMETABLE, &rc->names);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMETABLE) failed: %d";
+        goto failed;
+    }
+
+    return NGX_OK;
+
+failed:
+
+    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len, p, &rc->pattern, n)
+                  - rc->err.data;
+    return NGX_ERROR;
+
+nomem:
+
+    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                               "regex \"%V\" compilation failed: no memory",
+                               &rc->pattern)
+                  - rc->err.data;
+    return NGX_ERROR;
+}
+
+#else
+
 ngx_int_t
 ngx_regex_compile(ngx_regex_compile_t *rc)
 {
@@ -192,6 +347,74 @@ nomem:
     return NGX_ERROR;
 }
 
+#endif
+
+
+#if (NGX_PCRE2)
+
+ngx_int_t
+ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures, ngx_uint_t size)
+{
+    size_t      *ov;
+    ngx_int_t    rc;
+    ngx_uint_t   n, i;
+
+    /*
+     * The pcre2_match() function might allocate memory for backtracking
+     * frames, typical allocations are from 40k and above.  So the allocator
+     * is configured to do direct allocations from heap during matching.
+     */
+
+    ngx_regex_malloc_init(NULL);
+
+    if (ngx_regex_match_data == NULL
+        || size > ngx_regex_match_data_size)
+    {
+        /*
+         * Allocate a match data if not yet allocated or smaller than
+         * needed.
+         */
+
+        if (ngx_regex_match_data) {
+            pcre2_match_data_free(ngx_regex_match_data);
+        }
+
+        ngx_regex_match_data_size = size;
+        ngx_regex_match_data = pcre2_match_data_create(size / 3, NULL);
+
+        if (ngx_regex_match_data == NULL) {
+            rc = PCRE2_ERROR_NOMEMORY;
+            goto failed;
+        }
+    }
+
+    rc = pcre2_match(re, s->data, s->len, 0, 0, ngx_regex_match_data, NULL);
+
+    if (rc < 0) {
+        goto failed;
+    }
+
+    n = pcre2_get_ovector_count(ngx_regex_match_data);
+    ov = pcre2_get_ovector_pointer(ngx_regex_match_data);
+
+    if (n > size / 3) {
+        n = size / 3;
+    }
+
+    for (i = 0; i < n; i++) {
+        captures[i * 2] = ov[i * 2];
+        captures[i * 2 + 1] = ov[i * 2 + 1];
+    }
+
+failed:
+
+    ngx_regex_malloc_done();
+
+    return rc;
+}
+
+#endif
+
 
 ngx_int_t
 ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log)
@@ -226,6 +449,35 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
 }
 
 
+#if (NGX_PCRE2)
+
+static void * ngx_libc_cdecl
+ngx_regex_malloc(size_t size, void *data)
+{
+    if (ngx_regex_pool) {
+        return ngx_palloc(ngx_regex_pool, size);
+    }
+
+    if (ngx_regex_direct_alloc) {
+        return ngx_alloc(size, ngx_cycle->log);
+    }
+
+    return NULL;
+}
+
+
+static void ngx_libc_cdecl
+ngx_regex_free(void *p, void *data)
+{
+    if (ngx_regex_direct_alloc) {
+        ngx_free(p);
+    }
+
+    return;
+}
+
+#else
+
 static void * ngx_libc_cdecl
 ngx_regex_malloc(size_t size)
 {
@@ -243,11 +495,13 @@ ngx_regex_free(void *p)
     return;
 }
 
+#endif
+
 
 static void
 ngx_regex_cleanup(void *data)
 {
-#if (NGX_HAVE_PCRE_JIT)
+#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
     ngx_regex_conf_t *rcf = data;
 
     ngx_uint_t        i;
@@ -272,12 +526,17 @@ ngx_regex_cleanup(void *data)
         /*
          * The PCRE JIT compiler uses mmap for its executable codes, so we
          * have to explicitly call the pcre_free_study() function to free
-         * this memory.
+         * this memory.  In PCRE2, we call the pcre2_code_free() function
+         * for the same reason.
          */
 
+#if (NGX_PCRE2)
+        pcre2_code_free(elts[i].regex);
+#else
         if (elts[i].regex->extra != NULL) {
             pcre_free_study(elts[i].regex->extra);
         }
+#endif
     }
 #endif
 
@@ -287,6 +546,26 @@ ngx_regex_cleanup(void *data)
      */
 
     ngx_regex_studies = NULL;
+
+#if (NGX_PCRE2)
+
+    /*
+     * Free compile context and match data.  If needed at runtime by
+     * the new cycle, these will be re-allocated.
+     */
+
+    if (ngx_regex_compile_context) {
+        pcre2_compile_context_free(ngx_regex_compile_context);
+        ngx_regex_compile_context = NULL;
+    }
+
+    if (ngx_regex_match_data) {
+        pcre2_match_data_free(ngx_regex_match_data);
+        ngx_regex_match_data = NULL;
+        ngx_regex_match_data_size = 0;
+    }
+
+#endif
 }
 
 
@@ -294,7 +573,9 @@ static ngx_int_t
 ngx_regex_module_init(ngx_cycle_t *cycle)
 {
     int                opt;
+#if !(NGX_PCRE2)
     const char        *errstr;
+#endif
     ngx_uint_t         i;
     ngx_list_part_t   *part;
     ngx_regex_elt_t   *elts;
@@ -327,6 +608,23 @@ ngx_regex_module_init(ngx_cycle_t *cycle
             i = 0;
         }
 
+#if (NGX_PCRE2)
+
+        if (opt) {
+            int  n;
+
+            n = pcre2_jit_compile(elts[i].regex, PCRE2_JIT_COMPLETE);
+
+            if (n != 0) {
+                ngx_log_error(NGX_LOG_INFO, cycle->log, 0,
+                              "pcre2_jit_compile() failed: %d in \"%s\", "
+                              "ignored",
+                              n, elts[i].name);
+            }
+        }
+
+#else
+
         elts[i].regex->extra = pcre_study(elts[i].regex->code, opt, &errstr);
 
         if (errstr != NULL) {
@@ -350,11 +648,15 @@ ngx_regex_module_init(ngx_cycle_t *cycle
             }
         }
 #endif
+#endif
     }
 
     ngx_regex_malloc_done();
 
     ngx_regex_studies = NULL;
+#if (NGX_PCRE2)
+    ngx_regex_compile_context = NULL;
+#endif
 
     return NGX_OK;
 }
@@ -412,7 +714,21 @@ ngx_regex_pcre_jit(ngx_conf_t *cf, void 
         return NGX_CONF_OK;
     }
 
-#if (NGX_HAVE_PCRE_JIT)
+#if (NGX_PCRE2)
+    {
+    int       r;
+    uint32_t  jit;
+
+    jit = 0;
+    r = pcre2_config(PCRE2_CONFIG_JIT, &jit);
+
+    if (r != 0 || jit != 1) {
+        ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
+                           "PCRE2 library does not support JIT");
+        *fp = 0;
+    }
+    }
+#elif (NGX_HAVE_PCRE_JIT)
     {
     int  jit, r;
 
diff --git a/src/core/ngx_regex.h b/src/core/ngx_regex.h
--- a/src/core/ngx_regex.h
+++ b/src/core/ngx_regex.h
@@ -12,19 +12,31 @@
 #include 
 #include 
 
+
+#if (NGX_PCRE2)
+
+#define PCRE2_CODE_UNIT_WIDTH  8
+#include 
+
+#define NGX_REGEX_NO_MATCHED   PCRE2_ERROR_NOMATCH   /* -1 */
+#define NGX_REGEX_CASELESS     PCRE2_CASELESS
+
+typedef pcre2_code  ngx_regex_t;
+
+#else
+
 #include 
 
-
-#define NGX_REGEX_NO_MATCHED  PCRE_ERROR_NOMATCH   /* -1 */
-
-#define NGX_REGEX_CASELESS    PCRE_CASELESS
-
+#define NGX_REGEX_NO_MATCHED   PCRE_ERROR_NOMATCH   /* -1 */
+#define NGX_REGEX_CASELESS     PCRE_CASELESS
 
 typedef struct {
     pcre        *code;
     pcre_extra  *extra;
 } ngx_regex_t;
 
+#endif
+
 
 typedef struct {
     ngx_str_t     pattern;
@@ -49,10 +61,20 @@ typedef struct {
 void ngx_regex_init(void);
 ngx_int_t ngx_regex_compile(ngx_regex_compile_t *rc);
 
+#if (NGX_PCRE2)
+
+ngx_int_t ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures,
+    ngx_uint_t size);
+#define ngx_regex_exec_n       "pcre2_match()"
+
+#else
+
 #define ngx_regex_exec(re, s, captures, size)                                \
     pcre_exec(re->code, re->extra, (const char *) (s)->data, (s)->len, 0, 0, \
               captures, size)
-#define ngx_regex_exec_n      "pcre_exec()"
+#define ngx_regex_exec_n       "pcre_exec()"
+
+#endif
 
 ngx_int_t ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log);
 


From mdounin at mdounin.ru  Mon Dec 13 02:53:09 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 13 Dec 2021 05:53:09 +0300
Subject: [PATCH] on_no_ca parameter added to ssl_verify_client directive
In-Reply-To: 
References: 
Message-ID: 

Hello!

On Sun, Dec 12, 2021 at 03:51:50PM +0300, Evgenia Titova wrote:

> # HG changeset patch
> # User Evgenia Titova 
> # Date 1639309072 -10800
> #      Sun Dec 12 14:37:52 2021 +0300
> # Node ID f65a12913829b4032c390e16bafcefb7efdf27f4
> # Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
> on_no_ca parameter added to ssl_verify_client directive.
> 
> This parameter requires the client certificate but does not require it to be signed by a trusted CA certificate.
> 
> diff -r a7a77549265e -r f65a12913829 src/http/modules/ngx_http_ssl_module.c
> --- a/src/http/modules/ngx_http_ssl_module.c	Thu Nov 25 22:02:10 2021 +0300
> +++ b/src/http/modules/ngx_http_ssl_module.c	Sun Dec 12 14:37:52 2021 +0300
> @@ -70,6 +70,7 @@
>      { ngx_string("on"), 1 },
>      { ngx_string("optional"), 2 },
>      { ngx_string("optional_no_ca"), 3 },
> +    { ngx_string("on_no_ca"), 4 },
>      { ngx_null_string, 0 }
>  };
>  

Thanks you for the patch.

You may want to be more specific on how it is expected to be used 
and why existing options are not enough.  Quick look suggests that 
"optional_no_ca" with additional $ssl_client_verify and/or 
$ssl_client_cert tests should do the same.  Given that 
"optional_no_ca" anyway implies $ssl_client_cert verification by 
an external service, it is not clear why the new mode should be 
needed.

-- 
Maxim Dounin
http://mdounin.ru/

From arut at nginx.com  Mon Dec 13 12:03:58 2021
From: arut at nginx.com (Roman Arutyunyan)
Date: Mon, 13 Dec 2021 15:03:58 +0300
Subject: [PATCH 3 of 3] QUIC: stream recv shutdown support
In-Reply-To: 
References: 
 
 <20211126131133.ew64tszf5vpjxii7@Romans-MacBook-Pro.local>
 
Message-ID: <20211213120358.nwptjmsyczqakffu@Romans-MacBook-Pro.local>

On Fri, Dec 10, 2021 at 10:38:00AM +0300, Vladimir Homutov wrote:
> On Fri, Nov 26, 2021 at 04:11:33PM +0300, Roman Arutyunyan wrote:
> > On Thu, Nov 25, 2021 at 05:20:51PM +0300, Roman Arutyunyan wrote:
> > > # HG changeset patch
> > > # User Roman Arutyunyan 
> > > # Date 1637695967 -10800
> > > #      Tue Nov 23 22:32:47 2021 +0300
> > > # Branch quic
> > > # Node ID e1de02d829f7f85b1e2e6b289ec4c20318712321
> > > # Parent  3d2354bfa1a2a257b9f73772ad0836585be85a6c
> > > QUIC: stream recv shutdown support.
> > >
> > > Recv shutdown sends STOP_SENDING to client.  Both send and recv shutdown
> > > functions are now called from stream cleanup handler.  While here, setting
> > > c->read->pending_eof is moved down to fix recv shutdown in the cleanup handler.
> >
> > This definitely needs some improvement.  Now it's two patches.
> 
> I suggest merging both into one (also, second needs rebasing)

OK let's merge them.

> > [..]
> >
> > --
> > Roman Arutyunyan
> 
> > # HG changeset patch
> > # User Roman Arutyunyan 
> > # Date 1637931593 -10800
> > #      Fri Nov 26 15:59:53 2021 +0300
> > # Branch quic
> > # Node ID c2fa3e7689a4e286f45ccbac2288ade5966273b8
> > # Parent  3d2354bfa1a2a257b9f73772ad0836585be85a6c
> > QUIC: do not shutdown write part of a client uni stream.
> >
> > diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> > --- a/src/event/quic/ngx_event_quic_streams.c
> > +++ b/src/event/quic/ngx_event_quic_streams.c
> > @@ -267,13 +267,20 @@ ngx_quic_shutdown_stream(ngx_connection_
> >          return NGX_OK;
> >      }
> >
> > +    qs = c->quic;
> > +
> > +    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> > +        && (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL))
> > +    {
> > +        return NGX_OK;
> > +    }
> > +
> >      wev = c->write;
> >
> >      if (wev->error) {
> >          return NGX_OK;
> >      }
> >
> > -    qs = c->quic;
> >      pc = qs->parent;
> >      qc = ngx_quic_get_connection(pc);
> >
> 
> this one looks good
> 
> 
> > # HG changeset patch
> > # User Roman Arutyunyan 
> > # Date 1637932014 -10800
> > #      Fri Nov 26 16:06:54 2021 +0300
> > # Branch quic
> > # Node ID ed0cefd9fc434a7593f2f9e4b9a98ce65aaf05e9
> > # Parent  c2fa3e7689a4e286f45ccbac2288ade5966273b8
> > QUIC: write and full stream shutdown support.
> >
> > Full stream shutdown is now called from stream cleanup handler instead of
> > explicitly sending frames.  The call is moved up not to be influenced by
> > setting c->read->pending_eof, which was erroneously set too early.
> >
> > diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> > --- a/src/event/quic/ngx_event_quic_streams.c
> > +++ b/src/event/quic/ngx_event_quic_streams.c
> > @@ -13,6 +13,8 @@
> >  #define NGX_QUIC_STREAM_GONE     (void *) -1
> >
> >
> > +static ngx_int_t ngx_quic_shutdown_stream_send(ngx_connection_t *c);
> > +static ngx_int_t ngx_quic_shutdown_stream_recv(ngx_connection_t *c);
> >  static ngx_quic_stream_t *ngx_quic_get_stream(ngx_connection_t *c, uint64_t id);
> >  static ngx_int_t ngx_quic_reject_stream(ngx_connection_t *c, uint64_t id);
> >  static void ngx_quic_init_stream_handler(ngx_event_t *ev);
> > @@ -257,16 +259,31 @@ ngx_quic_reset_stream(ngx_connection_t *
> >  ngx_int_t
> >  ngx_quic_shutdown_stream(ngx_connection_t *c, int how)
> >  {
> > +    if (how == NGX_RW_SHUTDOWN || how == NGX_WRITE_SHUTDOWN) {
> > +        if (ngx_quic_shutdown_stream_send(c) != NGX_OK) {
> > +            return NGX_ERROR;
> > +        }
> > +    }
> > +
> > +    if (how == NGX_RW_SHUTDOWN || how == NGX_READ_SHUTDOWN) {
> > +        if (ngx_quic_shutdown_stream_recv(c) != NGX_OK) {
> > +            return NGX_ERROR;
> > +        }
> > +    }
> > +
> > +    return NGX_OK;
> > +}
> > +
> > +
> > +static ngx_int_t
> > +ngx_quic_shutdown_stream_send(ngx_connection_t *c)
> > +{
> >      ngx_event_t            *wev;
> >      ngx_connection_t       *pc;
> >      ngx_quic_frame_t       *frame;
> >      ngx_quic_stream_t      *qs;
> >      ngx_quic_connection_t  *qc;
> >
> > -    if (how != NGX_WRITE_SHUTDOWN) {
> > -        return NGX_OK;
> > -    }
> > -
> >      qs = c->quic;
> >
> >      if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> > @@ -290,7 +307,7 @@ ngx_quic_shutdown_stream(ngx_connection_
> >      }
> >
> >      ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > -                   "quic stream id:0x%xL shutdown", qs->id);
> > +                   "quic stream id:0x%xL send shutdown", qs->id);
> >
> >      frame->level = ssl_encryption_application;
> >      frame->type = NGX_QUIC_FT_STREAM;
> > @@ -311,6 +328,55 @@ ngx_quic_shutdown_stream(ngx_connection_
> >  }
> >
> >
> > +static ngx_int_t
> > +ngx_quic_shutdown_stream_recv(ngx_connection_t *c)
> > +{
> > +    ngx_event_t            *rev;
> > +    ngx_connection_t       *pc;
> > +    ngx_quic_frame_t       *frame;
> > +    ngx_quic_stream_t      *qs;
> > +    ngx_quic_connection_t  *qc;
> > +
> > +    qs = c->quic;
> > +
> > +    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED)
> > +        && (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL))
> > +    {
> 
> maybe it's worth trying to move server/client bidi/uni tests into
> ngx_quic_shutdown_stream() ? It looks like more natural place to
> test which end to shut, and whether we need to do it at all.

OK, we can do this.  I think these checks will be changed in the future anyway
when we introduce stream lingering.

> > +        return NGX_OK;
> > +    }
> > +
> > +    rev = c->read;
> > +
> > +    if (rev->pending_eof || rev->error) {
> > +        return NGX_OK;
> > +    }
> > +
> > +    pc = qs->parent;
> > +    qc = ngx_quic_get_connection(pc);
> > +
> > +    if (qc->conf->stream_close_code == 0) {
> > +        return NGX_OK;
> > +    }
> > +
> > +    frame = ngx_quic_alloc_frame(pc);
> > +    if (frame == NULL) {
> > +        return NGX_ERROR;
> > +    }
> > +
> > +    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > +                   "quic stream id:0x%xL recv shutdown", qs->id);
> > +
> > +    frame->level = ssl_encryption_application;
> > +    frame->type = NGX_QUIC_FT_STOP_SENDING;
> > +    frame->u.stop_sending.id = qs->id;
> > +    frame->u.stop_sending.error_code = qc->conf->stream_close_code;
> > +
> > +    ngx_quic_queue_frame(qc, frame);
> > +
> > +    return NGX_OK;
> > +}
> > +
> > +
> >  static ngx_quic_stream_t *
> >  ngx_quic_get_stream(ngx_connection_t *c, uint64_t id)
> >  {
> > @@ -925,30 +991,12 @@ ngx_quic_stream_cleanup_handler(void *da
> >          goto done;
> >      }
> >
> > +    (void) ngx_quic_shutdown_stream(c, NGX_RW_SHUTDOWN);
> > +
> >      c->read->pending_eof = 1;
> 
> I think we need to move setting pending_eof this into
> ngx_quic_shutdown_stream_recv():
> 
>  - the shutdown is supposed to be called only once
>  - this is the flag tested for this purpose
>  - this will be symmetrical with send that sets wev->error and tests it
>    on enter

I suggest that we use rev->error instead, which is even more symmetrical.
Also, I removed setting wev->ready, which makes no sense in the shutdown
function.

>  - it seems perfectly natural for recv shutdown to set some flag
>    prevents reading

rev->pending_eof does not prevent reading.  We can still read everything that's
buffered.  Also, I feel like messing with this flag will break its semantics,
so I'd rather leave it alone and use rev->error.

I think event flag checks will also be removed in the future and replaced
with stream send/recv states as specified by the RFC.  This itself is a good
change, but we'll also need this to implement stream lingering.

> or probalby this need to be done in ngx_quic_shutdown_stream(), as
> currently it is set without taking into account uni/bidi server/client
> difference.

The reason for setting rev->pending_eof was to skip updating stream flow
control in ngx_quic_update_flow().  The rev->error flag can do that too.
I removed setting rev->pending_eof from ngx_quic_stream_cleanup_handler().

> >      (void) ngx_quic_update_flow(c, qs->recv_last);
> >
> > -    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> > -        || (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)
> > -    {
> > -        if (!c->read->pending_eof && !c->read->error
> > -            && qc->conf->stream_close_code)
> > -        {
> > -            frame = ngx_quic_alloc_frame(pc);
> > -            if (frame == NULL) {
> > -                goto done;
> > -            }
> > -
> > -            frame->level = ssl_encryption_application;
> > -            frame->type = NGX_QUIC_FT_STOP_SENDING;
> > -            frame->u.stop_sending.id = qs->id;
> > -            frame->u.stop_sending.error_code = qc->conf->stream_close_code;
> > -
> > -            ngx_quic_queue_frame(qc, frame);
> > -        }
> > -    }
> > -
> >      if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0) {
> >          frame = ngx_quic_alloc_frame(pc);
> >          if (frame == NULL) {
> > @@ -968,37 +1016,8 @@ ngx_quic_stream_cleanup_handler(void *da
> >          }
> >
> >          ngx_quic_queue_frame(qc, frame);
> > -
> > -        if (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) {
> > -            /* do not send fin for client unidirectional streams */
> > -            goto done;
> > -        }
> >      }
> >
> > -    if (c->write->error) {
> > -        goto done;
> > -    }
> > -
> > -    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > -                   "quic stream id:0x%xL send fin", qs->id);
> > -
> > -    frame = ngx_quic_alloc_frame(pc);
> > -    if (frame == NULL) {
> > -        goto done;
> > -    }
> > -
> > -    frame->level = ssl_encryption_application;
> > -    frame->type = NGX_QUIC_FT_STREAM;
> > -    frame->u.stream.off = 1;
> > -    frame->u.stream.len = 1;
> > -    frame->u.stream.fin = 1;
> > -
> > -    frame->u.stream.stream_id = qs->id;
> > -    frame->u.stream.offset = c->sent;
> > -    frame->u.stream.length = 0;
> > -
> > -    ngx_quic_queue_frame(qc, frame);
> > -
> >  done:
> >
> >      (void) ngx_quic_output(pc);
> > diff --git a/src/os/unix/ngx_socket.h b/src/os/unix/ngx_socket.h
> > --- a/src/os/unix/ngx_socket.h
> > +++ b/src/os/unix/ngx_socket.h
> > @@ -13,6 +13,8 @@
> >
> >
> >  #define NGX_WRITE_SHUTDOWN SHUT_WR
> > +#define NGX_READ_SHUTDOWN  SHUT_RD
> > +#define NGX_RW_SHUTDOWN    SHUT_RDWR
> 
> I suggest renaming this to NGX_RDWR_SHUTDOWN  - 'RW' looks to much like
> 'WR' and makes confusion

OK, switched to RDWR.  Although, I don't like either name.  Since
NGX_WRITE_SHUTDOWN already exists, NGX_READ_SHUTDOWN is an obvious one, but
there's no good name for RDWR.  I thought about calling it NGX_FULL_SHUTDOWN.
Anyway, we can leave NGX_RDWR_SHUTDOWN for now and keep in mind a possible
rename in the future.

> >  typedef int  ngx_socket_t;
> >
> 
> > _______________________________________________
> > nginx-devel mailing list
> > nginx-devel at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> 
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

-- 
Roman Arutyunyan
-------------- next part --------------
# HG changeset patch
# User Roman Arutyunyan 
# Date 1639396182 -10800
#      Mon Dec 13 14:49:42 2021 +0300
# Branch quic
# Node ID 23880e4ad3e2986c189cf61d8409066f2b31590e
# Parent  0692355a3519024ed9b3a71a7216dcf6fe7e31ca
QUIC: write and full stream shutdown support.

Full stream shutdown is now called from stream cleanup handler instead of
explicitly sending frames.

diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
--- a/src/event/quic/ngx_event_quic_streams.c
+++ b/src/event/quic/ngx_event_quic_streams.c
@@ -13,6 +13,8 @@
 #define NGX_QUIC_STREAM_GONE     (void *) -1
 
 
+static ngx_int_t ngx_quic_shutdown_stream_send(ngx_connection_t *c);
+static ngx_int_t ngx_quic_shutdown_stream_recv(ngx_connection_t *c);
 static ngx_quic_stream_t *ngx_quic_get_stream(ngx_connection_t *c, uint64_t id);
 static ngx_int_t ngx_quic_reject_stream(ngx_connection_t *c, uint64_t id);
 static void ngx_quic_init_stream_handler(ngx_event_t *ev);
@@ -257,16 +259,43 @@ ngx_quic_reset_stream(ngx_connection_t *
 ngx_int_t
 ngx_quic_shutdown_stream(ngx_connection_t *c, int how)
 {
+    ngx_quic_stream_t  *qs;
+
+    qs = c->quic;
+
+    if (how == NGX_RDWR_SHUTDOWN || how == NGX_WRITE_SHUTDOWN) {
+        if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED)
+            || (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)
+        {
+            if (ngx_quic_shutdown_stream_send(c) != NGX_OK) {
+                return NGX_ERROR;
+            }
+        }
+    }
+
+    if (how == NGX_RDWR_SHUTDOWN || how == NGX_READ_SHUTDOWN) {
+        if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
+            || (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)
+        {
+            if (ngx_quic_shutdown_stream_recv(c) != NGX_OK) {
+                return NGX_ERROR;
+            }
+        }
+    }
+
+    return NGX_OK;
+}
+
+
+static ngx_int_t
+ngx_quic_shutdown_stream_send(ngx_connection_t *c)
+{
     ngx_event_t            *wev;
     ngx_connection_t       *pc;
     ngx_quic_frame_t       *frame;
     ngx_quic_stream_t      *qs;
     ngx_quic_connection_t  *qc;
 
-    if (how != NGX_WRITE_SHUTDOWN) {
-        return NGX_OK;
-    }
-
     wev = c->write;
 
     if (wev->error) {
@@ -283,7 +312,7 @@ ngx_quic_shutdown_stream(ngx_connection_
     }
 
     ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
-                   "quic stream id:0x%xL shutdown", qs->id);
+                   "quic stream id:0x%xL send shutdown", qs->id);
 
     frame->level = ssl_encryption_application;
     frame->type = NGX_QUIC_FT_STREAM;
@@ -297,13 +326,56 @@ ngx_quic_shutdown_stream(ngx_connection_
 
     ngx_quic_queue_frame(qc, frame);
 
-    wev->ready = 1;
     wev->error = 1;
 
     return NGX_OK;
 }
 
 
+static ngx_int_t
+ngx_quic_shutdown_stream_recv(ngx_connection_t *c)
+{
+    ngx_event_t            *rev;
+    ngx_connection_t       *pc;
+    ngx_quic_frame_t       *frame;
+    ngx_quic_stream_t      *qs;
+    ngx_quic_connection_t  *qc;
+
+    rev = c->read;
+
+    if (rev->pending_eof || rev->error) {
+        return NGX_OK;
+    }
+
+    qs = c->quic;
+    pc = qs->parent;
+    qc = ngx_quic_get_connection(pc);
+
+    if (qc->conf->stream_close_code == 0) {
+        return NGX_OK;
+    }
+
+    frame = ngx_quic_alloc_frame(pc);
+    if (frame == NULL) {
+        return NGX_ERROR;
+    }
+
+    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
+                   "quic stream id:0x%xL recv shutdown", qs->id);
+
+    frame->level = ssl_encryption_application;
+    frame->type = NGX_QUIC_FT_STOP_SENDING;
+    frame->u.stop_sending.id = qs->id;
+    frame->u.stop_sending.error_code = qc->conf->stream_close_code;
+
+    ngx_quic_queue_frame(qc, frame);
+
+    rev->error = 1;
+
+    return NGX_OK;
+}
+
+
 static ngx_quic_stream_t *
 ngx_quic_get_stream(ngx_connection_t *c, uint64_t id)
 {
@@ -916,30 +988,10 @@ ngx_quic_stream_cleanup_handler(void *da
         goto done;
     }
 
-    c->read->pending_eof = 1;
+    (void) ngx_quic_shutdown_stream(c, NGX_RDWR_SHUTDOWN);
 
     (void) ngx_quic_update_flow(c, qs->recv_last);
 
-    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
-        || (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)
-    {
-        if (!c->read->pending_eof && !c->read->error
-            && qc->conf->stream_close_code)
-        {
-            frame = ngx_quic_alloc_frame(pc);
-            if (frame == NULL) {
-                goto done;
-            }
-
-            frame->level = ssl_encryption_application;
-            frame->type = NGX_QUIC_FT_STOP_SENDING;
-            frame->u.stop_sending.id = qs->id;
-            frame->u.stop_sending.error_code = qc->conf->stream_close_code;
-
-            ngx_quic_queue_frame(qc, frame);
-        }
-    }
-
     if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0) {
         frame = ngx_quic_alloc_frame(pc);
         if (frame == NULL) {
@@ -959,37 +1011,8 @@ ngx_quic_stream_cleanup_handler(void *da
         }
 
         ngx_quic_queue_frame(qc, frame);
-
-        if (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) {
-            /* do not send fin for client unidirectional streams */
-            goto done;
-        }
     }
 
-    if (c->write->error) {
-        goto done;
-    }
-
-    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
-                   "quic stream id:0x%xL send fin", qs->id);
-
-    frame = ngx_quic_alloc_frame(pc);
-    if (frame == NULL) {
-        goto done;
-    }
-
-    frame->level = ssl_encryption_application;
-    frame->type = NGX_QUIC_FT_STREAM;
-    frame->u.stream.off = 1;
-    frame->u.stream.len = 1;
-    frame->u.stream.fin = 1;
-
-    frame->u.stream.stream_id = qs->id;
-    frame->u.stream.offset = c->sent;
-    frame->u.stream.length = 0;
-
-    ngx_quic_queue_frame(qc, frame);
-
 done:
 
     (void) ngx_quic_output(pc);
diff --git a/src/os/unix/ngx_socket.h b/src/os/unix/ngx_socket.h
--- a/src/os/unix/ngx_socket.h
+++ b/src/os/unix/ngx_socket.h
@@ -13,6 +13,8 @@
 
 
 #define NGX_WRITE_SHUTDOWN SHUT_WR
+#define NGX_READ_SHUTDOWN  SHUT_RD
+#define NGX_RDWR_SHUTDOWN  SHUT_RDWR
 
 typedef int  ngx_socket_t;
 

From arut at nginx.com  Mon Dec 13 12:15:30 2021
From: arut at nginx.com (Roman Arutyunyan)
Date: Mon, 13 Dec 2021 15:15:30 +0300
Subject: [PATCH 1 of 3] QUIC: post stream events instead of calling their
 handlers
In-Reply-To: <4A4C02F0-D125-474C-B4E7-E3A6F1BD0144@nginx.com>
References: 
 <5b03ffd757804542daec.1637850049@arut-laptop>
 <4A4C02F0-D125-474C-B4E7-E3A6F1BD0144@nginx.com>
Message-ID: <20211213121530.ndcjmkt4cjtl6wlx@Romans-MacBook-Pro.local>

On Mon, Dec 06, 2021 at 03:37:28PM +0300, Sergey Kandaurov wrote:
> 
> > On 25 Nov 2021, at 17:20, Roman Arutyunyan  wrote:
> > 
> > # HG changeset patch
> > # User Roman Arutyunyan 
> > # Date 1637692791 -10800
> > #      Tue Nov 23 21:39:51 2021 +0300
> > # Branch quic
> > # Node ID 5b03ffd757804542daec73188a509b02e6b2c596
> > # Parent  d041b8d6ab0b2dea150536531345fa47c696b303
> > QUIC: post stream events instead of calling their handlers.
> > 
> > This potentially reduces the number of handler calls.
> > 
> > diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> > --- a/src/event/quic/ngx_event_quic_streams.c
> > +++ b/src/event/quic/ngx_event_quic_streams.c
> > @@ -1122,7 +1122,7 @@ ngx_quic_handle_stream_frame(ngx_connect
> >         rev->ready = 1;
> > 
> >         if (rev->active) {
> > -            rev->handler(rev);
> > +            ngx_post_event(rev, &ngx_posted_events);
> >         }
> >     }
> > 
> > @@ -1369,7 +1369,7 @@ ngx_quic_handle_reset_stream_frame(ngx_c
> >     }
> > 
> >     if (rev->active) {
> > -        rev->handler(rev);
> > +        ngx_post_event(rev, &ngx_posted_events);
> >     }
> > 
> >     return NGX_OK;
> > @@ -1438,7 +1438,7 @@ ngx_quic_handle_stop_sending_frame(ngx_c
> >     wev = qs->connection->write;
> > 
> >     if (wev->active) {
> > -        wev->handler(wev);
> > +        ngx_post_event(wev, &ngx_posted_events);
> >     }
> > 
> >     return NGX_OK;
> 
> I don't like this change.
> 
> While in practice this indeed tends to reduce the number of handler
> invocations in certain edge cases, it also entails negative effects.
> First, postponing processing the next part of a stream means that
> if the stream spans multiple packets, its handling will be deferred
> inadvertently until after the next packet is finished processing.
> Next, if a stream is the encoder stream, used to send new dynamic
> entries, then new request streams will be inevitably blocked.
> I've run intentionally on this to see what happens:
> looks like the blocked stream isn't woken up until connection close
> (which consumes active streams in ngx_quic_close_streams()).

Indeed, the change adds some reordering.  Stream listen handler is always
called right after stream creation, but read handler is now postponed.
The change alters the order of these calls when multiple streams are involved.
This is not a problem per se since reordering can happen naturally as well.
Also, the next patch in series introduces listen handler call postpone, which
brings back the old ordering.

The problem discovered is triggered by another (not yet committed) change
which apparently breaks stream unblock.  That change obviously needs some
improvement.

> 0x6 is the encoder stream, 0x40 is the request stream.
> 
> 2021/12/06 14:45:36 [debug] 17379#0: *1 quic frame rx app STREAM id:0x6 off:1 len:428
> 2021/12/06 14:45:36 [debug] 17379#0: *4 post event 000062F000000740
> 2021/12/06 14:45:36 [debug] 17379#0: *1 quic frame rx app STREAM id:0x40 len:38 fin:1
> ...
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 check insert count req:23, have:0
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 block stream
> ...
> 2021/12/06 14:45:36 [debug] 17379#0: posted event 000062F000000740
> 2021/12/06 14:45:36 [debug] 17379#0: *4 delete posted event 000062F000000740
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 read handler
> 2021/12/06 14:45:36 [debug] 17379#0: *4 quic stream id:0x6 recv eof:0 buf:128
> 2021/12/06 14:45:36 [debug] 17379#0: *4 quic stream id:0x6 recv len:128
> 2021/12/06 14:45:36 [debug] 17379#0: *4 quic flow update 129
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse encoder instruction
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse prefix int 16384
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 set capacity 16384
> 2021/12/06 14:45:36 [debug] 17379#0: *1 malloc: 000062100005DD00:4096
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse encoder instruction
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse field inr
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse prefix int 0
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse prefix int 6
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse literal huff:1, len:6
> 2021/12/06 14:45:36 [debug] 17379#0: *4 malloc: 00006020000A5630:10
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse literal done "localhost"
> 2021/12/06 14:45:36 [debug] 17379#0: *4 http3 parse field inr done static[0] "localhost"
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 ref insert static[0] "localhost"
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 static[0] lookup ":authority":""
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 insert [0] ":authority":"localhost", size:51
> 2021/12/06 14:45:36 [debug] 17379#0: *1 malloc: 0000606000008D80:51
> 2021/12/06 14:45:36 [debug] 17379#0: *1 post event 0000614000005A78
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 new dynamic entry, blocked:1
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 unblock stream
> 2021/12/06 14:45:36 [debug] 17379#0: *1 post event 000062F000000608
> ...
> 2021/12/06 14:45:36 [debug] 17379#0: posted event 0000614000005A78
> 2021/12/06 14:45:36 [debug] 17379#0: *1 delete posted event 0000614000005A78
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 inc insert count handler
> 2021/12/06 14:45:36 [debug] 17379#0: *1 http3 send insert count increment 23
> ...
> 2021/12/06 14:45:36 [debug] 17379#0: posted event 000062F000000608
> 2021/12/06 14:45:36 [debug] 17379#0: *1 delete posted event 000062F000000608
> 2021/12/06 14:45:36 [debug] 17379#0: *1 quic input handler
> [nothing]
> 2021/12/06 14:45:36 [debug] 17379#0: posted event 000061D00003C648
> ...
> ...
> 2021/12/06 14:45:37 [debug] 17379#0: *1 quic frame rx app CONNECTION_CLOSE err:0 ft:0
> 2021/12/06 14:45:37 [debug] 17379#0: *1 quic immediate close drain:1
> 2021/12/06 14:45:37 [debug] 17379#0: *22 post event 000062F000000A18
> ...
> 2021/12/06 14:45:37 [debug] 17379#0: posted event 000062F000000A18
> 2021/12/06 14:45:37 [debug] 17379#0: *22 delete posted event 000062F000000A18
> 2021/12/06 14:45:37 [debug] 17379#0: *22 http3 process request
> 2021/12/06 14:45:37 [debug] 17379#0: *1 http3 check insert count req:23, have:23
> ...
> 2021/12/06 14:45:37 [debug] 17379#0: *22 http3 parse field representation done
> 2021/12/06 14:45:37 [debug] 17379#0: *22 http3 parse headers done
> 2021/12/06 14:45:37 [debug] 17379#0: *1 http3 send section acknowledgement 64
> 2021/12/06 14:45:37 [debug] 17379#0: *1 quic creating server uni stream streams:2 max:100 id:0xb
> 2021/12/06 14:45:37 [debug] 17379#0: *1 quic stream id:0xb create
> ...
> 2021/12/06 14:45:37 [debug] 17379#0: *22 http3 header: "user-agent: something something"
> 2021/12/06 14:45:37 [debug] 17379#0: *22 http request count:1 blk:0
> 2021/12/06 14:45:37 [debug] 17379#0: *22 http close request
> 2021/12/06 14:45:37 [debug] 17379#0: *22 http log handler
> 2021/12/06 14:45:37 [debug] 17379#0: *22 malloc: 000060D000007AF0:130
> 2021/12/06 14:45:37 [debug] 17379#0: *22 malloc: 000060F000019EA0:169
> 2021/12/06 14:45:37 [debug] 17379#0: *22 run cleanup: 000060300007D500
> 2021/12/06 14:45:37 [debug] 17379#0: *1 event timer add: 8: 75000:2579974395
> 2021/12/06 14:45:37 [debug] 17379#0: *22 close http connection: 8
> 2021/12/06 14:45:37 [debug] 17379#0: *1 http3 send stream cancellation 64
> 
> -- 
> Sergey Kandaurov
> 
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

-- 
Roman Arutyunyan

From vl at nginx.com  Mon Dec 13 13:24:22 2021
From: vl at nginx.com (Vladimir Homutov)
Date: Mon, 13 Dec 2021 16:24:22 +0300
Subject: [PATCH 3 of 3] QUIC: stream recv shutdown support
In-Reply-To: <20211213120358.nwptjmsyczqakffu@Romans-MacBook-Pro.local>
References: 
 
 <20211126131133.ew64tszf5vpjxii7@Romans-MacBook-Pro.local>
 
 <20211213120358.nwptjmsyczqakffu@Romans-MacBook-Pro.local>
Message-ID: 

On Mon, Dec 13, 2021 at 03:03:58PM +0300, Roman Arutyunyan wrote:
> On Fri, Dec 10, 2021 at 10:38:00AM +0300, Vladimir Homutov wrote:
> > On Fri, Nov 26, 2021 at 04:11:33PM +0300, Roman Arutyunyan wrote:
> > > On Thu, Nov 25, 2021 at 05:20:51PM +0300, Roman Arutyunyan wrote:
> > > > # HG changeset patch
> > > > # User Roman Arutyunyan 
> > > > # Date 1637695967 -10800
> > > > #      Tue Nov 23 22:32:47 2021 +0300
> > > > # Branch quic
> > > > # Node ID e1de02d829f7f85b1e2e6b289ec4c20318712321
> > > > # Parent  3d2354bfa1a2a257b9f73772ad0836585be85a6c
> > > > QUIC: stream recv shutdown support.
> > > >
> > > > Recv shutdown sends STOP_SENDING to client.  Both send and recv shutdown
> > > > functions are now called from stream cleanup handler.  While here, setting
> > > > c->read->pending_eof is moved down to fix recv shutdown in the cleanup handler.
> > >
> > > This definitely needs some improvement.  Now it's two patches.
> >
> > I suggest merging both into one (also, second needs rebasing)
>
> OK let's merge them.
>
> > > [..]
> > >
> > > --
> > > Roman Arutyunyan
> >
> > > # HG changeset patch
> > > # User Roman Arutyunyan 
> > > # Date 1637931593 -10800
> > > #      Fri Nov 26 15:59:53 2021 +0300
> > > # Branch quic
> > > # Node ID c2fa3e7689a4e286f45ccbac2288ade5966273b8
> > > # Parent  3d2354bfa1a2a257b9f73772ad0836585be85a6c
> > > QUIC: do not shutdown write part of a client uni stream.
> > >
> > > diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> > > --- a/src/event/quic/ngx_event_quic_streams.c
> > > +++ b/src/event/quic/ngx_event_quic_streams.c
> > > @@ -267,13 +267,20 @@ ngx_quic_shutdown_stream(ngx_connection_
> > >          return NGX_OK;
> > >      }
> > >
> > > +    qs = c->quic;
> > > +
> > > +    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> > > +        && (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL))
> > > +    {
> > > +        return NGX_OK;
> > > +    }
> > > +
> > >      wev = c->write;
> > >
> > >      if (wev->error) {
> > >          return NGX_OK;
> > >      }
> > >
> > > -    qs = c->quic;
> > >      pc = qs->parent;
> > >      qc = ngx_quic_get_connection(pc);
> > >
> >
> > this one looks good
> >
> >
> > > # HG changeset patch
> > > # User Roman Arutyunyan 
> > > # Date 1637932014 -10800
> > > #      Fri Nov 26 16:06:54 2021 +0300
> > > # Branch quic
> > > # Node ID ed0cefd9fc434a7593f2f9e4b9a98ce65aaf05e9
> > > # Parent  c2fa3e7689a4e286f45ccbac2288ade5966273b8
> > > QUIC: write and full stream shutdown support.
> > >
> > > Full stream shutdown is now called from stream cleanup handler instead of
> > > explicitly sending frames.  The call is moved up not to be influenced by
> > > setting c->read->pending_eof, which was erroneously set too early.
> > >
> > > diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> > > --- a/src/event/quic/ngx_event_quic_streams.c
> > > +++ b/src/event/quic/ngx_event_quic_streams.c
> > > @@ -13,6 +13,8 @@
> > >  #define NGX_QUIC_STREAM_GONE     (void *) -1
> > >
> > >
> > > +static ngx_int_t ngx_quic_shutdown_stream_send(ngx_connection_t *c);
> > > +static ngx_int_t ngx_quic_shutdown_stream_recv(ngx_connection_t *c);
> > >  static ngx_quic_stream_t *ngx_quic_get_stream(ngx_connection_t *c, uint64_t id);
> > >  static ngx_int_t ngx_quic_reject_stream(ngx_connection_t *c, uint64_t id);
> > >  static void ngx_quic_init_stream_handler(ngx_event_t *ev);
> > > @@ -257,16 +259,31 @@ ngx_quic_reset_stream(ngx_connection_t *
> > >  ngx_int_t
> > >  ngx_quic_shutdown_stream(ngx_connection_t *c, int how)
> > >  {
> > > +    if (how == NGX_RW_SHUTDOWN || how == NGX_WRITE_SHUTDOWN) {
> > > +        if (ngx_quic_shutdown_stream_send(c) != NGX_OK) {
> > > +            return NGX_ERROR;
> > > +        }
> > > +    }
> > > +
> > > +    if (how == NGX_RW_SHUTDOWN || how == NGX_READ_SHUTDOWN) {
> > > +        if (ngx_quic_shutdown_stream_recv(c) != NGX_OK) {
> > > +            return NGX_ERROR;
> > > +        }
> > > +    }
> > > +
> > > +    return NGX_OK;
> > > +}
> > > +
> > > +
> > > +static ngx_int_t
> > > +ngx_quic_shutdown_stream_send(ngx_connection_t *c)
> > > +{
> > >      ngx_event_t            *wev;
> > >      ngx_connection_t       *pc;
> > >      ngx_quic_frame_t       *frame;
> > >      ngx_quic_stream_t      *qs;
> > >      ngx_quic_connection_t  *qc;
> > >
> > > -    if (how != NGX_WRITE_SHUTDOWN) {
> > > -        return NGX_OK;
> > > -    }
> > > -
> > >      qs = c->quic;
> > >
> > >      if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> > > @@ -290,7 +307,7 @@ ngx_quic_shutdown_stream(ngx_connection_
> > >      }
> > >
> > >      ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > > -                   "quic stream id:0x%xL shutdown", qs->id);
> > > +                   "quic stream id:0x%xL send shutdown", qs->id);
> > >
> > >      frame->level = ssl_encryption_application;
> > >      frame->type = NGX_QUIC_FT_STREAM;
> > > @@ -311,6 +328,55 @@ ngx_quic_shutdown_stream(ngx_connection_
> > >  }
> > >
> > >
> > > +static ngx_int_t
> > > +ngx_quic_shutdown_stream_recv(ngx_connection_t *c)
> > > +{
> > > +    ngx_event_t            *rev;
> > > +    ngx_connection_t       *pc;
> > > +    ngx_quic_frame_t       *frame;
> > > +    ngx_quic_stream_t      *qs;
> > > +    ngx_quic_connection_t  *qc;
> > > +
> > > +    qs = c->quic;
> > > +
> > > +    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED)
> > > +        && (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL))
> > > +    {
> >
> > maybe it's worth trying to move server/client bidi/uni tests into
> > ngx_quic_shutdown_stream() ? It looks like more natural place to
> > test which end to shut, and whether we need to do it at all.
>
> OK, we can do this.  I think these checks will be changed in the future anyway
> when we introduce stream lingering.
>
> > > +        return NGX_OK;
> > > +    }
> > > +
> > > +    rev = c->read;
> > > +
> > > +    if (rev->pending_eof || rev->error) {
> > > +        return NGX_OK;
> > > +    }
> > > +
> > > +    pc = qs->parent;
> > > +    qc = ngx_quic_get_connection(pc);
> > > +
> > > +    if (qc->conf->stream_close_code == 0) {
> > > +        return NGX_OK;
> > > +    }
> > > +
> > > +    frame = ngx_quic_alloc_frame(pc);
> > > +    if (frame == NULL) {
> > > +        return NGX_ERROR;
> > > +    }
> > > +
> > > +    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > > +                   "quic stream id:0x%xL recv shutdown", qs->id);
> > > +
> > > +    frame->level = ssl_encryption_application;
> > > +    frame->type = NGX_QUIC_FT_STOP_SENDING;
> > > +    frame->u.stop_sending.id = qs->id;
> > > +    frame->u.stop_sending.error_code = qc->conf->stream_close_code;
> > > +
> > > +    ngx_quic_queue_frame(qc, frame);
> > > +
> > > +    return NGX_OK;
> > > +}
> > > +
> > > +
> > >  static ngx_quic_stream_t *
> > >  ngx_quic_get_stream(ngx_connection_t *c, uint64_t id)
> > >  {
> > > @@ -925,30 +991,12 @@ ngx_quic_stream_cleanup_handler(void *da
> > >          goto done;
> > >      }
> > >
> > > +    (void) ngx_quic_shutdown_stream(c, NGX_RW_SHUTDOWN);
> > > +
> > >      c->read->pending_eof = 1;
> >
> > I think we need to move setting pending_eof this into
> > ngx_quic_shutdown_stream_recv():
> >
> >  - the shutdown is supposed to be called only once
> >  - this is the flag tested for this purpose
> >  - this will be symmetrical with send that sets wev->error and tests it
> >    on enter
>
> I suggest that we use rev->error instead, which is even more symmetrical.
> Also, I removed setting wev->ready, which makes no sense in the shutdown
> function.
>
> >  - it seems perfectly natural for recv shutdown to set some flag
> >    prevents reading
>
> rev->pending_eof does not prevent reading.  We can still read everything that's
> buffered.  Also, I feel like messing with this flag will break its semantics,
> so I'd rather leave it alone and use rev->error.
>
> I think event flag checks will also be removed in the future and replaced
> with stream send/recv states as specified by the RFC.  This itself is a good
> change, but we'll also need this to implement stream lingering.
>
> > or probalby this need to be done in ngx_quic_shutdown_stream(), as
> > currently it is set without taking into account uni/bidi server/client
> > difference.
>
> The reason for setting rev->pending_eof was to skip updating stream flow
> control in ngx_quic_update_flow().  The rev->error flag can do that too.
> I removed setting rev->pending_eof from ngx_quic_stream_cleanup_handler().
>
> > >      (void) ngx_quic_update_flow(c, qs->recv_last);
> > >
> > > -    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> > > -        || (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)
> > > -    {
> > > -        if (!c->read->pending_eof && !c->read->error
> > > -            && qc->conf->stream_close_code)
> > > -        {
> > > -            frame = ngx_quic_alloc_frame(pc);
> > > -            if (frame == NULL) {
> > > -                goto done;
> > > -            }
> > > -
> > > -            frame->level = ssl_encryption_application;
> > > -            frame->type = NGX_QUIC_FT_STOP_SENDING;
> > > -            frame->u.stop_sending.id = qs->id;
> > > -            frame->u.stop_sending.error_code = qc->conf->stream_close_code;
> > > -
> > > -            ngx_quic_queue_frame(qc, frame);
> > > -        }
> > > -    }
> > > -
> > >      if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0) {
> > >          frame = ngx_quic_alloc_frame(pc);
> > >          if (frame == NULL) {
> > > @@ -968,37 +1016,8 @@ ngx_quic_stream_cleanup_handler(void *da
> > >          }
> > >
> > >          ngx_quic_queue_frame(qc, frame);
> > > -
> > > -        if (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) {
> > > -            /* do not send fin for client unidirectional streams */
> > > -            goto done;
> > > -        }
> > >      }
> > >
> > > -    if (c->write->error) {
> > > -        goto done;
> > > -    }
> > > -
> > > -    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > > -                   "quic stream id:0x%xL send fin", qs->id);
> > > -
> > > -    frame = ngx_quic_alloc_frame(pc);
> > > -    if (frame == NULL) {
> > > -        goto done;
> > > -    }
> > > -
> > > -    frame->level = ssl_encryption_application;
> > > -    frame->type = NGX_QUIC_FT_STREAM;
> > > -    frame->u.stream.off = 1;
> > > -    frame->u.stream.len = 1;
> > > -    frame->u.stream.fin = 1;
> > > -
> > > -    frame->u.stream.stream_id = qs->id;
> > > -    frame->u.stream.offset = c->sent;
> > > -    frame->u.stream.length = 0;
> > > -
> > > -    ngx_quic_queue_frame(qc, frame);
> > > -
> > >  done:
> > >
> > >      (void) ngx_quic_output(pc);
> > > diff --git a/src/os/unix/ngx_socket.h b/src/os/unix/ngx_socket.h
> > > --- a/src/os/unix/ngx_socket.h
> > > +++ b/src/os/unix/ngx_socket.h
> > > @@ -13,6 +13,8 @@
> > >
> > >
> > >  #define NGX_WRITE_SHUTDOWN SHUT_WR
> > > +#define NGX_READ_SHUTDOWN  SHUT_RD
> > > +#define NGX_RW_SHUTDOWN    SHUT_RDWR
> >
> > I suggest renaming this to NGX_RDWR_SHUTDOWN  - 'RW' looks to much like
> > 'WR' and makes confusion
>
> OK, switched to RDWR.  Although, I don't like either name.  Since
> NGX_WRITE_SHUTDOWN already exists, NGX_READ_SHUTDOWN is an obvious one, but
> there's no good name for RDWR.  I thought about calling it NGX_FULL_SHUTDOWN.
> Anyway, we can leave NGX_RDWR_SHUTDOWN for now and keep in mind a possible
> rename in the future.


> # HG changeset patch
> # User Roman Arutyunyan 
> # Date 1639396182 -10800
> #      Mon Dec 13 14:49:42 2021 +0300
> # Branch quic
> # Node ID 23880e4ad3e2986c189cf61d8409066f2b31590e
> # Parent  0692355a3519024ed9b3a71a7216dcf6fe7e31ca
> QUIC: write and full stream shutdown support.
>
> Full stream shutdown is now called from stream cleanup handler instead of
> explicitly sending frames.
>
> diff --git a/src/event/quic/ngx_event_quic_streams.c b/src/event/quic/ngx_event_quic_streams.c
> --- a/src/event/quic/ngx_event_quic_streams.c
> +++ b/src/event/quic/ngx_event_quic_streams.c
> @@ -13,6 +13,8 @@
>  #define NGX_QUIC_STREAM_GONE     (void *) -1
>
>
> +static ngx_int_t ngx_quic_shutdown_stream_send(ngx_connection_t *c);
> +static ngx_int_t ngx_quic_shutdown_stream_recv(ngx_connection_t *c);
>  static ngx_quic_stream_t *ngx_quic_get_stream(ngx_connection_t *c, uint64_t id);
>  static ngx_int_t ngx_quic_reject_stream(ngx_connection_t *c, uint64_t id);
>  static void ngx_quic_init_stream_handler(ngx_event_t *ev);
> @@ -257,16 +259,43 @@ ngx_quic_reset_stream(ngx_connection_t *
>  ngx_int_t
>  ngx_quic_shutdown_stream(ngx_connection_t *c, int how)
>  {
> +    ngx_quic_stream_t  *qs;
> +
> +    qs = c->quic;
> +
> +    if (how == NGX_RDWR_SHUTDOWN || how == NGX_WRITE_SHUTDOWN) {
> +        if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED)
> +            || (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)
> +        {
> +            if (ngx_quic_shutdown_stream_send(c) != NGX_OK) {
> +                return NGX_ERROR;
> +            }
> +        }
> +    }
> +
> +    if (how == NGX_RDWR_SHUTDOWN || how == NGX_READ_SHUTDOWN) {
> +        if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> +            || (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)
> +        {
> +            if (ngx_quic_shutdown_stream_recv(c) != NGX_OK) {
> +                return NGX_ERROR;
> +            }
> +        }
> +    }
> +
> +    return NGX_OK;
> +}
> +
> +
> +static ngx_int_t
> +ngx_quic_shutdown_stream_send(ngx_connection_t *c)
> +{
>      ngx_event_t            *wev;
>      ngx_connection_t       *pc;
>      ngx_quic_frame_t       *frame;
>      ngx_quic_stream_t      *qs;
>      ngx_quic_connection_t  *qc;
>
> -    if (how != NGX_WRITE_SHUTDOWN) {
> -        return NGX_OK;
> -    }
> -
>      wev = c->write;
>
>      if (wev->error) {
> @@ -283,7 +312,7 @@ ngx_quic_shutdown_stream(ngx_connection_
>      }
>
>      ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> -                   "quic stream id:0x%xL shutdown", qs->id);
> +                   "quic stream id:0x%xL send shutdown", qs->id);
>
>      frame->level = ssl_encryption_application;
>      frame->type = NGX_QUIC_FT_STREAM;
> @@ -297,13 +326,56 @@ ngx_quic_shutdown_stream(ngx_connection_
>
>      ngx_quic_queue_frame(qc, frame);
>
> -    wev->ready = 1;
>      wev->error = 1;
>
>      return NGX_OK;
>  }
>
>
> +static ngx_int_t
> +ngx_quic_shutdown_stream_recv(ngx_connection_t *c)
> +{
> +    ngx_event_t            *rev;
> +    ngx_connection_t       *pc;
> +    ngx_quic_frame_t       *frame;
> +    ngx_quic_stream_t      *qs;
> +    ngx_quic_connection_t  *qc;
> +
> +    rev = c->read;
> +
> +    if (rev->pending_eof || rev->error) {
> +        return NGX_OK;
> +    }
> +
> +    qs = c->quic;
> +    pc = qs->parent;
> +    qc = ngx_quic_get_connection(pc);
> +
> +    if (qc->conf->stream_close_code == 0) {
> +        return NGX_OK;
> +    }
> +
> +    frame = ngx_quic_alloc_frame(pc);
> +    if (frame == NULL) {
> +        return NGX_ERROR;
> +    }
> +
> +    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> +                   "quic stream id:0x%xL recv shutdown", qs->id);
> +
> +    frame->level = ssl_encryption_application;
> +    frame->type = NGX_QUIC_FT_STOP_SENDING;
> +    frame->u.stop_sending.id = qs->id;
> +    frame->u.stop_sending.error_code = qc->conf->stream_close_code;
> +
> +    ngx_quic_queue_frame(qc, frame);
> +
> +    rev->error = 1;
> +
> +    return NGX_OK;
> +}
> +
> +
>  static ngx_quic_stream_t *
>  ngx_quic_get_stream(ngx_connection_t *c, uint64_t id)
>  {
> @@ -916,30 +988,10 @@ ngx_quic_stream_cleanup_handler(void *da
>          goto done;
>      }
>
> -    c->read->pending_eof = 1;
> +    (void) ngx_quic_shutdown_stream(c, NGX_RDWR_SHUTDOWN);
>
>      (void) ngx_quic_update_flow(c, qs->recv_last);
>
> -    if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0
> -        || (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) == 0)
> -    {
> -        if (!c->read->pending_eof && !c->read->error
> -            && qc->conf->stream_close_code)
> -        {
> -            frame = ngx_quic_alloc_frame(pc);
> -            if (frame == NULL) {
> -                goto done;
> -            }
> -
> -            frame->level = ssl_encryption_application;
> -            frame->type = NGX_QUIC_FT_STOP_SENDING;
> -            frame->u.stop_sending.id = qs->id;
> -            frame->u.stop_sending.error_code = qc->conf->stream_close_code;
> -
> -            ngx_quic_queue_frame(qc, frame);
> -        }
> -    }
> -
>      if ((qs->id & NGX_QUIC_STREAM_SERVER_INITIATED) == 0) {
>          frame = ngx_quic_alloc_frame(pc);
>          if (frame == NULL) {
> @@ -959,37 +1011,8 @@ ngx_quic_stream_cleanup_handler(void *da
>          }
>
>          ngx_quic_queue_frame(qc, frame);
> -
> -        if (qs->id & NGX_QUIC_STREAM_UNIDIRECTIONAL) {
> -            /* do not send fin for client unidirectional streams */
> -            goto done;
> -        }
>      }
>
> -    if (c->write->error) {
> -        goto done;
> -    }
> -
> -    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
> -                   "quic stream id:0x%xL send fin", qs->id);
> -
> -    frame = ngx_quic_alloc_frame(pc);
> -    if (frame == NULL) {
> -        goto done;
> -    }
> -
> -    frame->level = ssl_encryption_application;
> -    frame->type = NGX_QUIC_FT_STREAM;
> -    frame->u.stream.off = 1;
> -    frame->u.stream.len = 1;
> -    frame->u.stream.fin = 1;
> -
> -    frame->u.stream.stream_id = qs->id;
> -    frame->u.stream.offset = c->sent;
> -    frame->u.stream.length = 0;
> -
> -    ngx_quic_queue_frame(qc, frame);
> -
>  done:
>
>      (void) ngx_quic_output(pc);
> diff --git a/src/os/unix/ngx_socket.h b/src/os/unix/ngx_socket.h
> --- a/src/os/unix/ngx_socket.h
> +++ b/src/os/unix/ngx_socket.h
> @@ -13,6 +13,8 @@
>
>
>  #define NGX_WRITE_SHUTDOWN SHUT_WR
> +#define NGX_READ_SHUTDOWN  SHUT_RD
> +#define NGX_RDWR_SHUTDOWN  SHUT_RDWR
>
>  typedef int  ngx_socket_t;

looks good!

From cubicdaiya at gmail.com  Tue Dec 14 06:09:43 2021
From: cubicdaiya at gmail.com (Tatsuhiko Kubo)
Date: Tue, 14 Dec 2021 15:09:43 +0900
Subject: [PATCH 0 of 4] PCRE2 support
In-Reply-To: 
References: 
Message-ID: 

Hello,

I'm trying these patches and the build of nginx with the option
"--with-pcre-jit" fails.

$ cd nginx-1.21.4 # with applied patches
$ ./configure --with-pcre=../pcre2/pcre2-10.39 --with-pcre-jit
$ make
...
cc -c -pipe  -O -Wall -Wextra -Wpointer-arith
-Wconditional-uninitialized -Wno-unused-parameter
-Wno-deprecated-declarations -Werror -g  -I src/core -I src/event -I
src/event/modules -I src/os/unix -I ../pcre2/pcre2-10.39/src/ -I objs
\
                -o objs/src/core/ngx_regex.o \
                src/core/ngx_regex.c
src/core/ngx_regex.c:590:15: error: use of undeclared identifier
'PCRE_STUDY_JIT_COMPILE'
        opt = PCRE_STUDY_JIT_COMPILE;

Thanks.

--
Tatsuhiko Kubo

2021?12?13?(?) 11:31 Maxim Dounin :
>
> Hello!
>
> The following patch series adds PCRE2 support.
> Review and testing appreciated.
>
> --
> Maxim Dounin
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

From mdounin at mdounin.ru  Tue Dec 14 14:15:47 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 14 Dec 2021 17:15:47 +0300
Subject: [PATCH 2 of 4] Simplified sendfile(SF_NODISKIO) usage
In-Reply-To: <20211130120503.552wawqq52udi3ag@MacBook-Air-Sergey.local>
References: 
 <4a954e89b1ae8539bbe0.1636604470@vm-bsd.mdounin.ru>
 <20211130120503.552wawqq52udi3ag@MacBook-Air-Sergey.local>
Message-ID: 

Hello!

On Tue, Nov 30, 2021 at 03:05:03PM +0300, Sergey Kandaurov wrote:

> On Thu, Nov 11, 2021 at 07:21:10AM +0300, Maxim Dounin wrote:
> > # HG changeset patch
> > # User Maxim Dounin 
> > # Date 1636603886 -10800
> > #      Thu Nov 11 07:11:26 2021 +0300
> > # Node ID 4a954e89b1ae8539bbe08c5afc1d5c9828d82d6f
> > # Parent  0fb75ef9dbca698e5e855145cf6a12180a36d400
> > Simplified sendfile(SF_NODISKIO) usage.
> > 
> > Starting with FreeBSD 11, there is no need to use AIO operations to preload
> > data into cache for sendfile(SF_NODISKIO) to work.  Instead, sendfile()
> > handles non-blocking loading data from disk by itself.  It still can, however,
> > return EBUSY if a page is already being loaded (for example, by a different
> > process).  If this happens, we now post an event for the next event loop
> > iteration, so sendfile() is retried "after a short period", as manpage
> > recommends.
> > 
> > The limit of the number of EBUSY tolerated without any progress is preserved,
> > but now it does not result in an alert, since on an idle system event loop
> > iteration might be very short and EBUSY can happen many times in a row.
> > Instead, SF_NODISKIO is simply disabled for one call once the limit is
> > reached.
> > 
> > With this change, sendfile(SF_NODISKIO) is now used automatically as long as
> > sendfile() is enabled, and no longer requires "aio on;".
> > 
> > diff --git a/auto/os/freebsd b/auto/os/freebsd
> > --- a/auto/os/freebsd
> > +++ b/auto/os/freebsd
> > @@ -44,12 +44,10 @@ if [ $osreldate -gt 300007 ]; then
> >      CORE_SRCS="$CORE_SRCS $FREEBSD_SENDFILE_SRCS"
> >  fi
> >  
> > -if [ $NGX_FILE_AIO = YES ]; then
> > -    if [ $osreldate -gt 502103 ]; then
> > -        echo " + sendfile()'s SF_NODISKIO found"
> > +if [ $osreldate -gt 1100000 ]; then
> > +    echo " + sendfile()'s SF_NODISKIO found"
> >  
> > -        have=NGX_HAVE_AIO_SENDFILE . auto/have
> > -    fi
> > +    have=NGX_HAVE_SENDFILE_NODISKIO . auto/have
> >  fi
> >  
> >  # POSIX semaphores
> 
> We could check the exact __FreeBSD_version number 1100093 that was
> at the time the new sendfile() appeared, which is more accurate.
> 
> https://cgit.freebsd.org/src/commit/?id=2bab0c553588
> https://cgit.freebsd.org/src/tree/sys/sys/param.h?id=2bab0c553588#n48
> 
> Unfortunately, it was not bumped (same as with SF_NODISKIO in 5.2.1).

Yes, probably.  But the next version bump would be more 
appropriate, changed to 1100093.  Similarly to 502103, which is 
the next version after SF_NODISKIO introduction:

https://cgit.freebsd.org/src/commit/?id=b49d824e8bc1
https://cgit.freebsd.org/src/tree/sys/sys/param.h?id=b49d824e8bc1

[...]

> > -static ngx_int_t
> > -ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx, ngx_file_t *file)
> > -{
> > -    ngx_event_aio_t  *aio;
> > -
> > -    if (file->aio == NULL && ngx_file_aio_init(file, ctx->pool) != NGX_OK) {
> > -        return NGX_ERROR;
> > -    }
> > -
> > -    aio = file->aio;
> > -
> > -    aio->data = ctx->filter_ctx;
> > -    aio->preload_handler = ctx->aio_preload;
> > -
> > -    return NGX_OK;
> > -}
> > -
> > -#endif
> > -
> > -
> >  static ngx_int_t
> >  ngx_output_chain_add_copy(ngx_pool_t *pool, ngx_chain_t **chain,
> >      ngx_chain_t *in)
> 
> After this change, ngx_file_aio_init() doesn't need to be external.
> It is only used in ngx_file_aio_read() in corresponding ngx_*_aio_read.c.

In practice, yes.  In theory, it might be needed if we'll ever 
implement other AIO operations, such as aio_write(), and/or will 
reconsider thread-based interface to use the aio structure.  So I 
would rather  preserve it as is, at least for now.

[...]

> >  ngx_chain_t *
> >  ngx_freebsd_sendfile_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
> >  {
> > -    int               rc, flags;
> > -    off_t             send, prev_send, sent;
> > -    size_t            file_size;
> > -    ssize_t           n;
> > -    ngx_uint_t        eintr, eagain;
> > -    ngx_err_t         err;
> > -    ngx_buf_t        *file;
> > -    ngx_event_t      *wev;
> > -    ngx_chain_t      *cl;
> > -    ngx_iovec_t       header, trailer;
> > -    struct sf_hdtr    hdtr;
> > -    struct iovec      headers[NGX_IOVS_PREALLOCATE];
> > -    struct iovec      trailers[NGX_IOVS_PREALLOCATE];
> > -#if (NGX_HAVE_AIO_SENDFILE)
> > -    ngx_uint_t        ebusy;
> > -    ngx_event_aio_t  *aio;
> > +    int              rc, flags;
> > +    off_t            send, prev_send, sent;
> > +    size_t           file_size;
> > +    ssize_t          n;
> > +    ngx_uint_t       eintr, eagain;
> > +    ngx_err_t        err;
> > +    ngx_buf_t       *file;
> > +    ngx_event_t     *wev;
> > +    ngx_chain_t     *cl;
> > +    ngx_iovec_t      header, trailer;
> > +    struct sf_hdtr   hdtr;
> > +    struct iovec     headers[NGX_IOVS_PREALLOCATE];
> > +    struct iovec     trailers[NGX_IOVS_PREALLOCATE];
> > +#if (NGX_HAVE_SENDFILE_NODISKIO)
> > +    ngx_uint_t       ebusy;
> >  #endif
> 
> After ngx_event_aio_t *aio variable removal,
> this block could be placed under "eintr, eagain" line
> (which by itself looks unsorted).
> 
> The remaining part looks good to me.

Yes, thanks.  Changed to:

diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -36,18 +36,18 @@ ngx_freebsd_sendfile_chain(ngx_connectio
     off_t            send, prev_send, sent;
     size_t           file_size;
     ssize_t          n;
-    ngx_uint_t       eintr, eagain;
     ngx_err_t        err;
     ngx_buf_t       *file;
+    ngx_uint_t       eintr, eagain;
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+    ngx_uint_t       ebusy;
+#endif
     ngx_event_t     *wev;
     ngx_chain_t     *cl;
     ngx_iovec_t      header, trailer;
     struct sf_hdtr   hdtr;
     struct iovec     headers[NGX_IOVS_PREALLOCATE];
     struct iovec     trailers[NGX_IOVS_PREALLOCATE];
-#if (NGX_HAVE_SENDFILE_NODISKIO)
-    ngx_uint_t       ebusy;
-#endif
 
     wev = c->write;
 

Full patch, updated:

# HG changeset patch
# User Maxim Dounin 
# Date 1639447743 -10800
#      Tue Dec 14 05:09:03 2021 +0300
# Node ID a149cd880ddd1473d21b66b85b9307bb17966df3
# Parent  42f1674706c9922fa910a7601af83307b3633b95
Simplified sendfile(SF_NODISKIO) usage.

Starting with FreeBSD 11, there is no need to use AIO operations to preload
data into cache for sendfile(SF_NODISKIO) to work.  Instead, sendfile()
handles non-blocking loading data from disk by itself.  It still can, however,
return EBUSY if a page is already being loaded (for example, by a different
process).  If this happens, we now post an event for the next event loop
iteration, so sendfile() is retried "after a short period", as manpage
recommends.

The limit of the number of EBUSY tolerated without any progress is preserved,
but now it does not result in an alert, since on an idle system event loop
iteration might be very short and EBUSY can happen many times in a row.
Instead, SF_NODISKIO is simply disabled for one call once the limit is
reached.

With this change, sendfile(SF_NODISKIO) is now used automatically as long as
sendfile() is enabled, and no longer requires "aio on;".

diff --git a/auto/os/freebsd b/auto/os/freebsd
--- a/auto/os/freebsd
+++ b/auto/os/freebsd
@@ -44,12 +44,10 @@ if [ $osreldate -gt 300007 ]; then
     CORE_SRCS="$CORE_SRCS $FREEBSD_SENDFILE_SRCS"
 fi
 
-if [ $NGX_FILE_AIO = YES ]; then
-    if [ $osreldate -gt 502103 ]; then
-        echo " + sendfile()'s SF_NODISKIO found"
+if [ $osreldate -gt 1100093 ]; then
+    echo " + sendfile()'s SF_NODISKIO found"
 
-        have=NGX_HAVE_AIO_SENDFILE . auto/have
-    fi
+    have=NGX_HAVE_SENDFILE_NODISKIO . auto/have
 fi
 
 # POSIX semaphores
diff --git a/src/core/ngx_buf.h b/src/core/ngx_buf.h
--- a/src/core/ngx_buf.h
+++ b/src/core/ngx_buf.h
@@ -90,9 +90,6 @@ struct ngx_output_chain_ctx_s {
 
 #if (NGX_HAVE_FILE_AIO || NGX_COMPAT)
     ngx_output_chain_aio_pt      aio_handler;
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
-    ssize_t                    (*aio_preload)(ngx_buf_t *file);
-#endif
 #endif
 
 #if (NGX_THREADS || NGX_COMPAT)
diff --git a/src/core/ngx_connection.h b/src/core/ngx_connection.h
--- a/src/core/ngx_connection.h
+++ b/src/core/ngx_connection.h
@@ -185,7 +185,7 @@ struct ngx_connection_s {
 
     unsigned            need_last_buf:1;
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
+#if (NGX_HAVE_SENDFILE_NODISKIO || NGX_COMPAT)
     unsigned            busy_count:2;
 #endif
 
diff --git a/src/core/ngx_module.h b/src/core/ngx_module.h
--- a/src/core/ngx_module.h
+++ b/src/core/ngx_module.h
@@ -41,7 +41,7 @@
 #define NGX_MODULE_SIGNATURE_3   "0"
 #endif
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
+#if (NGX_HAVE_SENDFILE_NODISKIO || NGX_COMPAT)
 #define NGX_MODULE_SIGNATURE_4   "1"
 #else
 #define NGX_MODULE_SIGNATURE_4   "0"
diff --git a/src/core/ngx_output_chain.c b/src/core/ngx_output_chain.c
--- a/src/core/ngx_output_chain.c
+++ b/src/core/ngx_output_chain.c
@@ -29,10 +29,6 @@
 
 static ngx_inline ngx_int_t
     ngx_output_chain_as_is(ngx_output_chain_ctx_t *ctx, ngx_buf_t *buf);
-#if (NGX_HAVE_AIO_SENDFILE)
-static ngx_int_t ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx,
-    ngx_file_t *file);
-#endif
 static ngx_int_t ngx_output_chain_add_copy(ngx_pool_t *pool,
     ngx_chain_t **chain, ngx_chain_t *in);
 static ngx_int_t ngx_output_chain_align_file_buf(ngx_output_chain_ctx_t *ctx,
@@ -283,12 +279,6 @@ ngx_output_chain_as_is(ngx_output_chain_
         buf->in_file = 0;
     }
 
-#if (NGX_HAVE_AIO_SENDFILE)
-    if (ctx->aio_preload && buf->in_file) {
-        (void) ngx_output_chain_aio_setup(ctx, buf->file);
-    }
-#endif
-
     if (ctx->need_in_memory && !ngx_buf_in_memory(buf)) {
         return 0;
     }
@@ -301,28 +291,6 @@ ngx_output_chain_as_is(ngx_output_chain_
 }
 
 
-#if (NGX_HAVE_AIO_SENDFILE)
-
-static ngx_int_t
-ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx, ngx_file_t *file)
-{
-    ngx_event_aio_t  *aio;
-
-    if (file->aio == NULL && ngx_file_aio_init(file, ctx->pool) != NGX_OK) {
-        return NGX_ERROR;
-    }
-
-    aio = file->aio;
-
-    aio->data = ctx->filter_ctx;
-    aio->preload_handler = ctx->aio_preload;
-
-    return NGX_OK;
-}
-
-#endif
-
-
 static ngx_int_t
 ngx_output_chain_add_copy(ngx_pool_t *pool, ngx_chain_t **chain,
     ngx_chain_t *in)
diff --git a/src/event/ngx_event.h b/src/event/ngx_event.h
--- a/src/event/ngx_event.h
+++ b/src/event/ngx_event.h
@@ -147,10 +147,6 @@ struct ngx_event_aio_s {
 
     ngx_fd_t                   fd;
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
-    ssize_t                  (*preload_handler)(ngx_buf_t *file);
-#endif
-
 #if (NGX_HAVE_EVENTFD)
     int64_t                    res;
 #endif
diff --git a/src/http/ngx_http_copy_filter_module.c b/src/http/ngx_http_copy_filter_module.c
--- a/src/http/ngx_http_copy_filter_module.c
+++ b/src/http/ngx_http_copy_filter_module.c
@@ -19,10 +19,6 @@ typedef struct {
 static void ngx_http_copy_aio_handler(ngx_output_chain_ctx_t *ctx,
     ngx_file_t *file);
 static void ngx_http_copy_aio_event_handler(ngx_event_t *ev);
-#if (NGX_HAVE_AIO_SENDFILE)
-static ssize_t ngx_http_copy_aio_sendfile_preload(ngx_buf_t *file);
-static void ngx_http_copy_aio_sendfile_event_handler(ngx_event_t *ev);
-#endif
 #endif
 #if (NGX_THREADS)
 static ngx_int_t ngx_http_copy_thread_handler(ngx_thread_task_t *task,
@@ -128,9 +124,6 @@ ngx_http_copy_filter(ngx_http_request_t 
 #if (NGX_HAVE_FILE_AIO)
         if (ngx_file_aio && clcf->aio == NGX_HTTP_AIO_ON) {
             ctx->aio_handler = ngx_http_copy_aio_handler;
-#if (NGX_HAVE_AIO_SENDFILE)
-            ctx->aio_preload = ngx_http_copy_aio_sendfile_preload;
-#endif
         }
 #endif
 
@@ -207,81 +200,6 @@ ngx_http_copy_aio_event_handler(ngx_even
     ngx_http_run_posted_requests(c);
 }
 
-
-#if (NGX_HAVE_AIO_SENDFILE)
-
-static ssize_t
-ngx_http_copy_aio_sendfile_preload(ngx_buf_t *file)
-{
-    ssize_t                  n;
-    static u_char            buf[1];
-    ngx_event_aio_t         *aio;
-    ngx_http_request_t      *r;
-    ngx_output_chain_ctx_t  *ctx;
-
-    aio = file->file->aio;
-    r = aio->data;
-
-    if (r->aio) {
-        /*
-         * tolerate sendfile() calls if another operation is already
-         * running; this can happen due to subrequests, multiple calls
-         * of the next body filter from a filter, or in HTTP/2 due to
-         * a write event on the main connection
-         */
-
-        return NGX_AGAIN;
-    }
-
-    n = ngx_file_aio_read(file->file, buf, 1, file->file_pos, NULL);
-
-    if (n == NGX_AGAIN) {
-        aio->handler = ngx_http_copy_aio_sendfile_event_handler;
-
-        r->main->blocked++;
-        r->aio = 1;
-
-        ctx = ngx_http_get_module_ctx(r, ngx_http_copy_filter_module);
-        ctx->aio = 1;
-    }
-
-    return n;
-}
-
-
-static void
-ngx_http_copy_aio_sendfile_event_handler(ngx_event_t *ev)
-{
-    ngx_event_aio_t     *aio;
-    ngx_connection_t    *c;
-    ngx_http_request_t  *r;
-
-    aio = ev->data;
-    r = aio->data;
-    c = r->connection;
-
-    r->main->blocked--;
-    r->aio = 0;
-    ev->complete = 0;
-
-#if (NGX_HTTP_V2)
-
-    if (r->stream) {
-        /*
-         * for HTTP/2, update write event to make sure processing will
-         * reach the main connection to handle sendfile() preload
-         */
-
-        c->write->ready = 1;
-        c->write->active = 0;
-    }
-
-#endif
-
-    c->write->handler(c->write);
-}
-
-#endif
 #endif
 
 
diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -32,23 +32,22 @@
 ngx_chain_t *
 ngx_freebsd_sendfile_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
 {
-    int               rc, flags;
-    off_t             send, prev_send, sent;
-    size_t            file_size;
-    ssize_t           n;
-    ngx_uint_t        eintr, eagain;
-    ngx_err_t         err;
-    ngx_buf_t        *file;
-    ngx_event_t      *wev;
-    ngx_chain_t      *cl;
-    ngx_iovec_t       header, trailer;
-    struct sf_hdtr    hdtr;
-    struct iovec      headers[NGX_IOVS_PREALLOCATE];
-    struct iovec      trailers[NGX_IOVS_PREALLOCATE];
-#if (NGX_HAVE_AIO_SENDFILE)
-    ngx_uint_t        ebusy;
-    ngx_event_aio_t  *aio;
+    int              rc, flags;
+    off_t            send, prev_send, sent;
+    size_t           file_size;
+    ssize_t          n;
+    ngx_err_t        err;
+    ngx_buf_t       *file;
+    ngx_uint_t       eintr, eagain;
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+    ngx_uint_t       ebusy;
 #endif
+    ngx_event_t     *wev;
+    ngx_chain_t     *cl;
+    ngx_iovec_t      header, trailer;
+    struct sf_hdtr   hdtr;
+    struct iovec     headers[NGX_IOVS_PREALLOCATE];
+    struct iovec     trailers[NGX_IOVS_PREALLOCATE];
 
     wev = c->write;
 
@@ -77,11 +76,6 @@ ngx_freebsd_sendfile_chain(ngx_connectio
     eagain = 0;
     flags = 0;
 
-#if (NGX_HAVE_AIO_SENDFILE && NGX_SUPPRESS_WARN)
-    aio = NULL;
-    file = NULL;
-#endif
-
     header.iovs = headers;
     header.nalloc = NGX_IOVS_PREALLOCATE;
 
@@ -90,7 +84,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
     for ( ;; ) {
         eintr = 0;
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
         ebusy = 0;
 #endif
         prev_send = send;
@@ -179,9 +173,8 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
             sent = 0;
 
-#if (NGX_HAVE_AIO_SENDFILE)
-            aio = file->file->aio;
-            flags = (aio && aio->preload_handler) ? SF_NODISKIO : 0;
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+            flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
 #endif
 
             rc = sendfile(file->file->fd, c->fd, file->file_pos,
@@ -199,7 +192,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
                     eintr = 1;
                     break;
 
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
                 case NGX_EBUSY:
                     ebusy = 1;
                     break;
@@ -258,35 +251,24 @@ ngx_freebsd_sendfile_chain(ngx_connectio
             if (sent == 0) {
                 c->busy_count++;
 
-                if (c->busy_count > 2) {
-                    ngx_log_error(NGX_LOG_ALERT, c->log, 0,
-                                  "sendfile(%V) returned busy again",
-                                  &file->file->name);
-
-                    c->busy_count = 0;
-                    aio->preload_handler = NULL;
-
-                    send = prev_send;
-                    continue;
-                }
+                ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
+                               "sendfile() busy, count:%d", c->busy_count);
 
             } else {
                 c->busy_count = 0;
             }
 
-            n = aio->preload_handler(file);
-
-            if (n > 0) {
-                send = prev_send + sent;
-                continue;
+            if (wev->posted) {
+                ngx_delete_posted_event(wev);
             }
 
+            ngx_post_event(wev, &ngx_posted_next_events);
+
+            wev->ready = 0;
             return in;
         }
 
-        if (flags == SF_NODISKIO) {
-            c->busy_count = 0;
-        }
+        c->busy_count = 0;
 
 #endif
 


-- 
Maxim Dounin
http://mdounin.ru/

From mdounin at mdounin.ru  Tue Dec 14 14:16:54 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 14 Dec 2021 17:16:54 +0300
Subject: [PATCH 4 of 4] Support for sendfile(SF_NOCACHE)
In-Reply-To: <20211130121550.2eh5sm5aedkph2oz@MacBook-Air-Sergey.local>
References: 
 <10f96e74ae73e1c53a3f.1636604472@vm-bsd.mdounin.ru>
 <20211130121550.2eh5sm5aedkph2oz@MacBook-Air-Sergey.local>
Message-ID: 

Hello!

On Tue, Nov 30, 2021 at 03:15:50PM +0300, Sergey Kandaurov wrote:

> On Thu, Nov 11, 2021 at 07:21:12AM +0300, Maxim Dounin wrote:
> > # HG changeset patch
> > # User Maxim Dounin 
> > # Date 1636603897 -10800
> > #      Thu Nov 11 07:11:37 2021 +0300
> > # Node ID 10f96e74ae73e1c53a3fd08e7e1c26754c8969ed
> > # Parent  98d3beb63f32cbb68d1cdcec385614d32129cad0
> > Support for sendfile(SF_NOCACHE).
> > 
> > The SF_NOCACHE flag, introduced in FreeBSD 11 along with the new non-blocking
> > sendfile() implementation by glebius@, makes it possible to use sendfile()
> > along with the "directio" directive.
> > 
> > diff --git a/src/core/ngx_output_chain.c b/src/core/ngx_output_chain.c
> > --- a/src/core/ngx_output_chain.c
> > +++ b/src/core/ngx_output_chain.c
> > @@ -256,9 +256,11 @@ ngx_output_chain_as_is(ngx_output_chain_
> >      }
> >  #endif
> >  
> > +#if !(NGX_HAVE_SENDFILE_NODISKIO)
> >      if (buf->in_file && buf->file->directio) {
> >          return 0;
> >      }
> > +#endif
> 
> This probably deserves a comment, why it depends on such a macro test.
> Though, it should be pretty clear from the commit log.

I also tend to think that the original code is wrong (or, rather, 
too aggressive), and it should only disable sendfile(), much like 
the ctx->sendfile and NGX_SENDFILE_LIMIT checks below.

Changed to the following:

diff --git a/src/core/ngx_output_chain.c b/src/core/ngx_output_chain.c
--- a/src/core/ngx_output_chain.c
+++ b/src/core/ngx_output_chain.c
@@ -256,12 +256,6 @@ ngx_output_chain_as_is(ngx_output_chain_
     }
 #endif
 
-#if !(NGX_HAVE_SENDFILE_NODISKIO)
-    if (buf->in_file && buf->file->directio) {
-        return 0;
-    }
-#endif
-
     sendfile = ctx->sendfile;
 
 #if (NGX_SENDFILE_LIMIT)
@@ -272,6 +266,19 @@ ngx_output_chain_as_is(ngx_output_chain_
 
 #endif
 
+#if !(NGX_HAVE_SENDFILE_NODISKIO)
+
+    /*
+     * With DIRECTIO, disable sendfile() unless sendfile(SF_NOCACHE)
+     * is available.
+     */
+
+    if (buf->in_file && buf->file->directio) {
+        sendfile = 0;
+    }
+
+#endif
+
     if (!sendfile) {
 
         if (!ngx_buf_in_memory(buf)) {


[...]

> Otherwise, looks good.

Full series, updated:

# HG changeset patch
# User Maxim Dounin 
# Date 1639425539 -10800
#      Mon Dec 13 22:58:59 2021 +0300
# Node ID 42f1674706c9922fa910a7601af83307b3633b95
# Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
Removed "aio sendfile", deprecated since 1.7.11.

diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c
+++ b/src/http/ngx_http_core_module.c
@@ -4568,19 +4568,6 @@ ngx_http_core_set_aio(ngx_conf_t *cf, ng
 #endif
     }
 
-#if (NGX_HAVE_AIO_SENDFILE)
-
-    if (ngx_strcmp(value[1].data, "sendfile") == 0) {
-        clcf->aio = NGX_HTTP_AIO_ON;
-
-        ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
-                           "the \"sendfile\" parameter of "
-                           "the \"aio\" directive is deprecated");
-        return NGX_CONF_OK;
-    }
-
-#endif
-
     if (ngx_strncmp(value[1].data, "threads", 7) == 0
         && (value[1].len == 7 || value[1].data[7] == '='))
     {
# HG changeset patch
# User Maxim Dounin 
# Date 1639447743 -10800
#      Tue Dec 14 05:09:03 2021 +0300
# Node ID a149cd880ddd1473d21b66b85b9307bb17966df3
# Parent  42f1674706c9922fa910a7601af83307b3633b95
Simplified sendfile(SF_NODISKIO) usage.

Starting with FreeBSD 11, there is no need to use AIO operations to preload
data into cache for sendfile(SF_NODISKIO) to work.  Instead, sendfile()
handles non-blocking loading data from disk by itself.  It still can, however,
return EBUSY if a page is already being loaded (for example, by a different
process).  If this happens, we now post an event for the next event loop
iteration, so sendfile() is retried "after a short period", as manpage
recommends.

The limit of the number of EBUSY tolerated without any progress is preserved,
but now it does not result in an alert, since on an idle system event loop
iteration might be very short and EBUSY can happen many times in a row.
Instead, SF_NODISKIO is simply disabled for one call once the limit is
reached.

With this change, sendfile(SF_NODISKIO) is now used automatically as long as
sendfile() is enabled, and no longer requires "aio on;".

diff --git a/auto/os/freebsd b/auto/os/freebsd
--- a/auto/os/freebsd
+++ b/auto/os/freebsd
@@ -44,12 +44,10 @@ if [ $osreldate -gt 300007 ]; then
     CORE_SRCS="$CORE_SRCS $FREEBSD_SENDFILE_SRCS"
 fi
 
-if [ $NGX_FILE_AIO = YES ]; then
-    if [ $osreldate -gt 502103 ]; then
-        echo " + sendfile()'s SF_NODISKIO found"
+if [ $osreldate -gt 1100093 ]; then
+    echo " + sendfile()'s SF_NODISKIO found"
 
-        have=NGX_HAVE_AIO_SENDFILE . auto/have
-    fi
+    have=NGX_HAVE_SENDFILE_NODISKIO . auto/have
 fi
 
 # POSIX semaphores
diff --git a/src/core/ngx_buf.h b/src/core/ngx_buf.h
--- a/src/core/ngx_buf.h
+++ b/src/core/ngx_buf.h
@@ -90,9 +90,6 @@ struct ngx_output_chain_ctx_s {
 
 #if (NGX_HAVE_FILE_AIO || NGX_COMPAT)
     ngx_output_chain_aio_pt      aio_handler;
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
-    ssize_t                    (*aio_preload)(ngx_buf_t *file);
-#endif
 #endif
 
 #if (NGX_THREADS || NGX_COMPAT)
diff --git a/src/core/ngx_connection.h b/src/core/ngx_connection.h
--- a/src/core/ngx_connection.h
+++ b/src/core/ngx_connection.h
@@ -185,7 +185,7 @@ struct ngx_connection_s {
 
     unsigned            need_last_buf:1;
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
+#if (NGX_HAVE_SENDFILE_NODISKIO || NGX_COMPAT)
     unsigned            busy_count:2;
 #endif
 
diff --git a/src/core/ngx_module.h b/src/core/ngx_module.h
--- a/src/core/ngx_module.h
+++ b/src/core/ngx_module.h
@@ -41,7 +41,7 @@
 #define NGX_MODULE_SIGNATURE_3   "0"
 #endif
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
+#if (NGX_HAVE_SENDFILE_NODISKIO || NGX_COMPAT)
 #define NGX_MODULE_SIGNATURE_4   "1"
 #else
 #define NGX_MODULE_SIGNATURE_4   "0"
diff --git a/src/core/ngx_output_chain.c b/src/core/ngx_output_chain.c
--- a/src/core/ngx_output_chain.c
+++ b/src/core/ngx_output_chain.c
@@ -29,10 +29,6 @@
 
 static ngx_inline ngx_int_t
     ngx_output_chain_as_is(ngx_output_chain_ctx_t *ctx, ngx_buf_t *buf);
-#if (NGX_HAVE_AIO_SENDFILE)
-static ngx_int_t ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx,
-    ngx_file_t *file);
-#endif
 static ngx_int_t ngx_output_chain_add_copy(ngx_pool_t *pool,
     ngx_chain_t **chain, ngx_chain_t *in);
 static ngx_int_t ngx_output_chain_align_file_buf(ngx_output_chain_ctx_t *ctx,
@@ -283,12 +279,6 @@ ngx_output_chain_as_is(ngx_output_chain_
         buf->in_file = 0;
     }
 
-#if (NGX_HAVE_AIO_SENDFILE)
-    if (ctx->aio_preload && buf->in_file) {
-        (void) ngx_output_chain_aio_setup(ctx, buf->file);
-    }
-#endif
-
     if (ctx->need_in_memory && !ngx_buf_in_memory(buf)) {
         return 0;
     }
@@ -301,28 +291,6 @@ ngx_output_chain_as_is(ngx_output_chain_
 }
 
 
-#if (NGX_HAVE_AIO_SENDFILE)
-
-static ngx_int_t
-ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx, ngx_file_t *file)
-{
-    ngx_event_aio_t  *aio;
-
-    if (file->aio == NULL && ngx_file_aio_init(file, ctx->pool) != NGX_OK) {
-        return NGX_ERROR;
-    }
-
-    aio = file->aio;
-
-    aio->data = ctx->filter_ctx;
-    aio->preload_handler = ctx->aio_preload;
-
-    return NGX_OK;
-}
-
-#endif
-
-
 static ngx_int_t
 ngx_output_chain_add_copy(ngx_pool_t *pool, ngx_chain_t **chain,
     ngx_chain_t *in)
diff --git a/src/event/ngx_event.h b/src/event/ngx_event.h
--- a/src/event/ngx_event.h
+++ b/src/event/ngx_event.h
@@ -147,10 +147,6 @@ struct ngx_event_aio_s {
 
     ngx_fd_t                   fd;
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
-    ssize_t                  (*preload_handler)(ngx_buf_t *file);
-#endif
-
 #if (NGX_HAVE_EVENTFD)
     int64_t                    res;
 #endif
diff --git a/src/http/ngx_http_copy_filter_module.c b/src/http/ngx_http_copy_filter_module.c
--- a/src/http/ngx_http_copy_filter_module.c
+++ b/src/http/ngx_http_copy_filter_module.c
@@ -19,10 +19,6 @@ typedef struct {
 static void ngx_http_copy_aio_handler(ngx_output_chain_ctx_t *ctx,
     ngx_file_t *file);
 static void ngx_http_copy_aio_event_handler(ngx_event_t *ev);
-#if (NGX_HAVE_AIO_SENDFILE)
-static ssize_t ngx_http_copy_aio_sendfile_preload(ngx_buf_t *file);
-static void ngx_http_copy_aio_sendfile_event_handler(ngx_event_t *ev);
-#endif
 #endif
 #if (NGX_THREADS)
 static ngx_int_t ngx_http_copy_thread_handler(ngx_thread_task_t *task,
@@ -128,9 +124,6 @@ ngx_http_copy_filter(ngx_http_request_t 
 #if (NGX_HAVE_FILE_AIO)
         if (ngx_file_aio && clcf->aio == NGX_HTTP_AIO_ON) {
             ctx->aio_handler = ngx_http_copy_aio_handler;
-#if (NGX_HAVE_AIO_SENDFILE)
-            ctx->aio_preload = ngx_http_copy_aio_sendfile_preload;
-#endif
         }
 #endif
 
@@ -207,81 +200,6 @@ ngx_http_copy_aio_event_handler(ngx_even
     ngx_http_run_posted_requests(c);
 }
 
-
-#if (NGX_HAVE_AIO_SENDFILE)
-
-static ssize_t
-ngx_http_copy_aio_sendfile_preload(ngx_buf_t *file)
-{
-    ssize_t                  n;
-    static u_char            buf[1];
-    ngx_event_aio_t         *aio;
-    ngx_http_request_t      *r;
-    ngx_output_chain_ctx_t  *ctx;
-
-    aio = file->file->aio;
-    r = aio->data;
-
-    if (r->aio) {
-        /*
-         * tolerate sendfile() calls if another operation is already
-         * running; this can happen due to subrequests, multiple calls
-         * of the next body filter from a filter, or in HTTP/2 due to
-         * a write event on the main connection
-         */
-
-        return NGX_AGAIN;
-    }
-
-    n = ngx_file_aio_read(file->file, buf, 1, file->file_pos, NULL);
-
-    if (n == NGX_AGAIN) {
-        aio->handler = ngx_http_copy_aio_sendfile_event_handler;
-
-        r->main->blocked++;
-        r->aio = 1;
-
-        ctx = ngx_http_get_module_ctx(r, ngx_http_copy_filter_module);
-        ctx->aio = 1;
-    }
-
-    return n;
-}
-
-
-static void
-ngx_http_copy_aio_sendfile_event_handler(ngx_event_t *ev)
-{
-    ngx_event_aio_t     *aio;
-    ngx_connection_t    *c;
-    ngx_http_request_t  *r;
-
-    aio = ev->data;
-    r = aio->data;
-    c = r->connection;
-
-    r->main->blocked--;
-    r->aio = 0;
-    ev->complete = 0;
-
-#if (NGX_HTTP_V2)
-
-    if (r->stream) {
-        /*
-         * for HTTP/2, update write event to make sure processing will
-         * reach the main connection to handle sendfile() preload
-         */
-
-        c->write->ready = 1;
-        c->write->active = 0;
-    }
-
-#endif
-
-    c->write->handler(c->write);
-}
-
-#endif
 #endif
 
 
diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -32,23 +32,22 @@
 ngx_chain_t *
 ngx_freebsd_sendfile_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
 {
-    int               rc, flags;
-    off_t             send, prev_send, sent;
-    size_t            file_size;
-    ssize_t           n;
-    ngx_uint_t        eintr, eagain;
-    ngx_err_t         err;
-    ngx_buf_t        *file;
-    ngx_event_t      *wev;
-    ngx_chain_t      *cl;
-    ngx_iovec_t       header, trailer;
-    struct sf_hdtr    hdtr;
-    struct iovec      headers[NGX_IOVS_PREALLOCATE];
-    struct iovec      trailers[NGX_IOVS_PREALLOCATE];
-#if (NGX_HAVE_AIO_SENDFILE)
-    ngx_uint_t        ebusy;
-    ngx_event_aio_t  *aio;
+    int              rc, flags;
+    off_t            send, prev_send, sent;
+    size_t           file_size;
+    ssize_t          n;
+    ngx_err_t        err;
+    ngx_buf_t       *file;
+    ngx_uint_t       eintr, eagain;
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+    ngx_uint_t       ebusy;
 #endif
+    ngx_event_t     *wev;
+    ngx_chain_t     *cl;
+    ngx_iovec_t      header, trailer;
+    struct sf_hdtr   hdtr;
+    struct iovec     headers[NGX_IOVS_PREALLOCATE];
+    struct iovec     trailers[NGX_IOVS_PREALLOCATE];
 
     wev = c->write;
 
@@ -77,11 +76,6 @@ ngx_freebsd_sendfile_chain(ngx_connectio
     eagain = 0;
     flags = 0;
 
-#if (NGX_HAVE_AIO_SENDFILE && NGX_SUPPRESS_WARN)
-    aio = NULL;
-    file = NULL;
-#endif
-
     header.iovs = headers;
     header.nalloc = NGX_IOVS_PREALLOCATE;
 
@@ -90,7 +84,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
     for ( ;; ) {
         eintr = 0;
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
         ebusy = 0;
 #endif
         prev_send = send;
@@ -179,9 +173,8 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
             sent = 0;
 
-#if (NGX_HAVE_AIO_SENDFILE)
-            aio = file->file->aio;
-            flags = (aio && aio->preload_handler) ? SF_NODISKIO : 0;
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+            flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
 #endif
 
             rc = sendfile(file->file->fd, c->fd, file->file_pos,
@@ -199,7 +192,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
                     eintr = 1;
                     break;
 
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
                 case NGX_EBUSY:
                     ebusy = 1;
                     break;
@@ -258,35 +251,24 @@ ngx_freebsd_sendfile_chain(ngx_connectio
             if (sent == 0) {
                 c->busy_count++;
 
-                if (c->busy_count > 2) {
-                    ngx_log_error(NGX_LOG_ALERT, c->log, 0,
-                                  "sendfile(%V) returned busy again",
-                                  &file->file->name);
-
-                    c->busy_count = 0;
-                    aio->preload_handler = NULL;
-
-                    send = prev_send;
-                    continue;
-                }
+                ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
+                               "sendfile() busy, count:%d", c->busy_count);
 
             } else {
                 c->busy_count = 0;
             }
 
-            n = aio->preload_handler(file);
-
-            if (n > 0) {
-                send = prev_send + sent;
-                continue;
+            if (wev->posted) {
+                ngx_delete_posted_event(wev);
             }
 
+            ngx_post_event(wev, &ngx_posted_next_events);
+
+            wev->ready = 0;
             return in;
         }
 
-        if (flags == SF_NODISKIO) {
-            c->busy_count = 0;
-        }
+        c->busy_count = 0;
 
 #endif
 
# HG changeset patch
# User Maxim Dounin 
# Date 1639450352 -10800
#      Tue Dec 14 05:52:32 2021 +0300
# Node ID e7605da2d39b6df7c31d41afb5699277ae872490
# Parent  a149cd880ddd1473d21b66b85b9307bb17966df3
SSL: SSL_sendfile(SF_NODISKIO) support.

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -2942,7 +2942,7 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
 {
 #ifdef BIO_get_ktls_send
 
-    int        sslerr;
+    int        sslerr, flags;
     ssize_t    n;
     ngx_err_t  err;
 
@@ -2954,8 +2954,14 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
 
     ngx_set_errno(0);
 
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+    flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
+#else
+    flags = 0;
+#endif
+
     n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,
-                     size, 0);
+                     size, flags);
 
     ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %d", n);
 
@@ -2974,6 +2980,10 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
             ngx_post_event(c->read, &ngx_posted_events);
         }
 
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+        c->busy_count = 0;
+#endif
+
         c->sent += n;
 
         return n;
@@ -3038,6 +3048,23 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
             ngx_post_event(c->read, &ngx_posted_events);
         }
 
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+
+        if (ngx_errno == EBUSY) {
+            c->busy_count++;
+
+            ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
+                           "SSL_sendfile() busy, count:%d", c->busy_count);
+
+            if (c->write->posted) {
+                ngx_delete_posted_event(c->write);
+            }
+
+            ngx_post_event(c->write, &ngx_posted_next_events);
+        }
+
+#endif
+
         c->write->ready = 0;
         return NGX_AGAIN;
     }
# HG changeset patch
# User Maxim Dounin 
# Date 1639491120 -10800
#      Tue Dec 14 17:12:00 2021 +0300
# Node ID 716e3f4765d065d5f59c3740f1e69bc7ea9cd368
# Parent  e7605da2d39b6df7c31d41afb5699277ae872490
Support for sendfile(SF_NOCACHE).

The SF_NOCACHE flag, introduced in FreeBSD 11 along with the new non-blocking
sendfile() implementation by glebius@, makes it possible to use sendfile()
along with the "directio" directive.

diff --git a/src/core/ngx_output_chain.c b/src/core/ngx_output_chain.c
--- a/src/core/ngx_output_chain.c
+++ b/src/core/ngx_output_chain.c
@@ -256,10 +256,6 @@ ngx_output_chain_as_is(ngx_output_chain_
     }
 #endif
 
-    if (buf->in_file && buf->file->directio) {
-        return 0;
-    }
-
     sendfile = ctx->sendfile;
 
 #if (NGX_SENDFILE_LIMIT)
@@ -270,6 +266,19 @@ ngx_output_chain_as_is(ngx_output_chain_
 
 #endif
 
+#if !(NGX_HAVE_SENDFILE_NODISKIO)
+
+    /*
+     * With DIRECTIO, disable sendfile() unless sendfile(SF_NOCACHE)
+     * is available.
+     */
+
+    if (buf->in_file && buf->file->directio) {
+        sendfile = 0;
+    }
+
+#endif
+
     if (!sendfile) {
 
         if (!ngx_buf_in_memory(buf)) {
diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -2955,7 +2955,13 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
     ngx_set_errno(0);
 
 #if (NGX_HAVE_SENDFILE_NODISKIO)
+
     flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
+
+    if (file->file->directio) {
+        flags |= SF_NOCACHE;
+    }
+
 #else
     flags = 0;
 #endif
diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -174,7 +174,13 @@ ngx_freebsd_sendfile_chain(ngx_connectio
             sent = 0;
 
 #if (NGX_HAVE_SENDFILE_NODISKIO)
+
             flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
+
+            if (file->file->directio) {
+                flags |= SF_NOCACHE;
+            }
+
 #endif
 
             rc = sendfile(file->file->fd, c->fd, file->file_pos,

-- 
Maxim Dounin
http://mdounin.ru/

From mdounin at mdounin.ru  Tue Dec 14 15:28:48 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 14 Dec 2021 18:28:48 +0300
Subject: [PATCH 2 of 4] Simplified sendfile(SF_NODISKIO) usage
In-Reply-To: 
References: 
 <4a954e89b1ae8539bbe0.1636604470@vm-bsd.mdounin.ru>
 <20211130120503.552wawqq52udi3ag@MacBook-Air-Sergey.local>
 
Message-ID: 

Hello!

On Tue, Dec 14, 2021 at 05:15:47PM +0300, Maxim Dounin wrote:

> On Tue, Nov 30, 2021 at 03:05:03PM +0300, Sergey Kandaurov wrote:
> 
> > On Thu, Nov 11, 2021 at 07:21:10AM +0300, Maxim Dounin wrote:
> > > # HG changeset patch
> > > # User Maxim Dounin 
> > > # Date 1636603886 -10800
> > > #      Thu Nov 11 07:11:26 2021 +0300
> > > # Node ID 4a954e89b1ae8539bbe08c5afc1d5c9828d82d6f
> > > # Parent  0fb75ef9dbca698e5e855145cf6a12180a36d400
> > > Simplified sendfile(SF_NODISKIO) usage.
> > > 
> > > Starting with FreeBSD 11, there is no need to use AIO operations to preload
> > > data into cache for sendfile(SF_NODISKIO) to work.  Instead, sendfile()
> > > handles non-blocking loading data from disk by itself.  It still can, however,
> > > return EBUSY if a page is already being loaded (for example, by a different
> > > process).  If this happens, we now post an event for the next event loop
> > > iteration, so sendfile() is retried "after a short period", as manpage
> > > recommends.
> > > 
> > > The limit of the number of EBUSY tolerated without any progress is preserved,
> > > but now it does not result in an alert, since on an idle system event loop
> > > iteration might be very short and EBUSY can happen many times in a row.
> > > Instead, SF_NODISKIO is simply disabled for one call once the limit is
> > > reached.
> > > 
> > > With this change, sendfile(SF_NODISKIO) is now used automatically as long as
> > > sendfile() is enabled, and no longer requires "aio on;".
> > > 
> > > diff --git a/auto/os/freebsd b/auto/os/freebsd
> > > --- a/auto/os/freebsd
> > > +++ b/auto/os/freebsd
> > > @@ -44,12 +44,10 @@ if [ $osreldate -gt 300007 ]; then
> > >      CORE_SRCS="$CORE_SRCS $FREEBSD_SENDFILE_SRCS"
> > >  fi
> > >  
> > > -if [ $NGX_FILE_AIO = YES ]; then
> > > -    if [ $osreldate -gt 502103 ]; then
> > > -        echo " + sendfile()'s SF_NODISKIO found"
> > > +if [ $osreldate -gt 1100000 ]; then
> > > +    echo " + sendfile()'s SF_NODISKIO found"
> > >  
> > > -        have=NGX_HAVE_AIO_SENDFILE . auto/have
> > > -    fi
> > > +    have=NGX_HAVE_SENDFILE_NODISKIO . auto/have
> > >  fi
> > >  
> > >  # POSIX semaphores
> > 
> > We could check the exact __FreeBSD_version number 1100093 that was
> > at the time the new sendfile() appeared, which is more accurate.
> > 
> > https://cgit.freebsd.org/src/commit/?id=2bab0c553588
> > https://cgit.freebsd.org/src/tree/sys/sys/param.h?id=2bab0c553588#n48
> > 
> > Unfortunately, it was not bumped (same as with SF_NODISKIO in 5.2.1).
> 
> Yes, probably.  But the next version bump would be more 
> appropriate, changed to 1100093.  Similarly to 502103, which is 
> the next version after SF_NODISKIO introduction:
> 
> https://cgit.freebsd.org/src/commit/?id=b49d824e8bc1
> https://cgit.freebsd.org/src/tree/sys/sys/param.h?id=b49d824e8bc1
> 
> [...]
> 
> > > -static ngx_int_t
> > > -ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx, ngx_file_t *file)
> > > -{
> > > -    ngx_event_aio_t  *aio;
> > > -
> > > -    if (file->aio == NULL && ngx_file_aio_init(file, ctx->pool) != NGX_OK) {
> > > -        return NGX_ERROR;
> > > -    }
> > > -
> > > -    aio = file->aio;
> > > -
> > > -    aio->data = ctx->filter_ctx;
> > > -    aio->preload_handler = ctx->aio_preload;
> > > -
> > > -    return NGX_OK;
> > > -}
> > > -
> > > -#endif
> > > -
> > > -
> > >  static ngx_int_t
> > >  ngx_output_chain_add_copy(ngx_pool_t *pool, ngx_chain_t **chain,
> > >      ngx_chain_t *in)
> > 
> > After this change, ngx_file_aio_init() doesn't need to be external.
> > It is only used in ngx_file_aio_read() in corresponding ngx_*_aio_read.c.
> 
> In practice, yes.  In theory, it might be needed if we'll ever 
> implement other AIO operations, such as aio_write(), and/or will 
> reconsider thread-based interface to use the aio structure.  So I 
> would rather  preserve it as is, at least for now.
> 
> [...]
> 
> > >  ngx_chain_t *
> > >  ngx_freebsd_sendfile_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
> > >  {
> > > -    int               rc, flags;
> > > -    off_t             send, prev_send, sent;
> > > -    size_t            file_size;
> > > -    ssize_t           n;
> > > -    ngx_uint_t        eintr, eagain;
> > > -    ngx_err_t         err;
> > > -    ngx_buf_t        *file;
> > > -    ngx_event_t      *wev;
> > > -    ngx_chain_t      *cl;
> > > -    ngx_iovec_t       header, trailer;
> > > -    struct sf_hdtr    hdtr;
> > > -    struct iovec      headers[NGX_IOVS_PREALLOCATE];
> > > -    struct iovec      trailers[NGX_IOVS_PREALLOCATE];
> > > -#if (NGX_HAVE_AIO_SENDFILE)
> > > -    ngx_uint_t        ebusy;
> > > -    ngx_event_aio_t  *aio;
> > > +    int              rc, flags;
> > > +    off_t            send, prev_send, sent;
> > > +    size_t           file_size;
> > > +    ssize_t          n;
> > > +    ngx_uint_t       eintr, eagain;
> > > +    ngx_err_t        err;
> > > +    ngx_buf_t       *file;
> > > +    ngx_event_t     *wev;
> > > +    ngx_chain_t     *cl;
> > > +    ngx_iovec_t      header, trailer;
> > > +    struct sf_hdtr   hdtr;
> > > +    struct iovec     headers[NGX_IOVS_PREALLOCATE];
> > > +    struct iovec     trailers[NGX_IOVS_PREALLOCATE];
> > > +#if (NGX_HAVE_SENDFILE_NODISKIO)
> > > +    ngx_uint_t       ebusy;
> > >  #endif
> > 
> > After ngx_event_aio_t *aio variable removal,
> > this block could be placed under "eintr, eagain" line
> > (which by itself looks unsorted).
> > 
> > The remaining part looks good to me.
> 
> Yes, thanks.  Changed to:
> 
> diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
> --- a/src/os/unix/ngx_freebsd_sendfile_chain.c
> +++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
> @@ -36,18 +36,18 @@ ngx_freebsd_sendfile_chain(ngx_connectio
>      off_t            send, prev_send, sent;
>      size_t           file_size;
>      ssize_t          n;
> -    ngx_uint_t       eintr, eagain;
>      ngx_err_t        err;
>      ngx_buf_t       *file;
> +    ngx_uint_t       eintr, eagain;
> +#if (NGX_HAVE_SENDFILE_NODISKIO)
> +    ngx_uint_t       ebusy;
> +#endif
>      ngx_event_t     *wev;
>      ngx_chain_t     *cl;
>      ngx_iovec_t      header, trailer;
>      struct sf_hdtr   hdtr;
>      struct iovec     headers[NGX_IOVS_PREALLOCATE];
>      struct iovec     trailers[NGX_IOVS_PREALLOCATE];
> -#if (NGX_HAVE_SENDFILE_NODISKIO)
> -    ngx_uint_t       ebusy;
> -#endif
>  
>      wev = c->write;
>  
> 
> Full patch, updated:

[...]

> diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
> --- a/src/os/unix/ngx_freebsd_sendfile_chain.c
> +++ b/src/os/unix/ngx_freebsd_sendfile_chain.c

[...]

> @@ -199,7 +192,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
>                      eintr = 1;
>                      break;
>  
> -#if (NGX_HAVE_AIO_SENDFILE)
> +#if (NGX_HAVE_SENDFILE_NODISKIO)
>                  case NGX_EBUSY:
>                      ebusy = 1;
>                      break;
> @@ -258,35 +251,24 @@ ngx_freebsd_sendfile_chain(ngx_connectio
>              if (sent == 0) {
>                  c->busy_count++;
>  

Err, mismerge here.  Should be:

diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -245,7 +245,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
         in = ngx_chain_update_sent(in, sent);
 
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
 
         if (ebusy) {
             if (sent == 0) {


Full series updated:

# HG changeset patch
# User Maxim Dounin 
# Date 1639425539 -10800
#      Mon Dec 13 22:58:59 2021 +0300
# Node ID 42f1674706c9922fa910a7601af83307b3633b95
# Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
Removed "aio sendfile", deprecated since 1.7.11.

diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c
+++ b/src/http/ngx_http_core_module.c
@@ -4568,19 +4568,6 @@ ngx_http_core_set_aio(ngx_conf_t *cf, ng
 #endif
     }
 
-#if (NGX_HAVE_AIO_SENDFILE)
-
-    if (ngx_strcmp(value[1].data, "sendfile") == 0) {
-        clcf->aio = NGX_HTTP_AIO_ON;
-
-        ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
-                           "the \"sendfile\" parameter of "
-                           "the \"aio\" directive is deprecated");
-        return NGX_CONF_OK;
-    }
-
-#endif
-
     if (ngx_strncmp(value[1].data, "threads", 7) == 0
         && (value[1].len == 7 || value[1].data[7] == '='))
     {
# HG changeset patch
# User Maxim Dounin 
# Date 1639447743 -10800
#      Tue Dec 14 05:09:03 2021 +0300
# Node ID a149cd880ddd1473d21b66b85b9307bb17966df3
# Parent  42f1674706c9922fa910a7601af83307b3633b95
Simplified sendfile(SF_NODISKIO) usage.

Starting with FreeBSD 11, there is no need to use AIO operations to preload
data into cache for sendfile(SF_NODISKIO) to work.  Instead, sendfile()
handles non-blocking loading data from disk by itself.  It still can, however,
return EBUSY if a page is already being loaded (for example, by a different
process).  If this happens, we now post an event for the next event loop
iteration, so sendfile() is retried "after a short period", as manpage
recommends.

The limit of the number of EBUSY tolerated without any progress is preserved,
but now it does not result in an alert, since on an idle system event loop
iteration might be very short and EBUSY can happen many times in a row.
Instead, SF_NODISKIO is simply disabled for one call once the limit is
reached.

With this change, sendfile(SF_NODISKIO) is now used automatically as long as
sendfile() is enabled, and no longer requires "aio on;".

diff --git a/auto/os/freebsd b/auto/os/freebsd
--- a/auto/os/freebsd
+++ b/auto/os/freebsd
@@ -44,12 +44,10 @@ if [ $osreldate -gt 300007 ]; then
     CORE_SRCS="$CORE_SRCS $FREEBSD_SENDFILE_SRCS"
 fi
 
-if [ $NGX_FILE_AIO = YES ]; then
-    if [ $osreldate -gt 502103 ]; then
-        echo " + sendfile()'s SF_NODISKIO found"
+if [ $osreldate -gt 1100093 ]; then
+    echo " + sendfile()'s SF_NODISKIO found"
 
-        have=NGX_HAVE_AIO_SENDFILE . auto/have
-    fi
+    have=NGX_HAVE_SENDFILE_NODISKIO . auto/have
 fi
 
 # POSIX semaphores
diff --git a/src/core/ngx_buf.h b/src/core/ngx_buf.h
--- a/src/core/ngx_buf.h
+++ b/src/core/ngx_buf.h
@@ -90,9 +90,6 @@ struct ngx_output_chain_ctx_s {
 
 #if (NGX_HAVE_FILE_AIO || NGX_COMPAT)
     ngx_output_chain_aio_pt      aio_handler;
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
-    ssize_t                    (*aio_preload)(ngx_buf_t *file);
-#endif
 #endif
 
 #if (NGX_THREADS || NGX_COMPAT)
diff --git a/src/core/ngx_connection.h b/src/core/ngx_connection.h
--- a/src/core/ngx_connection.h
+++ b/src/core/ngx_connection.h
@@ -185,7 +185,7 @@ struct ngx_connection_s {
 
     unsigned            need_last_buf:1;
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
+#if (NGX_HAVE_SENDFILE_NODISKIO || NGX_COMPAT)
     unsigned            busy_count:2;
 #endif
 
diff --git a/src/core/ngx_module.h b/src/core/ngx_module.h
--- a/src/core/ngx_module.h
+++ b/src/core/ngx_module.h
@@ -41,7 +41,7 @@
 #define NGX_MODULE_SIGNATURE_3   "0"
 #endif
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
+#if (NGX_HAVE_SENDFILE_NODISKIO || NGX_COMPAT)
 #define NGX_MODULE_SIGNATURE_4   "1"
 #else
 #define NGX_MODULE_SIGNATURE_4   "0"
diff --git a/src/core/ngx_output_chain.c b/src/core/ngx_output_chain.c
--- a/src/core/ngx_output_chain.c
+++ b/src/core/ngx_output_chain.c
@@ -29,10 +29,6 @@
 
 static ngx_inline ngx_int_t
     ngx_output_chain_as_is(ngx_output_chain_ctx_t *ctx, ngx_buf_t *buf);
-#if (NGX_HAVE_AIO_SENDFILE)
-static ngx_int_t ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx,
-    ngx_file_t *file);
-#endif
 static ngx_int_t ngx_output_chain_add_copy(ngx_pool_t *pool,
     ngx_chain_t **chain, ngx_chain_t *in);
 static ngx_int_t ngx_output_chain_align_file_buf(ngx_output_chain_ctx_t *ctx,
@@ -283,12 +279,6 @@ ngx_output_chain_as_is(ngx_output_chain_
         buf->in_file = 0;
     }
 
-#if (NGX_HAVE_AIO_SENDFILE)
-    if (ctx->aio_preload && buf->in_file) {
-        (void) ngx_output_chain_aio_setup(ctx, buf->file);
-    }
-#endif
-
     if (ctx->need_in_memory && !ngx_buf_in_memory(buf)) {
         return 0;
     }
@@ -301,28 +291,6 @@ ngx_output_chain_as_is(ngx_output_chain_
 }
 
 
-#if (NGX_HAVE_AIO_SENDFILE)
-
-static ngx_int_t
-ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx, ngx_file_t *file)
-{
-    ngx_event_aio_t  *aio;
-
-    if (file->aio == NULL && ngx_file_aio_init(file, ctx->pool) != NGX_OK) {
-        return NGX_ERROR;
-    }
-
-    aio = file->aio;
-
-    aio->data = ctx->filter_ctx;
-    aio->preload_handler = ctx->aio_preload;
-
-    return NGX_OK;
-}
-
-#endif
-
-
 static ngx_int_t
 ngx_output_chain_add_copy(ngx_pool_t *pool, ngx_chain_t **chain,
     ngx_chain_t *in)
diff --git a/src/event/ngx_event.h b/src/event/ngx_event.h
--- a/src/event/ngx_event.h
+++ b/src/event/ngx_event.h
@@ -147,10 +147,6 @@ struct ngx_event_aio_s {
 
     ngx_fd_t                   fd;
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
-    ssize_t                  (*preload_handler)(ngx_buf_t *file);
-#endif
-
 #if (NGX_HAVE_EVENTFD)
     int64_t                    res;
 #endif
diff --git a/src/http/ngx_http_copy_filter_module.c b/src/http/ngx_http_copy_filter_module.c
--- a/src/http/ngx_http_copy_filter_module.c
+++ b/src/http/ngx_http_copy_filter_module.c
@@ -19,10 +19,6 @@ typedef struct {
 static void ngx_http_copy_aio_handler(ngx_output_chain_ctx_t *ctx,
     ngx_file_t *file);
 static void ngx_http_copy_aio_event_handler(ngx_event_t *ev);
-#if (NGX_HAVE_AIO_SENDFILE)
-static ssize_t ngx_http_copy_aio_sendfile_preload(ngx_buf_t *file);
-static void ngx_http_copy_aio_sendfile_event_handler(ngx_event_t *ev);
-#endif
 #endif
 #if (NGX_THREADS)
 static ngx_int_t ngx_http_copy_thread_handler(ngx_thread_task_t *task,
@@ -128,9 +124,6 @@ ngx_http_copy_filter(ngx_http_request_t 
 #if (NGX_HAVE_FILE_AIO)
         if (ngx_file_aio && clcf->aio == NGX_HTTP_AIO_ON) {
             ctx->aio_handler = ngx_http_copy_aio_handler;
-#if (NGX_HAVE_AIO_SENDFILE)
-            ctx->aio_preload = ngx_http_copy_aio_sendfile_preload;
-#endif
         }
 #endif
 
@@ -207,81 +200,6 @@ ngx_http_copy_aio_event_handler(ngx_even
     ngx_http_run_posted_requests(c);
 }
 
-
-#if (NGX_HAVE_AIO_SENDFILE)
-
-static ssize_t
-ngx_http_copy_aio_sendfile_preload(ngx_buf_t *file)
-{
-    ssize_t                  n;
-    static u_char            buf[1];
-    ngx_event_aio_t         *aio;
-    ngx_http_request_t      *r;
-    ngx_output_chain_ctx_t  *ctx;
-
-    aio = file->file->aio;
-    r = aio->data;
-
-    if (r->aio) {
-        /*
-         * tolerate sendfile() calls if another operation is already
-         * running; this can happen due to subrequests, multiple calls
-         * of the next body filter from a filter, or in HTTP/2 due to
-         * a write event on the main connection
-         */
-
-        return NGX_AGAIN;
-    }
-
-    n = ngx_file_aio_read(file->file, buf, 1, file->file_pos, NULL);
-
-    if (n == NGX_AGAIN) {
-        aio->handler = ngx_http_copy_aio_sendfile_event_handler;
-
-        r->main->blocked++;
-        r->aio = 1;
-
-        ctx = ngx_http_get_module_ctx(r, ngx_http_copy_filter_module);
-        ctx->aio = 1;
-    }
-
-    return n;
-}
-
-
-static void
-ngx_http_copy_aio_sendfile_event_handler(ngx_event_t *ev)
-{
-    ngx_event_aio_t     *aio;
-    ngx_connection_t    *c;
-    ngx_http_request_t  *r;
-
-    aio = ev->data;
-    r = aio->data;
-    c = r->connection;
-
-    r->main->blocked--;
-    r->aio = 0;
-    ev->complete = 0;
-
-#if (NGX_HTTP_V2)
-
-    if (r->stream) {
-        /*
-         * for HTTP/2, update write event to make sure processing will
-         * reach the main connection to handle sendfile() preload
-         */
-
-        c->write->ready = 1;
-        c->write->active = 0;
-    }
-
-#endif
-
-    c->write->handler(c->write);
-}
-
-#endif
 #endif
 
 
diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -32,23 +32,22 @@
 ngx_chain_t *
 ngx_freebsd_sendfile_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
 {
-    int               rc, flags;
-    off_t             send, prev_send, sent;
-    size_t            file_size;
-    ssize_t           n;
-    ngx_uint_t        eintr, eagain;
-    ngx_err_t         err;
-    ngx_buf_t        *file;
-    ngx_event_t      *wev;
-    ngx_chain_t      *cl;
-    ngx_iovec_t       header, trailer;
-    struct sf_hdtr    hdtr;
-    struct iovec      headers[NGX_IOVS_PREALLOCATE];
-    struct iovec      trailers[NGX_IOVS_PREALLOCATE];
-#if (NGX_HAVE_AIO_SENDFILE)
-    ngx_uint_t        ebusy;
-    ngx_event_aio_t  *aio;
+    int              rc, flags;
+    off_t            send, prev_send, sent;
+    size_t           file_size;
+    ssize_t          n;
+    ngx_err_t        err;
+    ngx_buf_t       *file;
+    ngx_uint_t       eintr, eagain;
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+    ngx_uint_t       ebusy;
 #endif
+    ngx_event_t     *wev;
+    ngx_chain_t     *cl;
+    ngx_iovec_t      header, trailer;
+    struct sf_hdtr   hdtr;
+    struct iovec     headers[NGX_IOVS_PREALLOCATE];
+    struct iovec     trailers[NGX_IOVS_PREALLOCATE];
 
     wev = c->write;
 
@@ -77,11 +76,6 @@ ngx_freebsd_sendfile_chain(ngx_connectio
     eagain = 0;
     flags = 0;
 
-#if (NGX_HAVE_AIO_SENDFILE && NGX_SUPPRESS_WARN)
-    aio = NULL;
-    file = NULL;
-#endif
-
     header.iovs = headers;
     header.nalloc = NGX_IOVS_PREALLOCATE;
 
@@ -90,7 +84,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
     for ( ;; ) {
         eintr = 0;
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
         ebusy = 0;
 #endif
         prev_send = send;
@@ -179,9 +173,8 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
             sent = 0;
 
-#if (NGX_HAVE_AIO_SENDFILE)
-            aio = file->file->aio;
-            flags = (aio && aio->preload_handler) ? SF_NODISKIO : 0;
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+            flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
 #endif
 
             rc = sendfile(file->file->fd, c->fd, file->file_pos,
@@ -199,7 +192,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
                     eintr = 1;
                     break;
 
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
                 case NGX_EBUSY:
                     ebusy = 1;
                     break;
@@ -258,35 +251,24 @@ ngx_freebsd_sendfile_chain(ngx_connectio
             if (sent == 0) {
                 c->busy_count++;
 
-                if (c->busy_count > 2) {
-                    ngx_log_error(NGX_LOG_ALERT, c->log, 0,
-                                  "sendfile(%V) returned busy again",
-                                  &file->file->name);
-
-                    c->busy_count = 0;
-                    aio->preload_handler = NULL;
-
-                    send = prev_send;
-                    continue;
-                }
+                ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
+                               "sendfile() busy, count:%d", c->busy_count);
 
             } else {
                 c->busy_count = 0;
             }
 
-            n = aio->preload_handler(file);
-
-            if (n > 0) {
-                send = prev_send + sent;
-                continue;
+            if (wev->posted) {
+                ngx_delete_posted_event(wev);
             }
 
+            ngx_post_event(wev, &ngx_posted_next_events);
+
+            wev->ready = 0;
             return in;
         }
 
-        if (flags == SF_NODISKIO) {
-            c->busy_count = 0;
-        }
+        c->busy_count = 0;
 
 #endif
 
# HG changeset patch
# User Maxim Dounin 
# Date 1639493908 -10800
#      Tue Dec 14 17:58:28 2021 +0300
# Node ID 7ff2d0ce6b28d00782c3f9d471b71f85a13c8b00
# Parent  a149cd880ddd1473d21b66b85b9307bb17966df3
SSL: SSL_sendfile(SF_NODISKIO) support.

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -2942,7 +2942,7 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
 {
 #ifdef BIO_get_ktls_send
 
-    int        sslerr;
+    int        sslerr, flags;
     ssize_t    n;
     ngx_err_t  err;
 
@@ -2954,8 +2954,14 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
 
     ngx_set_errno(0);
 
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+    flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
+#else
+    flags = 0;
+#endif
+
     n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,
-                     size, 0);
+                     size, flags);
 
     ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %d", n);
 
@@ -2974,6 +2980,10 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
             ngx_post_event(c->read, &ngx_posted_events);
         }
 
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+        c->busy_count = 0;
+#endif
+
         c->sent += n;
 
         return n;
@@ -3038,6 +3048,23 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
             ngx_post_event(c->read, &ngx_posted_events);
         }
 
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+
+        if (ngx_errno == EBUSY) {
+            c->busy_count++;
+
+            ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
+                           "SSL_sendfile() busy, count:%d", c->busy_count);
+
+            if (c->write->posted) {
+                ngx_delete_posted_event(c->write);
+            }
+
+            ngx_post_event(c->write, &ngx_posted_next_events);
+        }
+
+#endif
+
         c->write->ready = 0;
         return NGX_AGAIN;
     }
diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -245,7 +245,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
         in = ngx_chain_update_sent(in, sent);
 
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
 
         if (ebusy) {
             if (sent == 0) {
# HG changeset patch
# User Maxim Dounin 
# Date 1639493913 -10800
#      Tue Dec 14 17:58:33 2021 +0300
# Node ID fc13e8123bd797b3f5221988fc5e35654bd588f0
# Parent  7ff2d0ce6b28d00782c3f9d471b71f85a13c8b00
Support for sendfile(SF_NOCACHE).

The SF_NOCACHE flag, introduced in FreeBSD 11 along with the new non-blocking
sendfile() implementation by glebius@, makes it possible to use sendfile()
along with the "directio" directive.

diff --git a/src/core/ngx_output_chain.c b/src/core/ngx_output_chain.c
--- a/src/core/ngx_output_chain.c
+++ b/src/core/ngx_output_chain.c
@@ -256,10 +256,6 @@ ngx_output_chain_as_is(ngx_output_chain_
     }
 #endif
 
-    if (buf->in_file && buf->file->directio) {
-        return 0;
-    }
-
     sendfile = ctx->sendfile;
 
 #if (NGX_SENDFILE_LIMIT)
@@ -270,6 +266,19 @@ ngx_output_chain_as_is(ngx_output_chain_
 
 #endif
 
+#if !(NGX_HAVE_SENDFILE_NODISKIO)
+
+    /*
+     * With DIRECTIO, disable sendfile() unless sendfile(SF_NOCACHE)
+     * is available.
+     */
+
+    if (buf->in_file && buf->file->directio) {
+        sendfile = 0;
+    }
+
+#endif
+
     if (!sendfile) {
 
         if (!ngx_buf_in_memory(buf)) {
diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -2955,7 +2955,13 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
     ngx_set_errno(0);
 
 #if (NGX_HAVE_SENDFILE_NODISKIO)
+
     flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
+
+    if (file->file->directio) {
+        flags |= SF_NOCACHE;
+    }
+
 #else
     flags = 0;
 #endif
diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -174,7 +174,13 @@ ngx_freebsd_sendfile_chain(ngx_connectio
             sent = 0;
 
 #if (NGX_HAVE_SENDFILE_NODISKIO)
+
             flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
+
+            if (file->file->directio) {
+                flags |= SF_NOCACHE;
+            }
+
 #endif
 
             rc = sendfile(file->file->fd, c->fd, file->file_pos,

-- 
Maxim Dounin
http://mdounin.ru/

From mdounin at mdounin.ru  Tue Dec 14 16:09:09 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 14 Dec 2021 19:09:09 +0300
Subject: [PATCH 0 of 4] PCRE2 support
In-Reply-To: 
References: 
 
Message-ID: 

Hello!

On Tue, Dec 14, 2021 at 03:09:43PM +0900, Tatsuhiko Kubo wrote:

> Hello,
> 
> I'm trying these patches and the build of nginx with the option
> "--with-pcre-jit" fails.
> 
> $ cd nginx-1.21.4 # with applied patches
> $ ./configure --with-pcre=../pcre2/pcre2-10.39 --with-pcre-jit
> $ make
> ...
> cc -c -pipe  -O -Wall -Wextra -Wpointer-arith
> -Wconditional-uninitialized -Wno-unused-parameter
> -Wno-deprecated-declarations -Werror -g  -I src/core -I src/event -I
> src/event/modules -I src/os/unix -I ../pcre2/pcre2-10.39/src/ -I objs
> \
>                 -o objs/src/core/ngx_regex.o \
>                 src/core/ngx_regex.c
> src/core/ngx_regex.c:590:15: error: use of undeclared identifier
> 'PCRE_STUDY_JIT_COMPILE'
>         opt = PCRE_STUDY_JIT_COMPILE;

Thanks, looks like a mismerge in the last patch at some point during
development.  The following patch should fix this:

diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
--- a/src/core/ngx_regex.c
+++ b/src/core/ngx_regex.c
@@ -585,10 +585,16 @@ ngx_regex_module_init(ngx_cycle_t *cycle
 
     rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
 
-#if (NGX_HAVE_PCRE_JIT)
+#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
+
     if (rcf->pcre_jit) {
+#if (NGX_PCRE2)
+        opt = 1;
+#else
         opt = PCRE_STUDY_JIT_COMPILE;
+#endif
     }
+
 #endif
 
     ngx_regex_malloc_init(cycle->pool);


Full series updated:

# HG changeset patch
# User Maxim Dounin 
# Date 1639495851 -10800
#      Tue Dec 14 18:30:51 2021 +0300
# Node ID 4f979a9f2a68b25b8b2ce8a0bd15671095f6c327
# Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
Core: fixed ngx_pcre_studies cleanup.

If a configuration parsing fails for some reason, ngx_regex_module_init()
is not called, and ngx_pcre_studies remained set despite the fact that
the pool it was allocated from is already freed.  This might result in
a segmentation fault during runtime regular expression compilation, such
as in SSI, for example, in the single process mode, or if a worker process
dies and respawn from a master process in such an inconsistent state.

Fix is to clear ngx_pcre_studies from the pool cleanup handler (which is
anyway used to free JIT-compiled patterns).

diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
--- a/src/core/ngx_regex.c
+++ b/src/core/ngx_regex.c
@@ -10,15 +10,14 @@
 
 
 typedef struct {
-    ngx_flag_t  pcre_jit;
+    ngx_flag_t   pcre_jit;
+    ngx_list_t  *studies;
 } ngx_regex_conf_t;
 
 
 static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
 static void ngx_libc_cdecl ngx_regex_free(void *p);
-#if (NGX_HAVE_PCRE_JIT)
-static void ngx_pcre_free_studies(void *data);
-#endif
+static void ngx_regex_cleanup(void *data);
 
 static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
 
@@ -248,18 +247,17 @@ ngx_regex_free(void *p)
 }
 
 
+static void
+ngx_regex_cleanup(void *data)
+{
 #if (NGX_HAVE_PCRE_JIT)
-
-static void
-ngx_pcre_free_studies(void *data)
-{
-    ngx_list_t *studies = data;
+    ngx_regex_conf_t *rcf = data;
 
     ngx_uint_t        i;
     ngx_list_part_t  *part;
     ngx_regex_elt_t  *elts;
 
-    part = &studies->part;
+    part = &rcf->studies->part;
     elts = part->elts;
 
     for (i = 0; /* void */ ; i++) {
@@ -274,56 +272,50 @@ ngx_pcre_free_studies(void *data)
             i = 0;
         }
 
-        if (elts[i].regex->extra != NULL) {
-            pcre_free_study(elts[i].regex->extra);
-        }
-    }
-}
-
-#endif
-
-
-static ngx_int_t
-ngx_regex_module_init(ngx_cycle_t *cycle)
-{
-    int               opt;
-    const char       *errstr;
-    ngx_uint_t        i;
-    ngx_list_part_t  *part;
-    ngx_regex_elt_t  *elts;
-
-    opt = 0;
-
-#if (NGX_HAVE_PCRE_JIT)
-    {
-    ngx_regex_conf_t    *rcf;
-    ngx_pool_cleanup_t  *cln;
-
-    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
-
-    if (rcf->pcre_jit) {
-        opt = PCRE_STUDY_JIT_COMPILE;
-
         /*
          * The PCRE JIT compiler uses mmap for its executable codes, so we
          * have to explicitly call the pcre_free_study() function to free
          * this memory.
          */
 
-        cln = ngx_pool_cleanup_add(cycle->pool, 0);
-        if (cln == NULL) {
-            return NGX_ERROR;
+        if (elts[i].regex->extra != NULL) {
+            pcre_free_study(elts[i].regex->extra);
         }
+    }
+#endif
 
-        cln->handler = ngx_pcre_free_studies;
-        cln->data = ngx_pcre_studies;
-    }
+    /*
+     * On configuration parsing errors ngx_regex_module_init() will not
+     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
+     */
+
+    ngx_pcre_studies = NULL;
+}
+
+
+static ngx_int_t
+ngx_regex_module_init(ngx_cycle_t *cycle)
+{
+    int                opt;
+    const char        *errstr;
+    ngx_uint_t         i;
+    ngx_list_part_t   *part;
+    ngx_regex_elt_t   *elts;
+    ngx_regex_conf_t  *rcf;
+
+    opt = 0;
+
+    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
+
+#if (NGX_HAVE_PCRE_JIT)
+    if (rcf->pcre_jit) {
+        opt = PCRE_STUDY_JIT_COMPILE;
     }
 #endif
 
     ngx_regex_malloc_init(cycle->pool);
 
-    part = &ngx_pcre_studies->part;
+    part = &rcf->studies->part;
     elts = part->elts;
 
     for (i = 0; /* void */ ; i++) {
@@ -374,7 +366,8 @@ ngx_regex_module_init(ngx_cycle_t *cycle
 static void *
 ngx_regex_create_conf(ngx_cycle_t *cycle)
 {
-    ngx_regex_conf_t  *rcf;
+    ngx_regex_conf_t    *rcf;
+    ngx_pool_cleanup_t  *cln;
 
     rcf = ngx_pcalloc(cycle->pool, sizeof(ngx_regex_conf_t));
     if (rcf == NULL) {
@@ -383,11 +376,21 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
 
     rcf->pcre_jit = NGX_CONF_UNSET;
 
-    ngx_pcre_studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
-    if (ngx_pcre_studies == NULL) {
+    cln = ngx_pool_cleanup_add(cycle->pool, 0);
+    if (cln == NULL) {
         return NULL;
     }
 
+    cln->handler = ngx_regex_cleanup;
+    cln->data = rcf;
+
+    rcf->studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
+    if (rcf->studies == NULL) {
+        return NULL;
+    }
+
+    ngx_pcre_studies = rcf->studies;
+
     return rcf;
 }
 
# HG changeset patch
# User Maxim Dounin 
# Date 1639497274 -10800
#      Tue Dec 14 18:54:34 2021 +0300
# Node ID 8f00e80ade356c768a678c44711e8c2a7223ceef
# Parent  4f979a9f2a68b25b8b2ce8a0bd15671095f6c327
Core: ngx_regex.c style cleanup.

Notably, ngx_pcre_pool and ngx_pcre_studies are renamed to ngx_regex_pool
and ngx_regex_studies, respectively.

diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
--- a/src/core/ngx_regex.c
+++ b/src/core/ngx_regex.c
@@ -64,8 +64,8 @@ ngx_module_t  ngx_regex_module = {
 };
 
 
-static ngx_pool_t  *ngx_pcre_pool;
-static ngx_list_t  *ngx_pcre_studies;
+static ngx_pool_t  *ngx_regex_pool;
+static ngx_list_t  *ngx_regex_studies;
 
 
 void
@@ -79,14 +79,14 @@ ngx_regex_init(void)
 static ngx_inline void
 ngx_regex_malloc_init(ngx_pool_t *pool)
 {
-    ngx_pcre_pool = pool;
+    ngx_regex_pool = pool;
 }
 
 
 static ngx_inline void
 ngx_regex_malloc_done(void)
 {
-    ngx_pcre_pool = NULL;
+    ngx_regex_pool = NULL;
 }
 
 
@@ -112,13 +112,13 @@ ngx_regex_compile(ngx_regex_compile_t *r
            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
                               "pcre_compile() failed: %s in \"%V\"",
                                errstr, &rc->pattern)
-                      - rc->err.data;
+                         - rc->err.data;
 
         } else {
            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
                               "pcre_compile() failed: %s in \"%V\" at \"%s\"",
                                errstr, &rc->pattern, rc->pattern.data + erroff)
-                      - rc->err.data;
+                         - rc->err.data;
         }
 
         return NGX_ERROR;
@@ -133,8 +133,8 @@ ngx_regex_compile(ngx_regex_compile_t *r
 
     /* do not study at runtime */
 
-    if (ngx_pcre_studies != NULL) {
-        elt = ngx_list_push(ngx_pcre_studies);
+    if (ngx_regex_studies != NULL) {
+        elt = ngx_list_push(ngx_regex_studies);
         if (elt == NULL) {
             goto nomem;
         }
@@ -229,11 +229,8 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
 static void * ngx_libc_cdecl
 ngx_regex_malloc(size_t size)
 {
-    ngx_pool_t      *pool;
-    pool = ngx_pcre_pool;
-
-    if (pool) {
-        return ngx_palloc(pool, size);
+    if (ngx_regex_pool) {
+        return ngx_palloc(ngx_regex_pool, size);
     }
 
     return NULL;
@@ -286,10 +283,10 @@ ngx_regex_cleanup(void *data)
 
     /*
      * On configuration parsing errors ngx_regex_module_init() will not
-     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
+     * be called.  Make sure ngx_regex_studies is properly cleared anyway.
      */
 
-    ngx_pcre_studies = NULL;
+    ngx_regex_studies = NULL;
 }
 
 
@@ -357,7 +354,7 @@ ngx_regex_module_init(ngx_cycle_t *cycle
 
     ngx_regex_malloc_done();
 
-    ngx_pcre_studies = NULL;
+    ngx_regex_studies = NULL;
 
     return NGX_OK;
 }
@@ -389,7 +386,7 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
         return NULL;
     }
 
-    ngx_pcre_studies = rcf->studies;
+    ngx_regex_studies = rcf->studies;
 
     return rcf;
 }
# HG changeset patch
# User Maxim Dounin 
# Date 1639497275 -10800
#      Tue Dec 14 18:54:35 2021 +0300
# Node ID 0a11d3d92a5d4c1ddcfdd1f07423d59b7c2aa863
# Parent  8f00e80ade356c768a678c44711e8c2a7223ceef
Configure: simplified PCRE compilation.

Removed ICC-specific PCRE optimizations which tried to link with PCRE
object files instead of the library.  Made compiler-specific code
minimal.

diff --git a/auto/lib/pcre/conf b/auto/lib/pcre/conf
--- a/auto/lib/pcre/conf
+++ b/auto/lib/pcre/conf
@@ -4,81 +4,24 @@
 
 
 if [ $PCRE != NONE ]; then
+
+    have=NGX_PCRE . auto/have
+
+    if [ "$NGX_PLATFORM" = win32 ]; then
+        have=PCRE_STATIC . auto/have
+    fi
+
     CORE_INCS="$CORE_INCS $PCRE"
+    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
 
     case "$NGX_CC_NAME" in
 
         msvc | owc | bcc)
-            have=NGX_PCRE . auto/have
-            have=PCRE_STATIC . auto/have
-            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
             LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
             CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
         ;;
 
-        icc)
-            have=NGX_PCRE . auto/have
-            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
-
-            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
-
-            echo $ngx_n "checking for PCRE library ...$ngx_c"
-
-            if [ -f $PCRE/pcre.h ]; then
-                ngx_pcre_ver=`grep PCRE_MAJOR $PCRE/pcre.h \
-                              | sed -e 's/^.*PCRE_MAJOR.* \(.*\)$/\1/'`
-
-            else if [ -f $PCRE/configure.in ]; then
-                ngx_pcre_ver=`grep PCRE_MAJOR= $PCRE/configure.in \
-                              | sed -e 's/^.*=\(.*\)$/\1/'`
-
-            else
-                ngx_pcre_ver=`grep pcre_major, $PCRE/configure.ac \
-                              | sed -e 's/^.*pcre_major,.*\[\(.*\)\].*$/\1/'`
-            fi
-            fi
-
-            echo " $ngx_pcre_ver major version found"
-
-            # to allow -ipo optimization we link with the *.o but not library
-
-            case "$ngx_pcre_ver" in
-                4|5)
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre.o"
-                ;;
-
-                6)
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
-                ;;
-
-                *)
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_newline.o"
-                ;;
-
-            esac
-        ;;
-
         *)
-            have=NGX_PCRE . auto/have
-
-            if [ "$NGX_PLATFORM" = win32 ]; then
-                have=PCRE_STATIC . auto/have
-            fi
-
-            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
             LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
             CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
         ;;
# HG changeset patch
# User Maxim Dounin 
# Date 1639497637 -10800
#      Tue Dec 14 19:00:37 2021 +0300
# Node ID 6915b92704ceca0030ea6d9e19f37e868a8d2303
# Parent  0a11d3d92a5d4c1ddcfdd1f07423d59b7c2aa863
PCRE2 library support.

The PCRE2 library is now used by default if found, instead of the
original PCRE library.  If needed for some reason, this can be disabled
with the --without-pcre2 configure option.

To make it possible to specify paths to the library and include files
via --with-cc-opt / --with-ld-opt, the library is first tested without
any additional paths and options.  If this fails, the pcre2-config script
is used.

Similarly to the original PCRE library, it is now possible to build PCRE2
from sources with nginx configure, by using the --with-pcre= option.
It automatically detects if PCRE or PCRE2 sources are provided.

Note that compiling PCRE2 10.33 and later requires inttypes.h.  When
compiling on Windows with MSVC, inttypes.h is only available starting
with MSVC 2013.  In older versions some replacement needs to be provided
("echo '#include ' > pcre2-10.xx/src/inttypes.h" is good enough
for MSVC 2010).

The interface on nginx side remains unchanged.

diff --git a/auto/lib/pcre/conf b/auto/lib/pcre/conf
--- a/auto/lib/pcre/conf
+++ b/auto/lib/pcre/conf
@@ -5,29 +5,61 @@
 
 if [ $PCRE != NONE ]; then
 
-    have=NGX_PCRE . auto/have
+    if [ -f $PCRE/src/pcre2.h.generic ]; then
+
+        PCRE_LIBRARY=PCRE2
+
+        have=NGX_PCRE . auto/have
+        have=NGX_PCRE2 . auto/have
+
+        if [ "$NGX_PLATFORM" = win32 ]; then
+            have=PCRE2_STATIC . auto/have
+        fi
+
+        CORE_INCS="$CORE_INCS $PCRE/src/"
+        CORE_DEPS="$CORE_DEPS $PCRE/src/pcre2.h"
 
-    if [ "$NGX_PLATFORM" = win32 ]; then
-        have=PCRE_STATIC . auto/have
-    fi
+        case "$NGX_CC_NAME" in
+
+            msvc)
+                LINK_DEPS="$LINK_DEPS $PCRE/src/pcre2-8.lib"
+                CORE_LIBS="$CORE_LIBS $PCRE/src/pcre2-8.lib"
+            ;;
 
-    CORE_INCS="$CORE_INCS $PCRE"
-    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
+            *)
+                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre2-8.a"
+                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre2-8.a"
+            ;;
 
-    case "$NGX_CC_NAME" in
+        esac
 
-        msvc | owc | bcc)
-            LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
-            CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
-        ;;
+    else
+
+        PCRE_LIBRARY=PCRE
+
+        have=NGX_PCRE . auto/have
+
+        if [ "$NGX_PLATFORM" = win32 ]; then
+            have=PCRE_STATIC . auto/have
+        fi
+
+        CORE_INCS="$CORE_INCS $PCRE"
+        CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
 
-        *)
-            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
-            CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
-        ;;
+        case "$NGX_CC_NAME" in
+
+            msvc | owc | bcc)
+                LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
+                CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
+            ;;
 
-    esac
+            *)
+                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
+                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
+            ;;
 
+        esac
+    fi
 
     if [ $PCRE_JIT = YES ]; then
         have=NGX_HAVE_PCRE_JIT . auto/have
@@ -37,8 +69,48 @@ if [ $PCRE != NONE ]; then
 else
 
     if [ "$NGX_PLATFORM" != win32 ]; then
+        PCRE=NO
+    fi
 
-        PCRE=NO
+    if [ $PCRE = NO -a $PCRE2 != DISABLED ]; then
+
+        ngx_feature="PCRE2 library"
+        ngx_feature_name="NGX_PCRE2"
+        ngx_feature_run=no
+        ngx_feature_incs="#define PCRE2_CODE_UNIT_WIDTH 8
+                          #include "
+        ngx_feature_path=
+        ngx_feature_libs="-lpcre2-8"
+        ngx_feature_test="pcre2_code *re;
+                          re = pcre2_compile(NULL, 0, 0, NULL, NULL, NULL);
+                          if (re == NULL) return 1"
+        . auto/feature
+
+        if [ $ngx_found = no ]; then
+
+            # pcre2-config
+
+            ngx_pcre2_prefix=`pcre2-config --prefix 2>/dev/null`
+
+            if [ -n "$ngx_pcre2_prefix" ]; then
+                ngx_feature="PCRE2 library in $ngx_pcre2_prefix"
+                ngx_feature_path=`pcre2-config --cflags \
+                                  | sed -n -e 's/.*-I *\([^ ][^ ]*\).*/\1/p'`
+                ngx_feature_libs=`pcre2-config --libs8`
+                . auto/feature
+            fi
+        fi
+
+        if [ $ngx_found = yes ]; then
+            have=NGX_PCRE . auto/have
+            CORE_INCS="$CORE_INCS $ngx_feature_path"
+            CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
+            PCRE=YES
+            PCRE_LIBRARY=PCRE2
+        fi
+    fi
+
+    if [ $PCRE = NO ]; then
 
         ngx_feature="PCRE library"
         ngx_feature_name="NGX_PCRE"
@@ -114,6 +186,7 @@ else
             CORE_INCS="$CORE_INCS $ngx_feature_path"
             CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
             PCRE=YES
+            PCRE_LIBRARY=PCRE
         fi
 
         if [ $PCRE = YES ]; then
diff --git a/auto/lib/pcre/make b/auto/lib/pcre/make
--- a/auto/lib/pcre/make
+++ b/auto/lib/pcre/make
@@ -3,36 +3,138 @@
 # Copyright (C) Nginx, Inc.
 
 
-case "$NGX_CC_NAME" in
+if [ $PCRE_LIBRARY = PCRE2 ]; then
+
+    # PCRE2
+
+    if [ $NGX_CC_NAME = msvc ]; then
+
+        # With PCRE2, it is not possible to compile all sources.
+        # Since list of source files changes between versions, we
+        # test files which might not be present.
 
-    msvc)
-        ngx_makefile=makefile.msvc
-        ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
-        ngx_pcre="PCRE=\"$PCRE\""
-    ;;
+        ngx_pcre_srcs="pcre2_auto_possess.c \
+                       pcre2_chartables.c \
+                       pcre2_compile.c \
+                       pcre2_config.c \
+                       pcre2_context.c \
+                       pcre2_dfa_match.c \
+                       pcre2_error.c \
+                       pcre2_jit_compile.c \
+                       pcre2_maketables.c \
+                       pcre2_match.c \
+                       pcre2_match_data.c \
+                       pcre2_newline.c \
+                       pcre2_ord2utf.c \
+                       pcre2_pattern_info.c \
+                       pcre2_string_utils.c \
+                       pcre2_study.c \
+                       pcre2_substitute.c \
+                       pcre2_substring.c \
+                       pcre2_tables.c \
+                       pcre2_ucd.c \
+                       pcre2_valid_utf.c \
+                       pcre2_xclass.c"
+
+        ngx_pcre_test="pcre2_convert.c \
+                       pcre2_extuni.c \
+                       pcre2_find_bracket.c \
+                       pcre2_script_run.c \
+                       pcre2_serialize.c"
+
+        for ngx_src in $ngx_pcre_test
+        do
+            if [ -f $PCRE/src/$ngx_src ]; then
+                ngx_pcre_srcs="$ngx_pcre_srcs $ngx_src"
+            fi
+        done
 
-    owc)
-        ngx_makefile=makefile.owc
-        ngx_opt="CPU_OPT=\"$CPU_OPT\""
-        ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
-    ;;
+        ngx_pcre_objs=`echo $ngx_pcre_srcs \
+            | sed -e "s#\([^ ]*\.\)c#\1$ngx_objext#g"`
+
+        ngx_pcre_srcs=`echo $ngx_pcre_srcs \
+            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
+        ngx_pcre_objs=`echo $ngx_pcre_objs \
+            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
+
+        cat << END                                            >> $NGX_MAKEFILE
+
+PCRE_CFLAGS =	-O2 -Ob1 -Oi -Gs $LIBC $CPU_OPT
+PCRE_FLAGS =	-DHAVE_CONFIG_H -DPCRE2_STATIC -DPCRE2_CODE_UNIT_WIDTH=8 \\
+		-DHAVE_MEMMOVE
+
+PCRE_SRCS =	 $ngx_pcre_srcs
+PCRE_OBJS =	 $ngx_pcre_objs
+
+$PCRE/src/pcre2.h:
+	cd $PCRE/src \\
+	&& copy /y config.h.generic config.h \\
+	&& copy /y pcre2.h.generic pcre2.h \\
+	&& copy /y pcre2_chartables.c.dist pcre2_chartables.c
 
-    bcc)
-        ngx_makefile=makefile.bcc
-        ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
-        ngx_pcre=`echo \-DPCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
-    ;;
+$PCRE/src/pcre2-8.lib:	$PCRE/src/pcre2.h $NGX_MAKEFILE
+	cd $PCRE/src \\
+	&& cl -nologo -c \$(PCRE_CFLAGS) -I . \$(PCRE_FLAGS) \$(PCRE_SRCS) \\
+	&& link -lib -out:pcre2-8.lib -verbose:lib \$(PCRE_OBJS)
+
+END
+
+    else
+
+        cat << END                                            >> $NGX_MAKEFILE
+
+$PCRE/src/pcre2.h:	$PCRE/Makefile
 
-    *)
-        ngx_makefile=
-    ;;
+$PCRE/Makefile:	$NGX_MAKEFILE
+	cd $PCRE \\
+	&& if [ -f Makefile ]; then \$(MAKE) distclean; fi \\
+	&& CC="\$(CC)" CFLAGS="$PCRE_OPT" \\
+	./configure --disable-shared $PCRE_CONF_OPT
 
-esac
+$PCRE/.libs/libpcre2-8.a:	$PCRE/Makefile
+	cd $PCRE \\
+	&& \$(MAKE) libpcre2-8.la
+
+END
+
+    fi
 
 
-if [ -n "$ngx_makefile" ]; then
+else
+
+    # PCRE
+
+    case "$NGX_CC_NAME" in
+
+        msvc)
+            ngx_makefile=makefile.msvc
+            ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
+            ngx_pcre="PCRE=\"$PCRE\""
+        ;;
+
+        owc)
+            ngx_makefile=makefile.owc
+            ngx_opt="CPU_OPT=\"$CPU_OPT\""
+            ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
+        ;;
 
-    cat << END                                                >> $NGX_MAKEFILE
+        bcc)
+            ngx_makefile=makefile.bcc
+            ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
+            ngx_pcre=`echo \-DPCRE=\"$PCRE\" \
+                | sed -e "s/\//$ngx_regex_dirsep/g"`
+        ;;
+
+        *)
+            ngx_makefile=
+        ;;
+
+    esac
+
+
+    if [ -n "$ngx_makefile" ]; then
+
+        cat << END                                            >> $NGX_MAKEFILE
 
 `echo "$PCRE/pcre.lib:	$PCRE/pcre.h $NGX_MAKEFILE"			\
 	| sed -e "s/\//$ngx_regex_dirsep/g"`
@@ -43,9 +145,9 @@ if [ -n "$ngx_makefile" ]; then
 
 END
 
-else
+    else
 
-    cat << END                                                >> $NGX_MAKEFILE
+        cat << END                                            >> $NGX_MAKEFILE
 
 $PCRE/pcre.h:	$PCRE/Makefile
 
@@ -61,4 +163,6 @@ else
 
 END
 
+    fi
+
 fi
diff --git a/auto/options b/auto/options
--- a/auto/options
+++ b/auto/options
@@ -146,6 +146,7 @@ PCRE=NONE
 PCRE_OPT=
 PCRE_CONF_OPT=
 PCRE_JIT=NO
+PCRE2=YES
 
 USE_OPENSSL=NO
 OPENSSL=NONE
@@ -357,6 +358,7 @@ use the \"--with-mail_ssl_module\" optio
         --with-pcre=*)                   PCRE="$value"              ;;
         --with-pcre-opt=*)               PCRE_OPT="$value"          ;;
         --with-pcre-jit)                 PCRE_JIT=YES               ;;
+        --without-pcre2)                 PCRE2=DISABLED             ;;
 
         --with-openssl=*)                OPENSSL="$value"           ;;
         --with-openssl-opt=*)            OPENSSL_OPT="$value"       ;;
@@ -573,6 +575,7 @@ cat << END
   --with-pcre=DIR                    set path to PCRE library sources
   --with-pcre-opt=OPTIONS            set additional build options for PCRE
   --with-pcre-jit                    build PCRE with JIT compilation support
+  --without-pcre2                    do not use PCRE2 library
 
   --with-zlib=DIR                    set path to zlib library sources
   --with-zlib-opt=OPTIONS            set additional build options for zlib
diff --git a/auto/summary b/auto/summary
--- a/auto/summary
+++ b/auto/summary
@@ -16,9 +16,9 @@ if [ $USE_PCRE = DISABLED ]; then
 
 else
     case $PCRE in
-        YES)   echo "  + using system PCRE library" ;;
+        YES)   echo "  + using system $PCRE_LIBRARY library" ;;
         NONE)  echo "  + PCRE library is not used" ;;
-        *)     echo "  + using PCRE library: $PCRE" ;;
+        *)     echo "  + using $PCRE_LIBRARY library: $PCRE" ;;
     esac
 fi
 
diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
--- a/src/core/ngx_regex.c
+++ b/src/core/ngx_regex.c
@@ -15,8 +15,16 @@ typedef struct {
 } ngx_regex_conf_t;
 
 
+static ngx_inline void ngx_regex_malloc_init(ngx_pool_t *pool);
+static ngx_inline void ngx_regex_malloc_done(void);
+
+#if (NGX_PCRE2)
+static void * ngx_libc_cdecl ngx_regex_malloc(size_t size, void *data);
+static void ngx_libc_cdecl ngx_regex_free(void *p, void *data);
+#else
 static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
 static void ngx_libc_cdecl ngx_regex_free(void *p);
+#endif
 static void ngx_regex_cleanup(void *data);
 
 static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
@@ -64,15 +72,24 @@ ngx_module_t  ngx_regex_module = {
 };
 
 
-static ngx_pool_t  *ngx_regex_pool;
-static ngx_list_t  *ngx_regex_studies;
+static ngx_pool_t             *ngx_regex_pool;
+static ngx_list_t             *ngx_regex_studies;
+static ngx_uint_t              ngx_regex_direct_alloc;
+
+#if (NGX_PCRE2)
+static pcre2_compile_context  *ngx_regex_compile_context;
+static pcre2_match_data       *ngx_regex_match_data;
+static ngx_uint_t              ngx_regex_match_data_size;
+#endif
 
 
 void
 ngx_regex_init(void)
 {
+#if !(NGX_PCRE2)
     pcre_malloc = ngx_regex_malloc;
     pcre_free = ngx_regex_free;
+#endif
 }
 
 
@@ -80,6 +97,7 @@ static ngx_inline void
 ngx_regex_malloc_init(ngx_pool_t *pool)
 {
     ngx_regex_pool = pool;
+    ngx_regex_direct_alloc = (pool == NULL) ? 1 : 0;
 }
 
 
@@ -87,9 +105,146 @@ static ngx_inline void
 ngx_regex_malloc_done(void)
 {
     ngx_regex_pool = NULL;
+    ngx_regex_direct_alloc = 0;
 }
 
 
+#if (NGX_PCRE2)
+
+ngx_int_t
+ngx_regex_compile(ngx_regex_compile_t *rc)
+{
+    int                     n, errcode;
+    char                   *p;
+    u_char                  errstr[128];
+    size_t                  erroff;
+    pcre2_code             *re;
+    ngx_regex_elt_t        *elt;
+    pcre2_general_context  *gctx;
+    pcre2_compile_context  *cctx;
+
+    if (ngx_regex_compile_context == NULL) {
+        /*
+         * Allocte a compile context if not yet allocated.  This uses
+         * direct allocations from heap, so the result can be cached
+         * even at runtime.
+         */
+
+        ngx_regex_malloc_init(NULL);
+
+        gctx = pcre2_general_context_create(ngx_regex_malloc, ngx_regex_free,
+                                            NULL);
+        if (gctx == NULL) {
+            ngx_regex_malloc_done();
+            goto nomem;
+        }
+
+        cctx = pcre2_compile_context_create(gctx);
+        if (cctx == NULL) {
+            pcre2_general_context_free(gctx);
+            ngx_regex_malloc_done();
+            goto nomem;
+        }
+
+        ngx_regex_compile_context = cctx;
+
+        pcre2_general_context_free(gctx);
+        ngx_regex_malloc_done();
+    }
+
+    ngx_regex_malloc_init(rc->pool);
+
+    re = pcre2_compile(rc->pattern.data, rc->pattern.len,
+                       (uint32_t) rc->options, &errcode, &erroff,
+                       ngx_regex_compile_context);
+
+    /* ensure that there is no current pool */
+    ngx_regex_malloc_done();
+
+    if (re == NULL) {
+        pcre2_get_error_message(errcode, errstr, 128);
+
+        if ((size_t) erroff == rc->pattern.len) {
+            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                              "pcre2_compile() failed: %s in \"%V\"",
+                               errstr, &rc->pattern)
+                          - rc->err.data;
+
+        } else {
+            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                              "pcre2_compile() failed: %s in \"%V\" at \"%s\"",
+                               errstr, &rc->pattern, rc->pattern.data + erroff)
+                          - rc->err.data;
+        }
+
+        return NGX_ERROR;
+    }
+
+    rc->regex = re;
+
+    /* do not study at runtime */
+
+    if (ngx_regex_studies != NULL) {
+        elt = ngx_list_push(ngx_regex_studies);
+        if (elt == NULL) {
+            goto nomem;
+        }
+
+        elt->regex = rc->regex;
+        elt->name = rc->pattern.data;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_CAPTURECOUNT, &rc->captures);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_CAPTURECOUNT) failed: %d";
+        goto failed;
+    }
+
+    if (rc->captures == 0) {
+        return NGX_OK;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_NAMECOUNT, &rc->named_captures);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMECOUNT) failed: %d";
+        goto failed;
+    }
+
+    if (rc->named_captures == 0) {
+        return NGX_OK;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_NAMEENTRYSIZE, &rc->name_size);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMEENTRYSIZE) failed: %d";
+        goto failed;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_NAMETABLE, &rc->names);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMETABLE) failed: %d";
+        goto failed;
+    }
+
+    return NGX_OK;
+
+failed:
+
+    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len, p, &rc->pattern, n)
+                  - rc->err.data;
+    return NGX_ERROR;
+
+nomem:
+
+    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                               "regex \"%V\" compilation failed: no memory",
+                               &rc->pattern)
+                  - rc->err.data;
+    return NGX_ERROR;
+}
+
+#else
+
 ngx_int_t
 ngx_regex_compile(ngx_regex_compile_t *rc)
 {
@@ -192,6 +347,74 @@ nomem:
     return NGX_ERROR;
 }
 
+#endif
+
+
+#if (NGX_PCRE2)
+
+ngx_int_t
+ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures, ngx_uint_t size)
+{
+    size_t      *ov;
+    ngx_int_t    rc;
+    ngx_uint_t   n, i;
+
+    /*
+     * The pcre2_match() function might allocate memory for backtracking
+     * frames, typical allocations are from 40k and above.  So the allocator
+     * is configured to do direct allocations from heap during matching.
+     */
+
+    ngx_regex_malloc_init(NULL);
+
+    if (ngx_regex_match_data == NULL
+        || size > ngx_regex_match_data_size)
+    {
+        /*
+         * Allocate a match data if not yet allocated or smaller than
+         * needed.
+         */
+
+        if (ngx_regex_match_data) {
+            pcre2_match_data_free(ngx_regex_match_data);
+        }
+
+        ngx_regex_match_data_size = size;
+        ngx_regex_match_data = pcre2_match_data_create(size / 3, NULL);
+
+        if (ngx_regex_match_data == NULL) {
+            rc = PCRE2_ERROR_NOMEMORY;
+            goto failed;
+        }
+    }
+
+    rc = pcre2_match(re, s->data, s->len, 0, 0, ngx_regex_match_data, NULL);
+
+    if (rc < 0) {
+        goto failed;
+    }
+
+    n = pcre2_get_ovector_count(ngx_regex_match_data);
+    ov = pcre2_get_ovector_pointer(ngx_regex_match_data);
+
+    if (n > size / 3) {
+        n = size / 3;
+    }
+
+    for (i = 0; i < n; i++) {
+        captures[i * 2] = ov[i * 2];
+        captures[i * 2 + 1] = ov[i * 2 + 1];
+    }
+
+failed:
+
+    ngx_regex_malloc_done();
+
+    return rc;
+}
+
+#endif
+
 
 ngx_int_t
 ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log)
@@ -226,6 +449,35 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
 }
 
 
+#if (NGX_PCRE2)
+
+static void * ngx_libc_cdecl
+ngx_regex_malloc(size_t size, void *data)
+{
+    if (ngx_regex_pool) {
+        return ngx_palloc(ngx_regex_pool, size);
+    }
+
+    if (ngx_regex_direct_alloc) {
+        return ngx_alloc(size, ngx_cycle->log);
+    }
+
+    return NULL;
+}
+
+
+static void ngx_libc_cdecl
+ngx_regex_free(void *p, void *data)
+{
+    if (ngx_regex_direct_alloc) {
+        ngx_free(p);
+    }
+
+    return;
+}
+
+#else
+
 static void * ngx_libc_cdecl
 ngx_regex_malloc(size_t size)
 {
@@ -243,11 +495,13 @@ ngx_regex_free(void *p)
     return;
 }
 
+#endif
+
 
 static void
 ngx_regex_cleanup(void *data)
 {
-#if (NGX_HAVE_PCRE_JIT)
+#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
     ngx_regex_conf_t *rcf = data;
 
     ngx_uint_t        i;
@@ -272,12 +526,17 @@ ngx_regex_cleanup(void *data)
         /*
          * The PCRE JIT compiler uses mmap for its executable codes, so we
          * have to explicitly call the pcre_free_study() function to free
-         * this memory.
+         * this memory.  In PCRE2, we call the pcre2_code_free() function
+         * for the same reason.
          */
 
+#if (NGX_PCRE2)
+        pcre2_code_free(elts[i].regex);
+#else
         if (elts[i].regex->extra != NULL) {
             pcre_free_study(elts[i].regex->extra);
         }
+#endif
     }
 #endif
 
@@ -287,6 +546,26 @@ ngx_regex_cleanup(void *data)
      */
 
     ngx_regex_studies = NULL;
+
+#if (NGX_PCRE2)
+
+    /*
+     * Free compile context and match data.  If needed at runtime by
+     * the new cycle, these will be re-allocated.
+     */
+
+    if (ngx_regex_compile_context) {
+        pcre2_compile_context_free(ngx_regex_compile_context);
+        ngx_regex_compile_context = NULL;
+    }
+
+    if (ngx_regex_match_data) {
+        pcre2_match_data_free(ngx_regex_match_data);
+        ngx_regex_match_data = NULL;
+        ngx_regex_match_data_size = 0;
+    }
+
+#endif
 }
 
 
@@ -294,7 +573,9 @@ static ngx_int_t
 ngx_regex_module_init(ngx_cycle_t *cycle)
 {
     int                opt;
+#if !(NGX_PCRE2)
     const char        *errstr;
+#endif
     ngx_uint_t         i;
     ngx_list_part_t   *part;
     ngx_regex_elt_t   *elts;
@@ -304,10 +585,16 @@ ngx_regex_module_init(ngx_cycle_t *cycle
 
     rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
 
-#if (NGX_HAVE_PCRE_JIT)
+#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
+
     if (rcf->pcre_jit) {
+#if (NGX_PCRE2)
+        opt = 1;
+#else
         opt = PCRE_STUDY_JIT_COMPILE;
+#endif
     }
+
 #endif
 
     ngx_regex_malloc_init(cycle->pool);
@@ -327,6 +614,23 @@ ngx_regex_module_init(ngx_cycle_t *cycle
             i = 0;
         }
 
+#if (NGX_PCRE2)
+
+        if (opt) {
+            int  n;
+
+            n = pcre2_jit_compile(elts[i].regex, PCRE2_JIT_COMPLETE);
+
+            if (n != 0) {
+                ngx_log_error(NGX_LOG_INFO, cycle->log, 0,
+                              "pcre2_jit_compile() failed: %d in \"%s\", "
+                              "ignored",
+                              n, elts[i].name);
+            }
+        }
+
+#else
+
         elts[i].regex->extra = pcre_study(elts[i].regex->code, opt, &errstr);
 
         if (errstr != NULL) {
@@ -350,11 +654,15 @@ ngx_regex_module_init(ngx_cycle_t *cycle
             }
         }
 #endif
+#endif
     }
 
     ngx_regex_malloc_done();
 
     ngx_regex_studies = NULL;
+#if (NGX_PCRE2)
+    ngx_regex_compile_context = NULL;
+#endif
 
     return NGX_OK;
 }
@@ -412,7 +720,21 @@ ngx_regex_pcre_jit(ngx_conf_t *cf, void 
         return NGX_CONF_OK;
     }
 
-#if (NGX_HAVE_PCRE_JIT)
+#if (NGX_PCRE2)
+    {
+    int       r;
+    uint32_t  jit;
+
+    jit = 0;
+    r = pcre2_config(PCRE2_CONFIG_JIT, &jit);
+
+    if (r != 0 || jit != 1) {
+        ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
+                           "PCRE2 library does not support JIT");
+        *fp = 0;
+    }
+    }
+#elif (NGX_HAVE_PCRE_JIT)
     {
     int  jit, r;
 
diff --git a/src/core/ngx_regex.h b/src/core/ngx_regex.h
--- a/src/core/ngx_regex.h
+++ b/src/core/ngx_regex.h
@@ -12,19 +12,31 @@
 #include 
 #include 
 
+
+#if (NGX_PCRE2)
+
+#define PCRE2_CODE_UNIT_WIDTH  8
+#include 
+
+#define NGX_REGEX_NO_MATCHED   PCRE2_ERROR_NOMATCH   /* -1 */
+#define NGX_REGEX_CASELESS     PCRE2_CASELESS
+
+typedef pcre2_code  ngx_regex_t;
+
+#else
+
 #include 
 
-
-#define NGX_REGEX_NO_MATCHED  PCRE_ERROR_NOMATCH   /* -1 */
-
-#define NGX_REGEX_CASELESS    PCRE_CASELESS
-
+#define NGX_REGEX_NO_MATCHED   PCRE_ERROR_NOMATCH   /* -1 */
+#define NGX_REGEX_CASELESS     PCRE_CASELESS
 
 typedef struct {
     pcre        *code;
     pcre_extra  *extra;
 } ngx_regex_t;
 
+#endif
+
 
 typedef struct {
     ngx_str_t     pattern;
@@ -49,10 +61,20 @@ typedef struct {
 void ngx_regex_init(void);
 ngx_int_t ngx_regex_compile(ngx_regex_compile_t *rc);
 
+#if (NGX_PCRE2)
+
+ngx_int_t ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures,
+    ngx_uint_t size);
+#define ngx_regex_exec_n       "pcre2_match()"
+
+#else
+
 #define ngx_regex_exec(re, s, captures, size)                                \
     pcre_exec(re->code, re->extra, (const char *) (s)->data, (s)->len, 0, 0, \
               captures, size)
-#define ngx_regex_exec_n      "pcre_exec()"
+#define ngx_regex_exec_n       "pcre_exec()"
+
+#endif
 
 ngx_int_t ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log);
 

-- 
Maxim Dounin
http://mdounin.ru/

From cubicdaiya at gmail.com  Wed Dec 15 00:42:39 2021
From: cubicdaiya at gmail.com (Tatsuhiko Kubo)
Date: Wed, 15 Dec 2021 09:42:39 +0900
Subject: [PATCH 0 of 4] PCRE2 support
In-Reply-To: 
References: 
 
 
Message-ID: 

Hello,

Thank you for your reply. The build works fine after applying the
additional patch.

Thanks.

--
Tatsuhiko Kubo

2021?12?15?(?) 1:09 Maxim Dounin :

>
> Hello!
>
> On Tue, Dec 14, 2021 at 03:09:43PM +0900, Tatsuhiko Kubo wrote:
>
> > Hello,
> >
> > I'm trying these patches and the build of nginx with the option
> > "--with-pcre-jit" fails.
> >
> > $ cd nginx-1.21.4 # with applied patches
> > $ ./configure --with-pcre=../pcre2/pcre2-10.39 --with-pcre-jit
> > $ make
> > ...
> > cc -c -pipe  -O -Wall -Wextra -Wpointer-arith
> > -Wconditional-uninitialized -Wno-unused-parameter
> > -Wno-deprecated-declarations -Werror -g  -I src/core -I src/event -I
> > src/event/modules -I src/os/unix -I ../pcre2/pcre2-10.39/src/ -I objs
> > \
> >                 -o objs/src/core/ngx_regex.o \
> >                 src/core/ngx_regex.c
> > src/core/ngx_regex.c:590:15: error: use of undeclared identifier
> > 'PCRE_STUDY_JIT_COMPILE'
> >         opt = PCRE_STUDY_JIT_COMPILE;
>
> Thanks, looks like a mismerge in the last patch at some point during
> development.  The following patch should fix this:
>
> diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
> --- a/src/core/ngx_regex.c
> +++ b/src/core/ngx_regex.c
> @@ -585,10 +585,16 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>
>      rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
>
> -#if (NGX_HAVE_PCRE_JIT)
> +#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
> +
>      if (rcf->pcre_jit) {
> +#if (NGX_PCRE2)
> +        opt = 1;
> +#else
>          opt = PCRE_STUDY_JIT_COMPILE;
> +#endif
>      }
> +
>  #endif
>
>      ngx_regex_malloc_init(cycle->pool);
>
>
> Full series updated:
>
> # HG changeset patch
> # User Maxim Dounin 
> # Date 1639495851 -10800
> #      Tue Dec 14 18:30:51 2021 +0300
> # Node ID 4f979a9f2a68b25b8b2ce8a0bd15671095f6c327
> # Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
> Core: fixed ngx_pcre_studies cleanup.
>
> If a configuration parsing fails for some reason, ngx_regex_module_init()
> is not called, and ngx_pcre_studies remained set despite the fact that
> the pool it was allocated from is already freed.  This might result in
> a segmentation fault during runtime regular expression compilation, such
> as in SSI, for example, in the single process mode, or if a worker process
> dies and respawn from a master process in such an inconsistent state.
>
> Fix is to clear ngx_pcre_studies from the pool cleanup handler (which is
> anyway used to free JIT-compiled patterns).
>
> diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
> --- a/src/core/ngx_regex.c
> +++ b/src/core/ngx_regex.c
> @@ -10,15 +10,14 @@
>
>
>  typedef struct {
> -    ngx_flag_t  pcre_jit;
> +    ngx_flag_t   pcre_jit;
> +    ngx_list_t  *studies;
>  } ngx_regex_conf_t;
>
>
>  static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
>  static void ngx_libc_cdecl ngx_regex_free(void *p);
> -#if (NGX_HAVE_PCRE_JIT)
> -static void ngx_pcre_free_studies(void *data);
> -#endif
> +static void ngx_regex_cleanup(void *data);
>
>  static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
>
> @@ -248,18 +247,17 @@ ngx_regex_free(void *p)
>  }
>
>
> +static void
> +ngx_regex_cleanup(void *data)
> +{
>  #if (NGX_HAVE_PCRE_JIT)
> -
> -static void
> -ngx_pcre_free_studies(void *data)
> -{
> -    ngx_list_t *studies = data;
> +    ngx_regex_conf_t *rcf = data;
>
>      ngx_uint_t        i;
>      ngx_list_part_t  *part;
>      ngx_regex_elt_t  *elts;
>
> -    part = &studies->part;
> +    part = &rcf->studies->part;
>      elts = part->elts;
>
>      for (i = 0; /* void */ ; i++) {
> @@ -274,56 +272,50 @@ ngx_pcre_free_studies(void *data)
>              i = 0;
>          }
>
> -        if (elts[i].regex->extra != NULL) {
> -            pcre_free_study(elts[i].regex->extra);
> -        }
> -    }
> -}
> -
> -#endif
> -
> -
> -static ngx_int_t
> -ngx_regex_module_init(ngx_cycle_t *cycle)
> -{
> -    int               opt;
> -    const char       *errstr;
> -    ngx_uint_t        i;
> -    ngx_list_part_t  *part;
> -    ngx_regex_elt_t  *elts;
> -
> -    opt = 0;
> -
> -#if (NGX_HAVE_PCRE_JIT)
> -    {
> -    ngx_regex_conf_t    *rcf;
> -    ngx_pool_cleanup_t  *cln;
> -
> -    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
> -
> -    if (rcf->pcre_jit) {
> -        opt = PCRE_STUDY_JIT_COMPILE;
> -
>          /*
>           * The PCRE JIT compiler uses mmap for its executable codes, so we
>           * have to explicitly call the pcre_free_study() function to free
>           * this memory.
>           */
>
> -        cln = ngx_pool_cleanup_add(cycle->pool, 0);
> -        if (cln == NULL) {
> -            return NGX_ERROR;
> +        if (elts[i].regex->extra != NULL) {
> +            pcre_free_study(elts[i].regex->extra);
>          }
> +    }
> +#endif
>
> -        cln->handler = ngx_pcre_free_studies;
> -        cln->data = ngx_pcre_studies;
> -    }
> +    /*
> +     * On configuration parsing errors ngx_regex_module_init() will not
> +     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
> +     */
> +
> +    ngx_pcre_studies = NULL;
> +}
> +
> +
> +static ngx_int_t
> +ngx_regex_module_init(ngx_cycle_t *cycle)
> +{
> +    int                opt;
> +    const char        *errstr;
> +    ngx_uint_t         i;
> +    ngx_list_part_t   *part;
> +    ngx_regex_elt_t   *elts;
> +    ngx_regex_conf_t  *rcf;
> +
> +    opt = 0;
> +
> +    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
> +
> +#if (NGX_HAVE_PCRE_JIT)
> +    if (rcf->pcre_jit) {
> +        opt = PCRE_STUDY_JIT_COMPILE;
>      }
>  #endif
>
>      ngx_regex_malloc_init(cycle->pool);
>
> -    part = &ngx_pcre_studies->part;
> +    part = &rcf->studies->part;
>      elts = part->elts;
>
>      for (i = 0; /* void */ ; i++) {
> @@ -374,7 +366,8 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>  static void *
>  ngx_regex_create_conf(ngx_cycle_t *cycle)
>  {
> -    ngx_regex_conf_t  *rcf;
> +    ngx_regex_conf_t    *rcf;
> +    ngx_pool_cleanup_t  *cln;
>
>      rcf = ngx_pcalloc(cycle->pool, sizeof(ngx_regex_conf_t));
>      if (rcf == NULL) {
> @@ -383,11 +376,21 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
>
>      rcf->pcre_jit = NGX_CONF_UNSET;
>
> -    ngx_pcre_studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
> -    if (ngx_pcre_studies == NULL) {
> +    cln = ngx_pool_cleanup_add(cycle->pool, 0);
> +    if (cln == NULL) {
>          return NULL;
>      }
>
> +    cln->handler = ngx_regex_cleanup;
> +    cln->data = rcf;
> +
> +    rcf->studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
> +    if (rcf->studies == NULL) {
> +        return NULL;
> +    }
> +
> +    ngx_pcre_studies = rcf->studies;
> +
>      return rcf;
>  }
>
> # HG changeset patch
> # User Maxim Dounin 
> # Date 1639497274 -10800
> #      Tue Dec 14 18:54:34 2021 +0300
> # Node ID 8f00e80ade356c768a678c44711e8c2a7223ceef
> # Parent  4f979a9f2a68b25b8b2ce8a0bd15671095f6c327
> Core: ngx_regex.c style cleanup.
>
> Notably, ngx_pcre_pool and ngx_pcre_studies are renamed to ngx_regex_pool
> and ngx_regex_studies, respectively.
>
> diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
> --- a/src/core/ngx_regex.c
> +++ b/src/core/ngx_regex.c
> @@ -64,8 +64,8 @@ ngx_module_t  ngx_regex_module = {
>  };
>
>
> -static ngx_pool_t  *ngx_pcre_pool;
> -static ngx_list_t  *ngx_pcre_studies;
> +static ngx_pool_t  *ngx_regex_pool;
> +static ngx_list_t  *ngx_regex_studies;
>
>
>  void
> @@ -79,14 +79,14 @@ ngx_regex_init(void)
>  static ngx_inline void
>  ngx_regex_malloc_init(ngx_pool_t *pool)
>  {
> -    ngx_pcre_pool = pool;
> +    ngx_regex_pool = pool;
>  }
>
>
>  static ngx_inline void
>  ngx_regex_malloc_done(void)
>  {
> -    ngx_pcre_pool = NULL;
> +    ngx_regex_pool = NULL;
>  }
>
>
> @@ -112,13 +112,13 @@ ngx_regex_compile(ngx_regex_compile_t *r
>             rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
>                                "pcre_compile() failed: %s in \"%V\"",
>                                 errstr, &rc->pattern)
> -                      - rc->err.data;
> +                         - rc->err.data;
>
>          } else {
>             rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
>                                "pcre_compile() failed: %s in \"%V\" at \"%s\"",
>                                 errstr, &rc->pattern, rc->pattern.data + erroff)
> -                      - rc->err.data;
> +                         - rc->err.data;
>          }
>
>          return NGX_ERROR;
> @@ -133,8 +133,8 @@ ngx_regex_compile(ngx_regex_compile_t *r
>
>      /* do not study at runtime */
>
> -    if (ngx_pcre_studies != NULL) {
> -        elt = ngx_list_push(ngx_pcre_studies);
> +    if (ngx_regex_studies != NULL) {
> +        elt = ngx_list_push(ngx_regex_studies);
>          if (elt == NULL) {
>              goto nomem;
>          }
> @@ -229,11 +229,8 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
>  static void * ngx_libc_cdecl
>  ngx_regex_malloc(size_t size)
>  {
> -    ngx_pool_t      *pool;
> -    pool = ngx_pcre_pool;
> -
> -    if (pool) {
> -        return ngx_palloc(pool, size);
> +    if (ngx_regex_pool) {
> +        return ngx_palloc(ngx_regex_pool, size);
>      }
>
>      return NULL;
> @@ -286,10 +283,10 @@ ngx_regex_cleanup(void *data)
>
>      /*
>       * On configuration parsing errors ngx_regex_module_init() will not
> -     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
> +     * be called.  Make sure ngx_regex_studies is properly cleared anyway.
>       */
>
> -    ngx_pcre_studies = NULL;
> +    ngx_regex_studies = NULL;
>  }
>
>
> @@ -357,7 +354,7 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>
>      ngx_regex_malloc_done();
>
> -    ngx_pcre_studies = NULL;
> +    ngx_regex_studies = NULL;
>
>      return NGX_OK;
>  }
> @@ -389,7 +386,7 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
>          return NULL;
>      }
>
> -    ngx_pcre_studies = rcf->studies;
> +    ngx_regex_studies = rcf->studies;
>
>      return rcf;
>  }
> # HG changeset patch
> # User Maxim Dounin 
> # Date 1639497275 -10800
> #      Tue Dec 14 18:54:35 2021 +0300
> # Node ID 0a11d3d92a5d4c1ddcfdd1f07423d59b7c2aa863
> # Parent  8f00e80ade356c768a678c44711e8c2a7223ceef
> Configure: simplified PCRE compilation.
>
> Removed ICC-specific PCRE optimizations which tried to link with PCRE
> object files instead of the library.  Made compiler-specific code
> minimal.
>
> diff --git a/auto/lib/pcre/conf b/auto/lib/pcre/conf
> --- a/auto/lib/pcre/conf
> +++ b/auto/lib/pcre/conf
> @@ -4,81 +4,24 @@
>
>
>  if [ $PCRE != NONE ]; then
> +
> +    have=NGX_PCRE . auto/have
> +
> +    if [ "$NGX_PLATFORM" = win32 ]; then
> +        have=PCRE_STATIC . auto/have
> +    fi
> +
>      CORE_INCS="$CORE_INCS $PCRE"
> +    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
>
>      case "$NGX_CC_NAME" in
>
>          msvc | owc | bcc)
> -            have=NGX_PCRE . auto/have
> -            have=PCRE_STATIC . auto/have
> -            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
>              LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
>              CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
>          ;;
>
> -        icc)
> -            have=NGX_PCRE . auto/have
> -            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
> -
> -            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
> -
> -            echo $ngx_n "checking for PCRE library ...$ngx_c"
> -
> -            if [ -f $PCRE/pcre.h ]; then
> -                ngx_pcre_ver=`grep PCRE_MAJOR $PCRE/pcre.h \
> -                              | sed -e 's/^.*PCRE_MAJOR.* \(.*\)$/\1/'`
> -
> -            else if [ -f $PCRE/configure.in ]; then
> -                ngx_pcre_ver=`grep PCRE_MAJOR= $PCRE/configure.in \
> -                              | sed -e 's/^.*=\(.*\)$/\1/'`
> -
> -            else
> -                ngx_pcre_ver=`grep pcre_major, $PCRE/configure.ac \
> -                              | sed -e 's/^.*pcre_major,.*\[\(.*\)\].*$/\1/'`
> -            fi
> -            fi
> -
> -            echo " $ngx_pcre_ver major version found"
> -
> -            # to allow -ipo optimization we link with the *.o but not library
> -
> -            case "$ngx_pcre_ver" in
> -                4|5)
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre.o"
> -                ;;
> -
> -                6)
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
> -                ;;
> -
> -                *)
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_newline.o"
> -                ;;
> -
> -            esac
> -        ;;
> -
>          *)
> -            have=NGX_PCRE . auto/have
> -
> -            if [ "$NGX_PLATFORM" = win32 ]; then
> -                have=PCRE_STATIC . auto/have
> -            fi
> -
> -            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
>              LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
>              CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
>          ;;
> # HG changeset patch
> # User Maxim Dounin 
> # Date 1639497637 -10800
> #      Tue Dec 14 19:00:37 2021 +0300
> # Node ID 6915b92704ceca0030ea6d9e19f37e868a8d2303
> # Parent  0a11d3d92a5d4c1ddcfdd1f07423d59b7c2aa863
> PCRE2 library support.
>
> The PCRE2 library is now used by default if found, instead of the
> original PCRE library.  If needed for some reason, this can be disabled
> with the --without-pcre2 configure option.
>
> To make it possible to specify paths to the library and include files
> via --with-cc-opt / --with-ld-opt, the library is first tested without
> any additional paths and options.  If this fails, the pcre2-config script
> is used.
>
> Similarly to the original PCRE library, it is now possible to build PCRE2
> from sources with nginx configure, by using the --with-pcre= option.
> It automatically detects if PCRE or PCRE2 sources are provided.
>
> Note that compiling PCRE2 10.33 and later requires inttypes.h.  When
> compiling on Windows with MSVC, inttypes.h is only available starting
> with MSVC 2013.  In older versions some replacement needs to be provided
> ("echo '#include ' > pcre2-10.xx/src/inttypes.h" is good enough
> for MSVC 2010).
>
> The interface on nginx side remains unchanged.
>
> diff --git a/auto/lib/pcre/conf b/auto/lib/pcre/conf
> --- a/auto/lib/pcre/conf
> +++ b/auto/lib/pcre/conf
> @@ -5,29 +5,61 @@
>
>  if [ $PCRE != NONE ]; then
>
> -    have=NGX_PCRE . auto/have
> +    if [ -f $PCRE/src/pcre2.h.generic ]; then
> +
> +        PCRE_LIBRARY=PCRE2
> +
> +        have=NGX_PCRE . auto/have
> +        have=NGX_PCRE2 . auto/have
> +
> +        if [ "$NGX_PLATFORM" = win32 ]; then
> +            have=PCRE2_STATIC . auto/have
> +        fi
> +
> +        CORE_INCS="$CORE_INCS $PCRE/src/"
> +        CORE_DEPS="$CORE_DEPS $PCRE/src/pcre2.h"
>
> -    if [ "$NGX_PLATFORM" = win32 ]; then
> -        have=PCRE_STATIC . auto/have
> -    fi
> +        case "$NGX_CC_NAME" in
> +
> +            msvc)
> +                LINK_DEPS="$LINK_DEPS $PCRE/src/pcre2-8.lib"
> +                CORE_LIBS="$CORE_LIBS $PCRE/src/pcre2-8.lib"
> +            ;;
>
> -    CORE_INCS="$CORE_INCS $PCRE"
> -    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
> +            *)
> +                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre2-8.a"
> +                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre2-8.a"
> +            ;;
>
> -    case "$NGX_CC_NAME" in
> +        esac
>
> -        msvc | owc | bcc)
> -            LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
> -            CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
> -        ;;
> +    else
> +
> +        PCRE_LIBRARY=PCRE
> +
> +        have=NGX_PCRE . auto/have
> +
> +        if [ "$NGX_PLATFORM" = win32 ]; then
> +            have=PCRE_STATIC . auto/have
> +        fi
> +
> +        CORE_INCS="$CORE_INCS $PCRE"
> +        CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
>
> -        *)
> -            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
> -            CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
> -        ;;
> +        case "$NGX_CC_NAME" in
> +
> +            msvc | owc | bcc)
> +                LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
> +                CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
> +            ;;
>
> -    esac
> +            *)
> +                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
> +                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
> +            ;;
>
> +        esac
> +    fi
>
>      if [ $PCRE_JIT = YES ]; then
>          have=NGX_HAVE_PCRE_JIT . auto/have
> @@ -37,8 +69,48 @@ if [ $PCRE != NONE ]; then
>  else
>
>      if [ "$NGX_PLATFORM" != win32 ]; then
> +        PCRE=NO
> +    fi
>
> -        PCRE=NO
> +    if [ $PCRE = NO -a $PCRE2 != DISABLED ]; then
> +
> +        ngx_feature="PCRE2 library"
> +        ngx_feature_name="NGX_PCRE2"
> +        ngx_feature_run=no
> +        ngx_feature_incs="#define PCRE2_CODE_UNIT_WIDTH 8
> +                          #include "
> +        ngx_feature_path=
> +        ngx_feature_libs="-lpcre2-8"
> +        ngx_feature_test="pcre2_code *re;
> +                          re = pcre2_compile(NULL, 0, 0, NULL, NULL, NULL);
> +                          if (re == NULL) return 1"
> +        . auto/feature
> +
> +        if [ $ngx_found = no ]; then
> +
> +            # pcre2-config
> +
> +            ngx_pcre2_prefix=`pcre2-config --prefix 2>/dev/null`
> +
> +            if [ -n "$ngx_pcre2_prefix" ]; then
> +                ngx_feature="PCRE2 library in $ngx_pcre2_prefix"
> +                ngx_feature_path=`pcre2-config --cflags \
> +                                  | sed -n -e 's/.*-I *\([^ ][^ ]*\).*/\1/p'`
> +                ngx_feature_libs=`pcre2-config --libs8`
> +                . auto/feature
> +            fi
> +        fi
> +
> +        if [ $ngx_found = yes ]; then
> +            have=NGX_PCRE . auto/have
> +            CORE_INCS="$CORE_INCS $ngx_feature_path"
> +            CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
> +            PCRE=YES
> +            PCRE_LIBRARY=PCRE2
> +        fi
> +    fi
> +
> +    if [ $PCRE = NO ]; then
>
>          ngx_feature="PCRE library"
>          ngx_feature_name="NGX_PCRE"
> @@ -114,6 +186,7 @@ else
>              CORE_INCS="$CORE_INCS $ngx_feature_path"
>              CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
>              PCRE=YES
> +            PCRE_LIBRARY=PCRE
>          fi
>
>          if [ $PCRE = YES ]; then
> diff --git a/auto/lib/pcre/make b/auto/lib/pcre/make
> --- a/auto/lib/pcre/make
> +++ b/auto/lib/pcre/make
> @@ -3,36 +3,138 @@
>  # Copyright (C) Nginx, Inc.
>
>
> -case "$NGX_CC_NAME" in
> +if [ $PCRE_LIBRARY = PCRE2 ]; then
> +
> +    # PCRE2
> +
> +    if [ $NGX_CC_NAME = msvc ]; then
> +
> +        # With PCRE2, it is not possible to compile all sources.
> +        # Since list of source files changes between versions, we
> +        # test files which might not be present.
>
> -    msvc)
> -        ngx_makefile=makefile.msvc
> -        ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
> -        ngx_pcre="PCRE=\"$PCRE\""
> -    ;;
> +        ngx_pcre_srcs="pcre2_auto_possess.c \
> +                       pcre2_chartables.c \
> +                       pcre2_compile.c \
> +                       pcre2_config.c \
> +                       pcre2_context.c \
> +                       pcre2_dfa_match.c \
> +                       pcre2_error.c \
> +                       pcre2_jit_compile.c \
> +                       pcre2_maketables.c \
> +                       pcre2_match.c \
> +                       pcre2_match_data.c \
> +                       pcre2_newline.c \
> +                       pcre2_ord2utf.c \
> +                       pcre2_pattern_info.c \
> +                       pcre2_string_utils.c \
> +                       pcre2_study.c \
> +                       pcre2_substitute.c \
> +                       pcre2_substring.c \
> +                       pcre2_tables.c \
> +                       pcre2_ucd.c \
> +                       pcre2_valid_utf.c \
> +                       pcre2_xclass.c"
> +
> +        ngx_pcre_test="pcre2_convert.c \
> +                       pcre2_extuni.c \
> +                       pcre2_find_bracket.c \
> +                       pcre2_script_run.c \
> +                       pcre2_serialize.c"
> +
> +        for ngx_src in $ngx_pcre_test
> +        do
> +            if [ -f $PCRE/src/$ngx_src ]; then
> +                ngx_pcre_srcs="$ngx_pcre_srcs $ngx_src"
> +            fi
> +        done
>
> -    owc)
> -        ngx_makefile=makefile.owc
> -        ngx_opt="CPU_OPT=\"$CPU_OPT\""
> -        ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
> -    ;;
> +        ngx_pcre_objs=`echo $ngx_pcre_srcs \
> +            | sed -e "s#\([^ ]*\.\)c#\1$ngx_objext#g"`
> +
> +        ngx_pcre_srcs=`echo $ngx_pcre_srcs \
> +            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
> +        ngx_pcre_objs=`echo $ngx_pcre_objs \
> +            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
> +
> +        cat << END                                            >> $NGX_MAKEFILE
> +
> +PCRE_CFLAGS =  -O2 -Ob1 -Oi -Gs $LIBC $CPU_OPT
> +PCRE_FLAGS =   -DHAVE_CONFIG_H -DPCRE2_STATIC -DPCRE2_CODE_UNIT_WIDTH=8 \\
> +               -DHAVE_MEMMOVE
> +
> +PCRE_SRCS =     $ngx_pcre_srcs
> +PCRE_OBJS =     $ngx_pcre_objs
> +
> +$PCRE/src/pcre2.h:
> +       cd $PCRE/src \\
> +       && copy /y config.h.generic config.h \\
> +       && copy /y pcre2.h.generic pcre2.h \\
> +       && copy /y pcre2_chartables.c.dist pcre2_chartables.c
>
> -    bcc)
> -        ngx_makefile=makefile.bcc
> -        ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
> -        ngx_pcre=`echo \-DPCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
> -    ;;
> +$PCRE/src/pcre2-8.lib: $PCRE/src/pcre2.h $NGX_MAKEFILE
> +       cd $PCRE/src \\
> +       && cl -nologo -c \$(PCRE_CFLAGS) -I . \$(PCRE_FLAGS) \$(PCRE_SRCS) \\
> +       && link -lib -out:pcre2-8.lib -verbose:lib \$(PCRE_OBJS)
> +
> +END
> +
> +    else
> +
> +        cat << END                                            >> $NGX_MAKEFILE
> +
> +$PCRE/src/pcre2.h:     $PCRE/Makefile
>
> -    *)
> -        ngx_makefile=
> -    ;;
> +$PCRE/Makefile:        $NGX_MAKEFILE
> +       cd $PCRE \\
> +       && if [ -f Makefile ]; then \$(MAKE) distclean; fi \\
> +       && CC="\$(CC)" CFLAGS="$PCRE_OPT" \\
> +       ./configure --disable-shared $PCRE_CONF_OPT
>
> -esac
> +$PCRE/.libs/libpcre2-8.a:      $PCRE/Makefile
> +       cd $PCRE \\
> +       && \$(MAKE) libpcre2-8.la
> +
> +END
> +
> +    fi
>
>
> -if [ -n "$ngx_makefile" ]; then
> +else
> +
> +    # PCRE
> +
> +    case "$NGX_CC_NAME" in
> +
> +        msvc)
> +            ngx_makefile=makefile.msvc
> +            ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
> +            ngx_pcre="PCRE=\"$PCRE\""
> +        ;;
> +
> +        owc)
> +            ngx_makefile=makefile.owc
> +            ngx_opt="CPU_OPT=\"$CPU_OPT\""
> +            ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
> +        ;;
>
> -    cat << END                                                >> $NGX_MAKEFILE
> +        bcc)
> +            ngx_makefile=makefile.bcc
> +            ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
> +            ngx_pcre=`echo \-DPCRE=\"$PCRE\" \
> +                | sed -e "s/\//$ngx_regex_dirsep/g"`
> +        ;;
> +
> +        *)
> +            ngx_makefile=
> +        ;;
> +
> +    esac
> +
> +
> +    if [ -n "$ngx_makefile" ]; then
> +
> +        cat << END                                            >> $NGX_MAKEFILE
>
>  `echo "$PCRE/pcre.lib: $PCRE/pcre.h $NGX_MAKEFILE"                     \
>         | sed -e "s/\//$ngx_regex_dirsep/g"`
> @@ -43,9 +145,9 @@ if [ -n "$ngx_makefile" ]; then
>
>  END
>
> -else
> +    else
>
> -    cat << END                                                >> $NGX_MAKEFILE
> +        cat << END                                            >> $NGX_MAKEFILE
>
>  $PCRE/pcre.h:  $PCRE/Makefile
>
> @@ -61,4 +163,6 @@ else
>
>  END
>
> +    fi
> +
>  fi
> diff --git a/auto/options b/auto/options
> --- a/auto/options
> +++ b/auto/options
> @@ -146,6 +146,7 @@ PCRE=NONE
>  PCRE_OPT=
>  PCRE_CONF_OPT=
>  PCRE_JIT=NO
> +PCRE2=YES
>
>  USE_OPENSSL=NO
>  OPENSSL=NONE
> @@ -357,6 +358,7 @@ use the \"--with-mail_ssl_module\" optio
>          --with-pcre=*)                   PCRE="$value"              ;;
>          --with-pcre-opt=*)               PCRE_OPT="$value"          ;;
>          --with-pcre-jit)                 PCRE_JIT=YES               ;;
> +        --without-pcre2)                 PCRE2=DISABLED             ;;
>
>          --with-openssl=*)                OPENSSL="$value"           ;;
>          --with-openssl-opt=*)            OPENSSL_OPT="$value"       ;;
> @@ -573,6 +575,7 @@ cat << END
>    --with-pcre=DIR                    set path to PCRE library sources
>    --with-pcre-opt=OPTIONS            set additional build options for PCRE
>    --with-pcre-jit                    build PCRE with JIT compilation support
> +  --without-pcre2                    do not use PCRE2 library
>
>    --with-zlib=DIR                    set path to zlib library sources
>    --with-zlib-opt=OPTIONS            set additional build options for zlib
> diff --git a/auto/summary b/auto/summary
> --- a/auto/summary
> +++ b/auto/summary
> @@ -16,9 +16,9 @@ if [ $USE_PCRE = DISABLED ]; then
>
>  else
>      case $PCRE in
> -        YES)   echo "  + using system PCRE library" ;;
> +        YES)   echo "  + using system $PCRE_LIBRARY library" ;;
>          NONE)  echo "  + PCRE library is not used" ;;
> -        *)     echo "  + using PCRE library: $PCRE" ;;
> +        *)     echo "  + using $PCRE_LIBRARY library: $PCRE" ;;
>      esac
>  fi
>
> diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
> --- a/src/core/ngx_regex.c
> +++ b/src/core/ngx_regex.c
> @@ -15,8 +15,16 @@ typedef struct {
>  } ngx_regex_conf_t;
>
>
> +static ngx_inline void ngx_regex_malloc_init(ngx_pool_t *pool);
> +static ngx_inline void ngx_regex_malloc_done(void);
> +
> +#if (NGX_PCRE2)
> +static void * ngx_libc_cdecl ngx_regex_malloc(size_t size, void *data);
> +static void ngx_libc_cdecl ngx_regex_free(void *p, void *data);
> +#else
>  static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
>  static void ngx_libc_cdecl ngx_regex_free(void *p);
> +#endif
>  static void ngx_regex_cleanup(void *data);
>
>  static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
> @@ -64,15 +72,24 @@ ngx_module_t  ngx_regex_module = {
>  };
>
>
> -static ngx_pool_t  *ngx_regex_pool;
> -static ngx_list_t  *ngx_regex_studies;
> +static ngx_pool_t             *ngx_regex_pool;
> +static ngx_list_t             *ngx_regex_studies;
> +static ngx_uint_t              ngx_regex_direct_alloc;
> +
> +#if (NGX_PCRE2)
> +static pcre2_compile_context  *ngx_regex_compile_context;
> +static pcre2_match_data       *ngx_regex_match_data;
> +static ngx_uint_t              ngx_regex_match_data_size;
> +#endif
>
>
>  void
>  ngx_regex_init(void)
>  {
> +#if !(NGX_PCRE2)
>      pcre_malloc = ngx_regex_malloc;
>      pcre_free = ngx_regex_free;
> +#endif
>  }
>
>
> @@ -80,6 +97,7 @@ static ngx_inline void
>  ngx_regex_malloc_init(ngx_pool_t *pool)
>  {
>      ngx_regex_pool = pool;
> +    ngx_regex_direct_alloc = (pool == NULL) ? 1 : 0;
>  }
>
>
> @@ -87,9 +105,146 @@ static ngx_inline void
>  ngx_regex_malloc_done(void)
>  {
>      ngx_regex_pool = NULL;
> +    ngx_regex_direct_alloc = 0;
>  }
>
>
> +#if (NGX_PCRE2)
> +
> +ngx_int_t
> +ngx_regex_compile(ngx_regex_compile_t *rc)
> +{
> +    int                     n, errcode;
> +    char                   *p;
> +    u_char                  errstr[128];
> +    size_t                  erroff;
> +    pcre2_code             *re;
> +    ngx_regex_elt_t        *elt;
> +    pcre2_general_context  *gctx;
> +    pcre2_compile_context  *cctx;
> +
> +    if (ngx_regex_compile_context == NULL) {
> +        /*
> +         * Allocte a compile context if not yet allocated.  This uses
> +         * direct allocations from heap, so the result can be cached
> +         * even at runtime.
> +         */
> +
> +        ngx_regex_malloc_init(NULL);
> +
> +        gctx = pcre2_general_context_create(ngx_regex_malloc, ngx_regex_free,
> +                                            NULL);
> +        if (gctx == NULL) {
> +            ngx_regex_malloc_done();
> +            goto nomem;
> +        }
> +
> +        cctx = pcre2_compile_context_create(gctx);
> +        if (cctx == NULL) {
> +            pcre2_general_context_free(gctx);
> +            ngx_regex_malloc_done();
> +            goto nomem;
> +        }
> +
> +        ngx_regex_compile_context = cctx;
> +
> +        pcre2_general_context_free(gctx);
> +        ngx_regex_malloc_done();
> +    }
> +
> +    ngx_regex_malloc_init(rc->pool);
> +
> +    re = pcre2_compile(rc->pattern.data, rc->pattern.len,
> +                       (uint32_t) rc->options, &errcode, &erroff,
> +                       ngx_regex_compile_context);
> +
> +    /* ensure that there is no current pool */
> +    ngx_regex_malloc_done();
> +
> +    if (re == NULL) {
> +        pcre2_get_error_message(errcode, errstr, 128);
> +
> +        if ((size_t) erroff == rc->pattern.len) {
> +            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
> +                              "pcre2_compile() failed: %s in \"%V\"",
> +                               errstr, &rc->pattern)
> +                          - rc->err.data;
> +
> +        } else {
> +            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
> +                              "pcre2_compile() failed: %s in \"%V\" at \"%s\"",
> +                               errstr, &rc->pattern, rc->pattern.data + erroff)
> +                          - rc->err.data;
> +        }
> +
> +        return NGX_ERROR;
> +    }
> +
> +    rc->regex = re;
> +
> +    /* do not study at runtime */
> +
> +    if (ngx_regex_studies != NULL) {
> +        elt = ngx_list_push(ngx_regex_studies);
> +        if (elt == NULL) {
> +            goto nomem;
> +        }
> +
> +        elt->regex = rc->regex;
> +        elt->name = rc->pattern.data;
> +    }
> +
> +    n = pcre2_pattern_info(re, PCRE2_INFO_CAPTURECOUNT, &rc->captures);
> +    if (n < 0) {
> +        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_CAPTURECOUNT) failed: %d";
> +        goto failed;
> +    }
> +
> +    if (rc->captures == 0) {
> +        return NGX_OK;
> +    }
> +
> +    n = pcre2_pattern_info(re, PCRE2_INFO_NAMECOUNT, &rc->named_captures);
> +    if (n < 0) {
> +        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMECOUNT) failed: %d";
> +        goto failed;
> +    }
> +
> +    if (rc->named_captures == 0) {
> +        return NGX_OK;
> +    }
> +
> +    n = pcre2_pattern_info(re, PCRE2_INFO_NAMEENTRYSIZE, &rc->name_size);
> +    if (n < 0) {
> +        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMEENTRYSIZE) failed: %d";
> +        goto failed;
> +    }
> +
> +    n = pcre2_pattern_info(re, PCRE2_INFO_NAMETABLE, &rc->names);
> +    if (n < 0) {
> +        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMETABLE) failed: %d";
> +        goto failed;
> +    }
> +
> +    return NGX_OK;
> +
> +failed:
> +
> +    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len, p, &rc->pattern, n)
> +                  - rc->err.data;
> +    return NGX_ERROR;
> +
> +nomem:
> +
> +    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
> +                               "regex \"%V\" compilation failed: no memory",
> +                               &rc->pattern)
> +                  - rc->err.data;
> +    return NGX_ERROR;
> +}
> +
> +#else
> +
>  ngx_int_t
>  ngx_regex_compile(ngx_regex_compile_t *rc)
>  {
> @@ -192,6 +347,74 @@ nomem:
>      return NGX_ERROR;
>  }
>
> +#endif
> +
> +
> +#if (NGX_PCRE2)
> +
> +ngx_int_t
> +ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures, ngx_uint_t size)
> +{
> +    size_t      *ov;
> +    ngx_int_t    rc;
> +    ngx_uint_t   n, i;
> +
> +    /*
> +     * The pcre2_match() function might allocate memory for backtracking
> +     * frames, typical allocations are from 40k and above.  So the allocator
> +     * is configured to do direct allocations from heap during matching.
> +     */
> +
> +    ngx_regex_malloc_init(NULL);
> +
> +    if (ngx_regex_match_data == NULL
> +        || size > ngx_regex_match_data_size)
> +    {
> +        /*
> +         * Allocate a match data if not yet allocated or smaller than
> +         * needed.
> +         */
> +
> +        if (ngx_regex_match_data) {
> +            pcre2_match_data_free(ngx_regex_match_data);
> +        }
> +
> +        ngx_regex_match_data_size = size;
> +        ngx_regex_match_data = pcre2_match_data_create(size / 3, NULL);
> +
> +        if (ngx_regex_match_data == NULL) {
> +            rc = PCRE2_ERROR_NOMEMORY;
> +            goto failed;
> +        }
> +    }
> +
> +    rc = pcre2_match(re, s->data, s->len, 0, 0, ngx_regex_match_data, NULL);
> +
> +    if (rc < 0) {
> +        goto failed;
> +    }
> +
> +    n = pcre2_get_ovector_count(ngx_regex_match_data);
> +    ov = pcre2_get_ovector_pointer(ngx_regex_match_data);
> +
> +    if (n > size / 3) {
> +        n = size / 3;
> +    }
> +
> +    for (i = 0; i < n; i++) {
> +        captures[i * 2] = ov[i * 2];
> +        captures[i * 2 + 1] = ov[i * 2 + 1];
> +    }
> +
> +failed:
> +
> +    ngx_regex_malloc_done();
> +
> +    return rc;
> +}
> +
> +#endif
> +
>
>  ngx_int_t
>  ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log)
> @@ -226,6 +449,35 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
>  }
>
>
> +#if (NGX_PCRE2)
> +
> +static void * ngx_libc_cdecl
> +ngx_regex_malloc(size_t size, void *data)
> +{
> +    if (ngx_regex_pool) {
> +        return ngx_palloc(ngx_regex_pool, size);
> +    }
> +
> +    if (ngx_regex_direct_alloc) {
> +        return ngx_alloc(size, ngx_cycle->log);
> +    }
> +
> +    return NULL;
> +}
> +
> +
> +static void ngx_libc_cdecl
> +ngx_regex_free(void *p, void *data)
> +{
> +    if (ngx_regex_direct_alloc) {
> +        ngx_free(p);
> +    }
> +
> +    return;
> +}
> +
> +#else
> +
>  static void * ngx_libc_cdecl
>  ngx_regex_malloc(size_t size)
>  {
> @@ -243,11 +495,13 @@ ngx_regex_free(void *p)
>      return;
>  }
>
> +#endif
> +
>
>  static void
>  ngx_regex_cleanup(void *data)
>  {
> -#if (NGX_HAVE_PCRE_JIT)
> +#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
>      ngx_regex_conf_t *rcf = data;
>
>      ngx_uint_t        i;
> @@ -272,12 +526,17 @@ ngx_regex_cleanup(void *data)
>          /*
>           * The PCRE JIT compiler uses mmap for its executable codes, so we
>           * have to explicitly call the pcre_free_study() function to free
> -         * this memory.
> +         * this memory.  In PCRE2, we call the pcre2_code_free() function
> +         * for the same reason.
>           */
>
> +#if (NGX_PCRE2)
> +        pcre2_code_free(elts[i].regex);
> +#else
>          if (elts[i].regex->extra != NULL) {
>              pcre_free_study(elts[i].regex->extra);
>          }
> +#endif
>      }
>  #endif
>
> @@ -287,6 +546,26 @@ ngx_regex_cleanup(void *data)
>       */
>
>      ngx_regex_studies = NULL;
> +
> +#if (NGX_PCRE2)
> +
> +    /*
> +     * Free compile context and match data.  If needed at runtime by
> +     * the new cycle, these will be re-allocated.
> +     */
> +
> +    if (ngx_regex_compile_context) {
> +        pcre2_compile_context_free(ngx_regex_compile_context);
> +        ngx_regex_compile_context = NULL;
> +    }
> +
> +    if (ngx_regex_match_data) {
> +        pcre2_match_data_free(ngx_regex_match_data);
> +        ngx_regex_match_data = NULL;
> +        ngx_regex_match_data_size = 0;
> +    }
> +
> +#endif
>  }
>
>
> @@ -294,7 +573,9 @@ static ngx_int_t
>  ngx_regex_module_init(ngx_cycle_t *cycle)
>  {
>      int                opt;
> +#if !(NGX_PCRE2)
>      const char        *errstr;
> +#endif
>      ngx_uint_t         i;
>      ngx_list_part_t   *part;
>      ngx_regex_elt_t   *elts;
> @@ -304,10 +585,16 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>
>      rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
>
> -#if (NGX_HAVE_PCRE_JIT)
> +#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
> +
>      if (rcf->pcre_jit) {
> +#if (NGX_PCRE2)
> +        opt = 1;
> +#else
>          opt = PCRE_STUDY_JIT_COMPILE;
> +#endif
>      }
> +
>  #endif
>
>      ngx_regex_malloc_init(cycle->pool);
> @@ -327,6 +614,23 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>              i = 0;
>          }
>
> +#if (NGX_PCRE2)
> +
> +        if (opt) {
> +            int  n;
> +
> +            n = pcre2_jit_compile(elts[i].regex, PCRE2_JIT_COMPLETE);
> +
> +            if (n != 0) {
> +                ngx_log_error(NGX_LOG_INFO, cycle->log, 0,
> +                              "pcre2_jit_compile() failed: %d in \"%s\", "
> +                              "ignored",
> +                              n, elts[i].name);
> +            }
> +        }
> +
> +#else
> +
>          elts[i].regex->extra = pcre_study(elts[i].regex->code, opt, &errstr);
>
>          if (errstr != NULL) {
> @@ -350,11 +654,15 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>              }
>          }
>  #endif
> +#endif
>      }
>
>      ngx_regex_malloc_done();
>
>      ngx_regex_studies = NULL;
> +#if (NGX_PCRE2)
> +    ngx_regex_compile_context = NULL;
> +#endif
>
>      return NGX_OK;
>  }
> @@ -412,7 +720,21 @@ ngx_regex_pcre_jit(ngx_conf_t *cf, void
>          return NGX_CONF_OK;
>      }
>
> -#if (NGX_HAVE_PCRE_JIT)
> +#if (NGX_PCRE2)
> +    {
> +    int       r;
> +    uint32_t  jit;
> +
> +    jit = 0;
> +    r = pcre2_config(PCRE2_CONFIG_JIT, &jit);
> +
> +    if (r != 0 || jit != 1) {
> +        ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
> +                           "PCRE2 library does not support JIT");
> +        *fp = 0;
> +    }
> +    }
> +#elif (NGX_HAVE_PCRE_JIT)
>      {
>      int  jit, r;
>
> diff --git a/src/core/ngx_regex.h b/src/core/ngx_regex.h
> --- a/src/core/ngx_regex.h
> +++ b/src/core/ngx_regex.h
> @@ -12,19 +12,31 @@
>  #include 
>  #include 
>
> +
> +#if (NGX_PCRE2)
> +
> +#define PCRE2_CODE_UNIT_WIDTH  8
> +#include 
> +
> +#define NGX_REGEX_NO_MATCHED   PCRE2_ERROR_NOMATCH   /* -1 */
> +#define NGX_REGEX_CASELESS     PCRE2_CASELESS
> +
> +typedef pcre2_code  ngx_regex_t;
> +
> +#else
> +
>  #include 
>
> -
> -#define NGX_REGEX_NO_MATCHED  PCRE_ERROR_NOMATCH   /* -1 */
> -
> -#define NGX_REGEX_CASELESS    PCRE_CASELESS
> -
> +#define NGX_REGEX_NO_MATCHED   PCRE_ERROR_NOMATCH   /* -1 */
> +#define NGX_REGEX_CASELESS     PCRE_CASELESS
>
>  typedef struct {
>      pcre        *code;
>      pcre_extra  *extra;
>  } ngx_regex_t;
>
> +#endif
> +
>
>  typedef struct {
>      ngx_str_t     pattern;
> @@ -49,10 +61,20 @@ typedef struct {
>  void ngx_regex_init(void);
>  ngx_int_t ngx_regex_compile(ngx_regex_compile_t *rc);
>
> +#if (NGX_PCRE2)
> +
> +ngx_int_t ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures,
> +    ngx_uint_t size);
> +#define ngx_regex_exec_n       "pcre2_match()"
> +
> +#else
> +
>  #define ngx_regex_exec(re, s, captures, size)                                \
>      pcre_exec(re->code, re->extra, (const char *) (s)->data, (s)->len, 0, 0, \
>                captures, size)
> -#define ngx_regex_exec_n      "pcre_exec()"
> +#define ngx_regex_exec_n       "pcre_exec()"
> +
> +#endif
>
>  ngx_int_t ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log);
>
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

From xeioex at nginx.com  Wed Dec 15 13:14:35 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 15 Dec 2021 13:14:35 +0000
Subject: [njs] Using njs_arr_item() when accessing vm->protos elements.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/02da0ed4a778
branches:  
changeset: 1765:02da0ed4a778
user:      Dmitry Volyntsev 
date:      Tue Dec 14 14:08:12 2021 +0000
description:
Using njs_arr_item() when accessing vm->protos elements.

diffstat:

 src/njs_extern.c |  16 ++++++----------
 1 files changed, 6 insertions(+), 10 deletions(-)

diffs (60 lines):

diff -r dfcbfb5e27b2 -r 02da0ed4a778 src/njs_extern.c
--- a/src/njs_extern.c	Thu Dec 09 14:38:40 2021 +0000
+++ b/src/njs_extern.c	Tue Dec 14 14:08:12 2021 +0000
@@ -248,9 +248,8 @@ njs_int_t
 njs_vm_external_prototype(njs_vm_t *vm, const njs_external_t *definition,
     njs_uint_t n)
 {
-    njs_arr_t   *protos;
+    njs_arr_t   *protos, **pr;
     njs_int_t   ret;
-    uintptr_t   *pr;
     njs_uint_t  size;
 
     size = njs_external_protos(definition, n) + 1;
@@ -268,7 +267,7 @@ njs_vm_external_prototype(njs_vm_t *vm, 
     }
 
     if (vm->protos == NULL) {
-        vm->protos = njs_arr_create(vm->mem_pool, 4, sizeof(uintptr_t));
+        vm->protos = njs_arr_create(vm->mem_pool, 4, sizeof(njs_arr_t *));
         if (njs_slow_path(vm->protos == NULL)) {
             return -1;
         }
@@ -279,7 +278,7 @@ njs_vm_external_prototype(njs_vm_t *vm, 
         return -1;
     }
 
-    *pr = (uintptr_t) protos;
+    *pr = protos;
 
     return vm->protos->items - 1;
 }
@@ -289,8 +288,7 @@ njs_int_t
 njs_vm_external_create(njs_vm_t *vm, njs_value_t *value, njs_int_t proto_id,
     njs_external_ptr_t external, njs_bool_t shared)
 {
-    njs_arr_t           *protos;
-    uintptr_t           proto;
+    njs_arr_t           **pprotos;
     njs_object_value_t  *ov;
     njs_exotic_slots_t  *slots;
 
@@ -298,15 +296,13 @@ njs_vm_external_create(njs_vm_t *vm, njs
         return NJS_ERROR;
     }
 
-    proto = ((uintptr_t *) vm->protos->start)[proto_id];
-
     ov = njs_object_value_alloc(vm, NJS_OBJ_TYPE_OBJECT, 0, NULL);
     if (njs_slow_path(ov == NULL)) {
         return NJS_ERROR;
     }
 
-    protos = (njs_arr_t *) proto;
-    slots = protos->start;
+    pprotos = njs_arr_item(vm->protos, proto_id);
+    slots = (*pprotos)->start;
 
     ov->object.shared_hash = slots->external_shared_hash;
     ov->object.shared = shared;

From xeioex at nginx.com  Wed Dec 15 13:14:37 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 15 Dec 2021 13:14:37 +0000
Subject: [njs] Checking cwd path length in njs_module_relative_path().
Message-ID: 

details:   https://hg.nginx.org/njs/rev/ee500f43e2a2
branches:  
changeset: 1766:ee500f43e2a2
user:      Dmitry Volyntsev 
date:      Tue Dec 14 20:06:23 2021 +0000
description:
Checking cwd path length in njs_module_relative_path().

diffstat:

 src/njs_module.c |  4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diffs (14 lines):

diff -r 02da0ed4a778 -r ee500f43e2a2 src/njs_module.c
--- a/src/njs_module.c	Tue Dec 14 14:08:12 2021 +0000
+++ b/src/njs_module.c	Tue Dec 14 20:06:23 2021 +0000
@@ -361,6 +361,10 @@ njs_module_relative_path(njs_vm_t *vm, c
 
     file.length = dir->length;
 
+    if (file.length == 0) {
+        return NJS_DECLINED;
+    }
+
     trail = (dir->start[dir->length - 1] != '/');
 
     if (trail) {

From xeioex at nginx.com  Wed Dec 15 13:14:39 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 15 Dec 2021 13:14:39 +0000
Subject: [njs] Modules: removed dead code left after dfcbfb5e27b2.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/659995ad4f97
branches:  
changeset: 1767:659995ad4f97
user:      Dmitry Volyntsev 
date:      Tue Dec 14 20:14:23 2021 +0000
description:
Modules: removed dead code left after dfcbfb5e27b2.

Found by Coverity (CID 1495259).

diffstat:

 nginx/ngx_http_js_module.c   |  6 ------
 nginx/ngx_stream_js_module.c |  6 ------
 2 files changed, 0 insertions(+), 12 deletions(-)

diffs (60 lines):

diff -r ee500f43e2a2 -r 659995ad4f97 nginx/ngx_http_js_module.c
--- a/nginx/ngx_http_js_module.c	Tue Dec 14 20:06:23 2021 +0000
+++ b/nginx/ngx_http_js_module.c	Tue Dec 14 20:14:23 2021 +0000
@@ -3469,7 +3469,6 @@ ngx_http_js_init_main_conf(ngx_conf_t *c
 
     size_t                 size;
     u_char                *start, *end, *p;
-    ngx_fd_t               fd;
     ngx_str_t             *m, file;
     njs_int_t              rc;
     njs_str_t              text, path;
@@ -3488,7 +3487,6 @@ ngx_http_js_init_main_conf(ngx_conf_t *c
     }
 
     size = 0;
-    fd = NGX_INVALID_FILE;
 
     import = jmcf->imports->elts;
     for (i = 0; i < jmcf->imports->nelts; i++) {
@@ -3498,10 +3496,6 @@ ngx_http_js_init_main_conf(ngx_conf_t *c
 
     start = ngx_pnalloc(cf->pool, size);
     if (start == NULL) {
-        if (fd != NGX_INVALID_FILE) {
-            (void) ngx_close_file(fd);
-        }
-
         return NGX_CONF_ERROR;
     }
 
diff -r ee500f43e2a2 -r 659995ad4f97 nginx/ngx_stream_js_module.c
--- a/nginx/ngx_stream_js_module.c	Tue Dec 14 20:06:23 2021 +0000
+++ b/nginx/ngx_stream_js_module.c	Tue Dec 14 20:14:23 2021 +0000
@@ -1491,7 +1491,6 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
 
     size_t                   size;
     u_char                  *start, *end, *p;
-    ngx_fd_t                 fd;
     ngx_str_t               *m, file;
     njs_int_t                rc;
     njs_str_t                text, path;
@@ -1510,7 +1509,6 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
     }
 
     size = 0;
-    fd = NGX_INVALID_FILE;
 
     import = jmcf->imports->elts;
     for (i = 0; i < jmcf->imports->nelts; i++) {
@@ -1520,10 +1518,6 @@ ngx_stream_js_init_main_conf(ngx_conf_t 
 
     start = ngx_pnalloc(cf->pool, size);
     if (start == NULL) {
-        if (fd != NGX_INVALID_FILE) {
-            (void) ngx_close_file(fd);
-        }
-
         return NGX_CONF_ERROR;
     }
 

From mdounin at mdounin.ru  Thu Dec 16 19:06:07 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 16 Dec 2021 22:06:07 +0300
Subject: [PATCH 0 of 4] PCRE2 support
In-Reply-To: 
References: 
 
 
Message-ID: 

Hello!

On Tue, Dec 14, 2021 at 07:09:09PM +0300, Maxim Dounin wrote:

> Full series updated:
> 
> # HG changeset patch
> # User Maxim Dounin 
> # Date 1639495851 -10800
> #      Tue Dec 14 18:30:51 2021 +0300
> # Node ID 4f979a9f2a68b25b8b2ce8a0bd15671095f6c327
> # Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
> Core: fixed ngx_pcre_studies cleanup.
> 
> If a configuration parsing fails for some reason, ngx_regex_module_init()
> is not called, and ngx_pcre_studies remained set despite the fact that
> the pool it was allocated from is already freed.  This might result in
> a segmentation fault during runtime regular expression compilation, such
> as in SSI, for example, in the single process mode, or if a worker process
> dies and respawn from a master process in such an inconsistent state.
> 
> Fix is to clear ngx_pcre_studies from the pool cleanup handler (which is
> anyway used to free JIT-compiled patterns).
> 
> diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
> --- a/src/core/ngx_regex.c
> +++ b/src/core/ngx_regex.c
> @@ -10,15 +10,14 @@
>  
>  
>  typedef struct {
> -    ngx_flag_t  pcre_jit;
> +    ngx_flag_t   pcre_jit;
> +    ngx_list_t  *studies;
>  } ngx_regex_conf_t;
>  
>  
>  static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
>  static void ngx_libc_cdecl ngx_regex_free(void *p);
> -#if (NGX_HAVE_PCRE_JIT)
> -static void ngx_pcre_free_studies(void *data);
> -#endif
> +static void ngx_regex_cleanup(void *data);
>  
>  static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
>  
> @@ -248,18 +247,17 @@ ngx_regex_free(void *p)
>  }
>  
>  
> +static void
> +ngx_regex_cleanup(void *data)
> +{
>  #if (NGX_HAVE_PCRE_JIT)
> -
> -static void
> -ngx_pcre_free_studies(void *data)
> -{
> -    ngx_list_t *studies = data;
> +    ngx_regex_conf_t *rcf = data;
>  
>      ngx_uint_t        i;
>      ngx_list_part_t  *part;
>      ngx_regex_elt_t  *elts;
>  
> -    part = &studies->part;
> +    part = &rcf->studies->part;
>      elts = part->elts;
>  
>      for (i = 0; /* void */ ; i++) {
> @@ -274,56 +272,50 @@ ngx_pcre_free_studies(void *data)
>              i = 0;
>          }
>  
> -        if (elts[i].regex->extra != NULL) {
> -            pcre_free_study(elts[i].regex->extra);
> -        }
> -    }
> -}
> -
> -#endif
> -
> -
> -static ngx_int_t
> -ngx_regex_module_init(ngx_cycle_t *cycle)
> -{
> -    int               opt;
> -    const char       *errstr;
> -    ngx_uint_t        i;
> -    ngx_list_part_t  *part;
> -    ngx_regex_elt_t  *elts;
> -
> -    opt = 0;
> -
> -#if (NGX_HAVE_PCRE_JIT)
> -    {
> -    ngx_regex_conf_t    *rcf;
> -    ngx_pool_cleanup_t  *cln;
> -
> -    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
> -
> -    if (rcf->pcre_jit) {
> -        opt = PCRE_STUDY_JIT_COMPILE;
> -
>          /*
>           * The PCRE JIT compiler uses mmap for its executable codes, so we
>           * have to explicitly call the pcre_free_study() function to free
>           * this memory.
>           */
>  
> -        cln = ngx_pool_cleanup_add(cycle->pool, 0);
> -        if (cln == NULL) {
> -            return NGX_ERROR;
> +        if (elts[i].regex->extra != NULL) {
> +            pcre_free_study(elts[i].regex->extra);
>          }
> +    }
> +#endif
>  
> -        cln->handler = ngx_pcre_free_studies;
> -        cln->data = ngx_pcre_studies;
> -    }
> +    /*
> +     * On configuration parsing errors ngx_regex_module_init() will not
> +     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
> +     */
> +
> +    ngx_pcre_studies = NULL;
> +}
> +
> +
> +static ngx_int_t
> +ngx_regex_module_init(ngx_cycle_t *cycle)
> +{
> +    int                opt;
> +    const char        *errstr;
> +    ngx_uint_t         i;
> +    ngx_list_part_t   *part;
> +    ngx_regex_elt_t   *elts;
> +    ngx_regex_conf_t  *rcf;
> +
> +    opt = 0;
> +
> +    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
> +
> +#if (NGX_HAVE_PCRE_JIT)
> +    if (rcf->pcre_jit) {
> +        opt = PCRE_STUDY_JIT_COMPILE;
>      }
>  #endif
>  
>      ngx_regex_malloc_init(cycle->pool);
>  
> -    part = &ngx_pcre_studies->part;
> +    part = &rcf->studies->part;
>      elts = part->elts;
>  
>      for (i = 0; /* void */ ; i++) {
> @@ -374,7 +366,8 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>  static void *
>  ngx_regex_create_conf(ngx_cycle_t *cycle)
>  {
> -    ngx_regex_conf_t  *rcf;
> +    ngx_regex_conf_t    *rcf;
> +    ngx_pool_cleanup_t  *cln;
>  
>      rcf = ngx_pcalloc(cycle->pool, sizeof(ngx_regex_conf_t));
>      if (rcf == NULL) {
> @@ -383,11 +376,21 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
>  
>      rcf->pcre_jit = NGX_CONF_UNSET;
>  
> -    ngx_pcre_studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
> -    if (ngx_pcre_studies == NULL) {
> +    cln = ngx_pool_cleanup_add(cycle->pool, 0);
> +    if (cln == NULL) {
>          return NULL;
>      }
>  
> +    cln->handler = ngx_regex_cleanup;
> +    cln->data = rcf;
> +
> +    rcf->studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
> +    if (rcf->studies == NULL) {
> +        return NULL;
> +    }
> +
> +    ngx_pcre_studies = rcf->studies;
> +
>      return rcf;
>  }
>  
> # HG changeset patch
> # User Maxim Dounin 
> # Date 1639497274 -10800
> #      Tue Dec 14 18:54:34 2021 +0300
> # Node ID 8f00e80ade356c768a678c44711e8c2a7223ceef
> # Parent  4f979a9f2a68b25b8b2ce8a0bd15671095f6c327
> Core: ngx_regex.c style cleanup.
> 
> Notably, ngx_pcre_pool and ngx_pcre_studies are renamed to ngx_regex_pool
> and ngx_regex_studies, respectively.
> 
> diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
> --- a/src/core/ngx_regex.c
> +++ b/src/core/ngx_regex.c
> @@ -64,8 +64,8 @@ ngx_module_t  ngx_regex_module = {
>  };
>  
>  
> -static ngx_pool_t  *ngx_pcre_pool;
> -static ngx_list_t  *ngx_pcre_studies;
> +static ngx_pool_t  *ngx_regex_pool;
> +static ngx_list_t  *ngx_regex_studies;
>  
>  
>  void
> @@ -79,14 +79,14 @@ ngx_regex_init(void)
>  static ngx_inline void
>  ngx_regex_malloc_init(ngx_pool_t *pool)
>  {
> -    ngx_pcre_pool = pool;
> +    ngx_regex_pool = pool;
>  }
>  
>  
>  static ngx_inline void
>  ngx_regex_malloc_done(void)
>  {
> -    ngx_pcre_pool = NULL;
> +    ngx_regex_pool = NULL;
>  }
>  
>  
> @@ -112,13 +112,13 @@ ngx_regex_compile(ngx_regex_compile_t *r
>             rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
>                                "pcre_compile() failed: %s in \"%V\"",
>                                 errstr, &rc->pattern)
> -                      - rc->err.data;
> +                         - rc->err.data;
>  
>          } else {
>             rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
>                                "pcre_compile() failed: %s in \"%V\" at \"%s\"",
>                                 errstr, &rc->pattern, rc->pattern.data + erroff)
> -                      - rc->err.data;
> +                         - rc->err.data;
>          }
>  
>          return NGX_ERROR;
> @@ -133,8 +133,8 @@ ngx_regex_compile(ngx_regex_compile_t *r
>  
>      /* do not study at runtime */
>  
> -    if (ngx_pcre_studies != NULL) {
> -        elt = ngx_list_push(ngx_pcre_studies);
> +    if (ngx_regex_studies != NULL) {
> +        elt = ngx_list_push(ngx_regex_studies);
>          if (elt == NULL) {
>              goto nomem;
>          }
> @@ -229,11 +229,8 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
>  static void * ngx_libc_cdecl
>  ngx_regex_malloc(size_t size)
>  {
> -    ngx_pool_t      *pool;
> -    pool = ngx_pcre_pool;
> -
> -    if (pool) {
> -        return ngx_palloc(pool, size);
> +    if (ngx_regex_pool) {
> +        return ngx_palloc(ngx_regex_pool, size);
>      }
>  
>      return NULL;
> @@ -286,10 +283,10 @@ ngx_regex_cleanup(void *data)
>  
>      /*
>       * On configuration parsing errors ngx_regex_module_init() will not
> -     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
> +     * be called.  Make sure ngx_regex_studies is properly cleared anyway.
>       */
>  
> -    ngx_pcre_studies = NULL;
> +    ngx_regex_studies = NULL;
>  }
>  
>  
> @@ -357,7 +354,7 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>  
>      ngx_regex_malloc_done();
>  
> -    ngx_pcre_studies = NULL;
> +    ngx_regex_studies = NULL;
>  
>      return NGX_OK;
>  }
> @@ -389,7 +386,7 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
>          return NULL;
>      }
>  
> -    ngx_pcre_studies = rcf->studies;
> +    ngx_regex_studies = rcf->studies;
>  
>      return rcf;
>  }
> # HG changeset patch
> # User Maxim Dounin 
> # Date 1639497275 -10800
> #      Tue Dec 14 18:54:35 2021 +0300
> # Node ID 0a11d3d92a5d4c1ddcfdd1f07423d59b7c2aa863
> # Parent  8f00e80ade356c768a678c44711e8c2a7223ceef
> Configure: simplified PCRE compilation.
> 
> Removed ICC-specific PCRE optimizations which tried to link with PCRE
> object files instead of the library.  Made compiler-specific code
> minimal.
> 
> diff --git a/auto/lib/pcre/conf b/auto/lib/pcre/conf
> --- a/auto/lib/pcre/conf
> +++ b/auto/lib/pcre/conf
> @@ -4,81 +4,24 @@
>  
>  
>  if [ $PCRE != NONE ]; then
> +
> +    have=NGX_PCRE . auto/have
> +
> +    if [ "$NGX_PLATFORM" = win32 ]; then
> +        have=PCRE_STATIC . auto/have
> +    fi
> +
>      CORE_INCS="$CORE_INCS $PCRE"
> +    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
>  
>      case "$NGX_CC_NAME" in
>  
>          msvc | owc | bcc)
> -            have=NGX_PCRE . auto/have
> -            have=PCRE_STATIC . auto/have
> -            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
>              LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
>              CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
>          ;;
>  
> -        icc)
> -            have=NGX_PCRE . auto/have
> -            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
> -
> -            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
> -
> -            echo $ngx_n "checking for PCRE library ...$ngx_c"
> -
> -            if [ -f $PCRE/pcre.h ]; then
> -                ngx_pcre_ver=`grep PCRE_MAJOR $PCRE/pcre.h \
> -                              | sed -e 's/^.*PCRE_MAJOR.* \(.*\)$/\1/'`
> -
> -            else if [ -f $PCRE/configure.in ]; then
> -                ngx_pcre_ver=`grep PCRE_MAJOR= $PCRE/configure.in \
> -                              | sed -e 's/^.*=\(.*\)$/\1/'`
> -
> -            else
> -                ngx_pcre_ver=`grep pcre_major, $PCRE/configure.ac \
> -                              | sed -e 's/^.*pcre_major,.*\[\(.*\)\].*$/\1/'`
> -            fi
> -            fi
> -
> -            echo " $ngx_pcre_ver major version found"
> -
> -            # to allow -ipo optimization we link with the *.o but not library
> -
> -            case "$ngx_pcre_ver" in
> -                4|5)
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre.o"
> -                ;;
> -
> -                6)
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
> -                ;;
> -
> -                *)
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
> -                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_newline.o"
> -                ;;
> -
> -            esac
> -        ;;
> -
>          *)
> -            have=NGX_PCRE . auto/have
> -
> -            if [ "$NGX_PLATFORM" = win32 ]; then
> -                have=PCRE_STATIC . auto/have
> -            fi
> -
> -            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
>              LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
>              CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
>          ;;
> # HG changeset patch
> # User Maxim Dounin 
> # Date 1639497637 -10800
> #      Tue Dec 14 19:00:37 2021 +0300
> # Node ID 6915b92704ceca0030ea6d9e19f37e868a8d2303
> # Parent  0a11d3d92a5d4c1ddcfdd1f07423d59b7c2aa863
> PCRE2 library support.
> 
> The PCRE2 library is now used by default if found, instead of the
> original PCRE library.  If needed for some reason, this can be disabled
> with the --without-pcre2 configure option.
> 
> To make it possible to specify paths to the library and include files
> via --with-cc-opt / --with-ld-opt, the library is first tested without
> any additional paths and options.  If this fails, the pcre2-config script
> is used.
> 
> Similarly to the original PCRE library, it is now possible to build PCRE2
> from sources with nginx configure, by using the --with-pcre= option.
> It automatically detects if PCRE or PCRE2 sources are provided.
> 
> Note that compiling PCRE2 10.33 and later requires inttypes.h.  When
> compiling on Windows with MSVC, inttypes.h is only available starting
> with MSVC 2013.  In older versions some replacement needs to be provided
> ("echo '#include ' > pcre2-10.xx/src/inttypes.h" is good enough
> for MSVC 2010).
> 
> The interface on nginx side remains unchanged.
> 
> diff --git a/auto/lib/pcre/conf b/auto/lib/pcre/conf
> --- a/auto/lib/pcre/conf
> +++ b/auto/lib/pcre/conf
> @@ -5,29 +5,61 @@
>  
>  if [ $PCRE != NONE ]; then
>  
> -    have=NGX_PCRE . auto/have
> +    if [ -f $PCRE/src/pcre2.h.generic ]; then
> +
> +        PCRE_LIBRARY=PCRE2
> +
> +        have=NGX_PCRE . auto/have
> +        have=NGX_PCRE2 . auto/have
> +
> +        if [ "$NGX_PLATFORM" = win32 ]; then
> +            have=PCRE2_STATIC . auto/have
> +        fi
> +
> +        CORE_INCS="$CORE_INCS $PCRE/src/"
> +        CORE_DEPS="$CORE_DEPS $PCRE/src/pcre2.h"
>  
> -    if [ "$NGX_PLATFORM" = win32 ]; then
> -        have=PCRE_STATIC . auto/have
> -    fi
> +        case "$NGX_CC_NAME" in
> +
> +            msvc)
> +                LINK_DEPS="$LINK_DEPS $PCRE/src/pcre2-8.lib"
> +                CORE_LIBS="$CORE_LIBS $PCRE/src/pcre2-8.lib"
> +            ;;
>  
> -    CORE_INCS="$CORE_INCS $PCRE"
> -    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
> +            *)
> +                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre2-8.a"
> +                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre2-8.a"
> +            ;;
>  
> -    case "$NGX_CC_NAME" in
> +        esac
>  
> -        msvc | owc | bcc)
> -            LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
> -            CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
> -        ;;
> +    else
> +
> +        PCRE_LIBRARY=PCRE
> +
> +        have=NGX_PCRE . auto/have
> +
> +        if [ "$NGX_PLATFORM" = win32 ]; then
> +            have=PCRE_STATIC . auto/have
> +        fi
> +
> +        CORE_INCS="$CORE_INCS $PCRE"
> +        CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
>  
> -        *)
> -            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
> -            CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
> -        ;;
> +        case "$NGX_CC_NAME" in
> +
> +            msvc | owc | bcc)
> +                LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
> +                CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
> +            ;;
>  
> -    esac
> +            *)
> +                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
> +                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
> +            ;;
>  
> +        esac
> +    fi
>  
>      if [ $PCRE_JIT = YES ]; then
>          have=NGX_HAVE_PCRE_JIT . auto/have
> @@ -37,8 +69,48 @@ if [ $PCRE != NONE ]; then
>  else
>  
>      if [ "$NGX_PLATFORM" != win32 ]; then
> +        PCRE=NO
> +    fi
>  
> -        PCRE=NO
> +    if [ $PCRE = NO -a $PCRE2 != DISABLED ]; then
> +
> +        ngx_feature="PCRE2 library"
> +        ngx_feature_name="NGX_PCRE2"
> +        ngx_feature_run=no
> +        ngx_feature_incs="#define PCRE2_CODE_UNIT_WIDTH 8
> +                          #include "
> +        ngx_feature_path=
> +        ngx_feature_libs="-lpcre2-8"
> +        ngx_feature_test="pcre2_code *re;
> +                          re = pcre2_compile(NULL, 0, 0, NULL, NULL, NULL);
> +                          if (re == NULL) return 1"
> +        . auto/feature
> +
> +        if [ $ngx_found = no ]; then
> +
> +            # pcre2-config
> +
> +            ngx_pcre2_prefix=`pcre2-config --prefix 2>/dev/null`
> +
> +            if [ -n "$ngx_pcre2_prefix" ]; then
> +                ngx_feature="PCRE2 library in $ngx_pcre2_prefix"
> +                ngx_feature_path=`pcre2-config --cflags \
> +                                  | sed -n -e 's/.*-I *\([^ ][^ ]*\).*/\1/p'`
> +                ngx_feature_libs=`pcre2-config --libs8`
> +                . auto/feature
> +            fi
> +        fi
> +
> +        if [ $ngx_found = yes ]; then
> +            have=NGX_PCRE . auto/have
> +            CORE_INCS="$CORE_INCS $ngx_feature_path"
> +            CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
> +            PCRE=YES
> +            PCRE_LIBRARY=PCRE2
> +        fi
> +    fi
> +
> +    if [ $PCRE = NO ]; then
>  
>          ngx_feature="PCRE library"
>          ngx_feature_name="NGX_PCRE"
> @@ -114,6 +186,7 @@ else
>              CORE_INCS="$CORE_INCS $ngx_feature_path"
>              CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
>              PCRE=YES
> +            PCRE_LIBRARY=PCRE
>          fi
>  
>          if [ $PCRE = YES ]; then
> diff --git a/auto/lib/pcre/make b/auto/lib/pcre/make
> --- a/auto/lib/pcre/make
> +++ b/auto/lib/pcre/make
> @@ -3,36 +3,138 @@
>  # Copyright (C) Nginx, Inc.
>  
>  
> -case "$NGX_CC_NAME" in
> +if [ $PCRE_LIBRARY = PCRE2 ]; then
> +
> +    # PCRE2
> +
> +    if [ $NGX_CC_NAME = msvc ]; then
> +
> +        # With PCRE2, it is not possible to compile all sources.
> +        # Since list of source files changes between versions, we
> +        # test files which might not be present.
>  
> -    msvc)
> -        ngx_makefile=makefile.msvc
> -        ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
> -        ngx_pcre="PCRE=\"$PCRE\""
> -    ;;
> +        ngx_pcre_srcs="pcre2_auto_possess.c \
> +                       pcre2_chartables.c \
> +                       pcre2_compile.c \
> +                       pcre2_config.c \
> +                       pcre2_context.c \
> +                       pcre2_dfa_match.c \
> +                       pcre2_error.c \
> +                       pcre2_jit_compile.c \
> +                       pcre2_maketables.c \
> +                       pcre2_match.c \
> +                       pcre2_match_data.c \
> +                       pcre2_newline.c \
> +                       pcre2_ord2utf.c \
> +                       pcre2_pattern_info.c \
> +                       pcre2_string_utils.c \
> +                       pcre2_study.c \
> +                       pcre2_substitute.c \
> +                       pcre2_substring.c \
> +                       pcre2_tables.c \
> +                       pcre2_ucd.c \
> +                       pcre2_valid_utf.c \
> +                       pcre2_xclass.c"
> +
> +        ngx_pcre_test="pcre2_convert.c \
> +                       pcre2_extuni.c \
> +                       pcre2_find_bracket.c \
> +                       pcre2_script_run.c \
> +                       pcre2_serialize.c"
> +
> +        for ngx_src in $ngx_pcre_test
> +        do
> +            if [ -f $PCRE/src/$ngx_src ]; then
> +                ngx_pcre_srcs="$ngx_pcre_srcs $ngx_src"
> +            fi
> +        done
>  
> -    owc)
> -        ngx_makefile=makefile.owc
> -        ngx_opt="CPU_OPT=\"$CPU_OPT\""
> -        ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
> -    ;;
> +        ngx_pcre_objs=`echo $ngx_pcre_srcs \
> +            | sed -e "s#\([^ ]*\.\)c#\1$ngx_objext#g"`
> +
> +        ngx_pcre_srcs=`echo $ngx_pcre_srcs \
> +            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
> +        ngx_pcre_objs=`echo $ngx_pcre_objs \
> +            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
> +
> +        cat << END                                            >> $NGX_MAKEFILE
> +
> +PCRE_CFLAGS =	-O2 -Ob1 -Oi -Gs $LIBC $CPU_OPT
> +PCRE_FLAGS =	-DHAVE_CONFIG_H -DPCRE2_STATIC -DPCRE2_CODE_UNIT_WIDTH=8 \\
> +		-DHAVE_MEMMOVE
> +
> +PCRE_SRCS =	 $ngx_pcre_srcs
> +PCRE_OBJS =	 $ngx_pcre_objs
> +
> +$PCRE/src/pcre2.h:
> +	cd $PCRE/src \\
> +	&& copy /y config.h.generic config.h \\
> +	&& copy /y pcre2.h.generic pcre2.h \\
> +	&& copy /y pcre2_chartables.c.dist pcre2_chartables.c
>  
> -    bcc)
> -        ngx_makefile=makefile.bcc
> -        ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
> -        ngx_pcre=`echo \-DPCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
> -    ;;
> +$PCRE/src/pcre2-8.lib:	$PCRE/src/pcre2.h $NGX_MAKEFILE
> +	cd $PCRE/src \\
> +	&& cl -nologo -c \$(PCRE_CFLAGS) -I . \$(PCRE_FLAGS) \$(PCRE_SRCS) \\
> +	&& link -lib -out:pcre2-8.lib -verbose:lib \$(PCRE_OBJS)
> +
> +END
> +
> +    else
> +
> +        cat << END                                            >> $NGX_MAKEFILE
> +
> +$PCRE/src/pcre2.h:	$PCRE/Makefile
>  
> -    *)
> -        ngx_makefile=
> -    ;;
> +$PCRE/Makefile:	$NGX_MAKEFILE
> +	cd $PCRE \\
> +	&& if [ -f Makefile ]; then \$(MAKE) distclean; fi \\
> +	&& CC="\$(CC)" CFLAGS="$PCRE_OPT" \\
> +	./configure --disable-shared $PCRE_CONF_OPT
>  
> -esac
> +$PCRE/.libs/libpcre2-8.a:	$PCRE/Makefile
> +	cd $PCRE \\
> +	&& \$(MAKE) libpcre2-8.la
> +
> +END
> +
> +    fi
>  
>  
> -if [ -n "$ngx_makefile" ]; then
> +else
> +
> +    # PCRE
> +
> +    case "$NGX_CC_NAME" in
> +
> +        msvc)
> +            ngx_makefile=makefile.msvc
> +            ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
> +            ngx_pcre="PCRE=\"$PCRE\""
> +        ;;
> +
> +        owc)
> +            ngx_makefile=makefile.owc
> +            ngx_opt="CPU_OPT=\"$CPU_OPT\""
> +            ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
> +        ;;
>  
> -    cat << END                                                >> $NGX_MAKEFILE
> +        bcc)
> +            ngx_makefile=makefile.bcc
> +            ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
> +            ngx_pcre=`echo \-DPCRE=\"$PCRE\" \
> +                | sed -e "s/\//$ngx_regex_dirsep/g"`
> +        ;;
> +
> +        *)
> +            ngx_makefile=
> +        ;;
> +
> +    esac
> +
> +
> +    if [ -n "$ngx_makefile" ]; then
> +
> +        cat << END                                            >> $NGX_MAKEFILE
>  
>  `echo "$PCRE/pcre.lib:	$PCRE/pcre.h $NGX_MAKEFILE"			\
>  	| sed -e "s/\//$ngx_regex_dirsep/g"`
> @@ -43,9 +145,9 @@ if [ -n "$ngx_makefile" ]; then
>  
>  END
>  
> -else
> +    else
>  
> -    cat << END                                                >> $NGX_MAKEFILE
> +        cat << END                                            >> $NGX_MAKEFILE
>  
>  $PCRE/pcre.h:	$PCRE/Makefile
>  
> @@ -61,4 +163,6 @@ else
>  
>  END
>  
> +    fi
> +
>  fi
> diff --git a/auto/options b/auto/options
> --- a/auto/options
> +++ b/auto/options
> @@ -146,6 +146,7 @@ PCRE=NONE
>  PCRE_OPT=
>  PCRE_CONF_OPT=
>  PCRE_JIT=NO
> +PCRE2=YES
>  
>  USE_OPENSSL=NO
>  OPENSSL=NONE
> @@ -357,6 +358,7 @@ use the \"--with-mail_ssl_module\" optio
>          --with-pcre=*)                   PCRE="$value"              ;;
>          --with-pcre-opt=*)               PCRE_OPT="$value"          ;;
>          --with-pcre-jit)                 PCRE_JIT=YES               ;;
> +        --without-pcre2)                 PCRE2=DISABLED             ;;
>  
>          --with-openssl=*)                OPENSSL="$value"           ;;
>          --with-openssl-opt=*)            OPENSSL_OPT="$value"       ;;
> @@ -573,6 +575,7 @@ cat << END
>    --with-pcre=DIR                    set path to PCRE library sources
>    --with-pcre-opt=OPTIONS            set additional build options for PCRE
>    --with-pcre-jit                    build PCRE with JIT compilation support
> +  --without-pcre2                    do not use PCRE2 library
>  
>    --with-zlib=DIR                    set path to zlib library sources
>    --with-zlib-opt=OPTIONS            set additional build options for zlib
> diff --git a/auto/summary b/auto/summary
> --- a/auto/summary
> +++ b/auto/summary
> @@ -16,9 +16,9 @@ if [ $USE_PCRE = DISABLED ]; then
>  
>  else
>      case $PCRE in
> -        YES)   echo "  + using system PCRE library" ;;
> +        YES)   echo "  + using system $PCRE_LIBRARY library" ;;
>          NONE)  echo "  + PCRE library is not used" ;;
> -        *)     echo "  + using PCRE library: $PCRE" ;;
> +        *)     echo "  + using $PCRE_LIBRARY library: $PCRE" ;;
>      esac
>  fi
>  
> diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
> --- a/src/core/ngx_regex.c
> +++ b/src/core/ngx_regex.c
> @@ -15,8 +15,16 @@ typedef struct {
>  } ngx_regex_conf_t;
>  
>  
> +static ngx_inline void ngx_regex_malloc_init(ngx_pool_t *pool);
> +static ngx_inline void ngx_regex_malloc_done(void);
> +
> +#if (NGX_PCRE2)
> +static void * ngx_libc_cdecl ngx_regex_malloc(size_t size, void *data);
> +static void ngx_libc_cdecl ngx_regex_free(void *p, void *data);
> +#else
>  static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
>  static void ngx_libc_cdecl ngx_regex_free(void *p);
> +#endif
>  static void ngx_regex_cleanup(void *data);
>  
>  static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
> @@ -64,15 +72,24 @@ ngx_module_t  ngx_regex_module = {
>  };
>  
>  
> -static ngx_pool_t  *ngx_regex_pool;
> -static ngx_list_t  *ngx_regex_studies;
> +static ngx_pool_t             *ngx_regex_pool;
> +static ngx_list_t             *ngx_regex_studies;
> +static ngx_uint_t              ngx_regex_direct_alloc;
> +
> +#if (NGX_PCRE2)
> +static pcre2_compile_context  *ngx_regex_compile_context;
> +static pcre2_match_data       *ngx_regex_match_data;
> +static ngx_uint_t              ngx_regex_match_data_size;
> +#endif
>  
>  
>  void
>  ngx_regex_init(void)
>  {
> +#if !(NGX_PCRE2)
>      pcre_malloc = ngx_regex_malloc;
>      pcre_free = ngx_regex_free;
> +#endif
>  }
>  
>  
> @@ -80,6 +97,7 @@ static ngx_inline void
>  ngx_regex_malloc_init(ngx_pool_t *pool)
>  {
>      ngx_regex_pool = pool;
> +    ngx_regex_direct_alloc = (pool == NULL) ? 1 : 0;
>  }
>  
>  
> @@ -87,9 +105,146 @@ static ngx_inline void
>  ngx_regex_malloc_done(void)
>  {
>      ngx_regex_pool = NULL;
> +    ngx_regex_direct_alloc = 0;
>  }
>  
>  
> +#if (NGX_PCRE2)
> +
> +ngx_int_t
> +ngx_regex_compile(ngx_regex_compile_t *rc)
> +{
> +    int                     n, errcode;
> +    char                   *p;
> +    u_char                  errstr[128];
> +    size_t                  erroff;
> +    pcre2_code             *re;
> +    ngx_regex_elt_t        *elt;
> +    pcre2_general_context  *gctx;
> +    pcre2_compile_context  *cctx;
> +
> +    if (ngx_regex_compile_context == NULL) {
> +        /*
> +         * Allocte a compile context if not yet allocated.  This uses
> +         * direct allocations from heap, so the result can be cached
> +         * even at runtime.
> +         */
> +
> +        ngx_regex_malloc_init(NULL);
> +
> +        gctx = pcre2_general_context_create(ngx_regex_malloc, ngx_regex_free,
> +                                            NULL);
> +        if (gctx == NULL) {
> +            ngx_regex_malloc_done();
> +            goto nomem;
> +        }
> +
> +        cctx = pcre2_compile_context_create(gctx);
> +        if (cctx == NULL) {
> +            pcre2_general_context_free(gctx);
> +            ngx_regex_malloc_done();
> +            goto nomem;
> +        }
> +
> +        ngx_regex_compile_context = cctx;
> +
> +        pcre2_general_context_free(gctx);
> +        ngx_regex_malloc_done();
> +    }
> +
> +    ngx_regex_malloc_init(rc->pool);
> +
> +    re = pcre2_compile(rc->pattern.data, rc->pattern.len,
> +                       (uint32_t) rc->options, &errcode, &erroff,
> +                       ngx_regex_compile_context);
> +
> +    /* ensure that there is no current pool */
> +    ngx_regex_malloc_done();
> +
> +    if (re == NULL) {
> +        pcre2_get_error_message(errcode, errstr, 128);
> +
> +        if ((size_t) erroff == rc->pattern.len) {
> +            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
> +                              "pcre2_compile() failed: %s in \"%V\"",
> +                               errstr, &rc->pattern)
> +                          - rc->err.data;
> +
> +        } else {
> +            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
> +                              "pcre2_compile() failed: %s in \"%V\" at \"%s\"",
> +                               errstr, &rc->pattern, rc->pattern.data + erroff)
> +                          - rc->err.data;
> +        }
> +
> +        return NGX_ERROR;
> +    }
> +
> +    rc->regex = re;
> +
> +    /* do not study at runtime */
> +
> +    if (ngx_regex_studies != NULL) {
> +        elt = ngx_list_push(ngx_regex_studies);
> +        if (elt == NULL) {
> +            goto nomem;
> +        }
> +
> +        elt->regex = rc->regex;
> +        elt->name = rc->pattern.data;
> +    }
> +
> +    n = pcre2_pattern_info(re, PCRE2_INFO_CAPTURECOUNT, &rc->captures);
> +    if (n < 0) {
> +        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_CAPTURECOUNT) failed: %d";
> +        goto failed;
> +    }
> +
> +    if (rc->captures == 0) {
> +        return NGX_OK;
> +    }
> +
> +    n = pcre2_pattern_info(re, PCRE2_INFO_NAMECOUNT, &rc->named_captures);
> +    if (n < 0) {
> +        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMECOUNT) failed: %d";
> +        goto failed;
> +    }
> +
> +    if (rc->named_captures == 0) {
> +        return NGX_OK;
> +    }
> +
> +    n = pcre2_pattern_info(re, PCRE2_INFO_NAMEENTRYSIZE, &rc->name_size);
> +    if (n < 0) {
> +        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMEENTRYSIZE) failed: %d";
> +        goto failed;
> +    }
> +
> +    n = pcre2_pattern_info(re, PCRE2_INFO_NAMETABLE, &rc->names);
> +    if (n < 0) {
> +        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMETABLE) failed: %d";
> +        goto failed;
> +    }
> +
> +    return NGX_OK;
> +
> +failed:
> +
> +    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len, p, &rc->pattern, n)
> +                  - rc->err.data;
> +    return NGX_ERROR;
> +
> +nomem:
> +
> +    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
> +                               "regex \"%V\" compilation failed: no memory",
> +                               &rc->pattern)
> +                  - rc->err.data;
> +    return NGX_ERROR;
> +}
> +
> +#else
> +
>  ngx_int_t
>  ngx_regex_compile(ngx_regex_compile_t *rc)
>  {
> @@ -192,6 +347,74 @@ nomem:
>      return NGX_ERROR;
>  }
>  
> +#endif
> +
> +
> +#if (NGX_PCRE2)
> +
> +ngx_int_t
> +ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures, ngx_uint_t size)
> +{
> +    size_t      *ov;
> +    ngx_int_t    rc;
> +    ngx_uint_t   n, i;
> +
> +    /*
> +     * The pcre2_match() function might allocate memory for backtracking
> +     * frames, typical allocations are from 40k and above.  So the allocator
> +     * is configured to do direct allocations from heap during matching.
> +     */
> +
> +    ngx_regex_malloc_init(NULL);
> +
> +    if (ngx_regex_match_data == NULL
> +        || size > ngx_regex_match_data_size)
> +    {
> +        /*
> +         * Allocate a match data if not yet allocated or smaller than
> +         * needed.
> +         */
> +
> +        if (ngx_regex_match_data) {
> +            pcre2_match_data_free(ngx_regex_match_data);
> +        }
> +
> +        ngx_regex_match_data_size = size;
> +        ngx_regex_match_data = pcre2_match_data_create(size / 3, NULL);
> +
> +        if (ngx_regex_match_data == NULL) {
> +            rc = PCRE2_ERROR_NOMEMORY;
> +            goto failed;
> +        }
> +    }
> +
> +    rc = pcre2_match(re, s->data, s->len, 0, 0, ngx_regex_match_data, NULL);
> +
> +    if (rc < 0) {
> +        goto failed;
> +    }
> +
> +    n = pcre2_get_ovector_count(ngx_regex_match_data);
> +    ov = pcre2_get_ovector_pointer(ngx_regex_match_data);
> +
> +    if (n > size / 3) {
> +        n = size / 3;
> +    }
> +
> +    for (i = 0; i < n; i++) {
> +        captures[i * 2] = ov[i * 2];
> +        captures[i * 2 + 1] = ov[i * 2 + 1];
> +    }
> +
> +failed:
> +
> +    ngx_regex_malloc_done();
> +
> +    return rc;
> +}
> +
> +#endif
> +
>  
>  ngx_int_t
>  ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log)
> @@ -226,6 +449,35 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
>  }
>  
>  
> +#if (NGX_PCRE2)
> +
> +static void * ngx_libc_cdecl
> +ngx_regex_malloc(size_t size, void *data)
> +{
> +    if (ngx_regex_pool) {
> +        return ngx_palloc(ngx_regex_pool, size);
> +    }
> +
> +    if (ngx_regex_direct_alloc) {
> +        return ngx_alloc(size, ngx_cycle->log);
> +    }
> +
> +    return NULL;
> +}
> +
> +
> +static void ngx_libc_cdecl
> +ngx_regex_free(void *p, void *data)
> +{
> +    if (ngx_regex_direct_alloc) {
> +        ngx_free(p);
> +    }
> +
> +    return;
> +}
> +
> +#else
> +
>  static void * ngx_libc_cdecl
>  ngx_regex_malloc(size_t size)
>  {
> @@ -243,11 +495,13 @@ ngx_regex_free(void *p)
>      return;
>  }
>  
> +#endif
> +
>  
>  static void
>  ngx_regex_cleanup(void *data)
>  {
> -#if (NGX_HAVE_PCRE_JIT)
> +#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
>      ngx_regex_conf_t *rcf = data;
>  
>      ngx_uint_t        i;
> @@ -272,12 +526,17 @@ ngx_regex_cleanup(void *data)
>          /*
>           * The PCRE JIT compiler uses mmap for its executable codes, so we
>           * have to explicitly call the pcre_free_study() function to free
> -         * this memory.
> +         * this memory.  In PCRE2, we call the pcre2_code_free() function
> +         * for the same reason.
>           */
>  
> +#if (NGX_PCRE2)
> +        pcre2_code_free(elts[i].regex);
> +#else
>          if (elts[i].regex->extra != NULL) {
>              pcre_free_study(elts[i].regex->extra);
>          }
> +#endif
>      }
>  #endif
>  
> @@ -287,6 +546,26 @@ ngx_regex_cleanup(void *data)
>       */
>  
>      ngx_regex_studies = NULL;
> +
> +#if (NGX_PCRE2)
> +
> +    /*
> +     * Free compile context and match data.  If needed at runtime by
> +     * the new cycle, these will be re-allocated.
> +     */
> +
> +    if (ngx_regex_compile_context) {
> +        pcre2_compile_context_free(ngx_regex_compile_context);
> +        ngx_regex_compile_context = NULL;
> +    }
> +
> +    if (ngx_regex_match_data) {
> +        pcre2_match_data_free(ngx_regex_match_data);
> +        ngx_regex_match_data = NULL;
> +        ngx_regex_match_data_size = 0;
> +    }
> +
> +#endif
>  }
>  
>  
> @@ -294,7 +573,9 @@ static ngx_int_t
>  ngx_regex_module_init(ngx_cycle_t *cycle)
>  {
>      int                opt;
> +#if !(NGX_PCRE2)
>      const char        *errstr;
> +#endif
>      ngx_uint_t         i;
>      ngx_list_part_t   *part;
>      ngx_regex_elt_t   *elts;
> @@ -304,10 +585,16 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>  
>      rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
>  
> -#if (NGX_HAVE_PCRE_JIT)
> +#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
> +
>      if (rcf->pcre_jit) {
> +#if (NGX_PCRE2)
> +        opt = 1;
> +#else
>          opt = PCRE_STUDY_JIT_COMPILE;
> +#endif
>      }
> +
>  #endif
>  
>      ngx_regex_malloc_init(cycle->pool);
> @@ -327,6 +614,23 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>              i = 0;
>          }
>  
> +#if (NGX_PCRE2)
> +
> +        if (opt) {
> +            int  n;
> +
> +            n = pcre2_jit_compile(elts[i].regex, PCRE2_JIT_COMPLETE);
> +
> +            if (n != 0) {
> +                ngx_log_error(NGX_LOG_INFO, cycle->log, 0,
> +                              "pcre2_jit_compile() failed: %d in \"%s\", "
> +                              "ignored",
> +                              n, elts[i].name);
> +            }
> +        }
> +
> +#else
> +
>          elts[i].regex->extra = pcre_study(elts[i].regex->code, opt, &errstr);
>  
>          if (errstr != NULL) {
> @@ -350,11 +654,15 @@ ngx_regex_module_init(ngx_cycle_t *cycle
>              }
>          }
>  #endif
> +#endif
>      }
>  
>      ngx_regex_malloc_done();
>  
>      ngx_regex_studies = NULL;
> +#if (NGX_PCRE2)
> +    ngx_regex_compile_context = NULL;
> +#endif
>  
>      return NGX_OK;
>  }
> @@ -412,7 +720,21 @@ ngx_regex_pcre_jit(ngx_conf_t *cf, void 
>          return NGX_CONF_OK;
>      }
>  
> -#if (NGX_HAVE_PCRE_JIT)
> +#if (NGX_PCRE2)
> +    {
> +    int       r;
> +    uint32_t  jit;
> +
> +    jit = 0;
> +    r = pcre2_config(PCRE2_CONFIG_JIT, &jit);
> +
> +    if (r != 0 || jit != 1) {
> +        ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
> +                           "PCRE2 library does not support JIT");
> +        *fp = 0;
> +    }
> +    }
> +#elif (NGX_HAVE_PCRE_JIT)
>      {
>      int  jit, r;
>  
> diff --git a/src/core/ngx_regex.h b/src/core/ngx_regex.h
> --- a/src/core/ngx_regex.h
> +++ b/src/core/ngx_regex.h
> @@ -12,19 +12,31 @@
>  #include 
>  #include 
>  
> +
> +#if (NGX_PCRE2)
> +
> +#define PCRE2_CODE_UNIT_WIDTH  8
> +#include 
> +
> +#define NGX_REGEX_NO_MATCHED   PCRE2_ERROR_NOMATCH   /* -1 */
> +#define NGX_REGEX_CASELESS     PCRE2_CASELESS
> +
> +typedef pcre2_code  ngx_regex_t;
> +
> +#else
> +
>  #include 
>  
> -
> -#define NGX_REGEX_NO_MATCHED  PCRE_ERROR_NOMATCH   /* -1 */
> -
> -#define NGX_REGEX_CASELESS    PCRE_CASELESS
> -
> +#define NGX_REGEX_NO_MATCHED   PCRE_ERROR_NOMATCH   /* -1 */
> +#define NGX_REGEX_CASELESS     PCRE_CASELESS
>  
>  typedef struct {
>      pcre        *code;
>      pcre_extra  *extra;
>  } ngx_regex_t;
>  
> +#endif
> +
>  
>  typedef struct {
>      ngx_str_t     pattern;
> @@ -49,10 +61,20 @@ typedef struct {
>  void ngx_regex_init(void);
>  ngx_int_t ngx_regex_compile(ngx_regex_compile_t *rc);
>  
> +#if (NGX_PCRE2)
> +
> +ngx_int_t ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures,
> +    ngx_uint_t size);
> +#define ngx_regex_exec_n       "pcre2_match()"
> +
> +#else
> +
>  #define ngx_regex_exec(re, s, captures, size)                                \
>      pcre_exec(re->code, re->extra, (const char *) (s)->data, (s)->len, 0, 0, \
>                captures, size)
> -#define ngx_regex_exec_n      "pcre_exec()"
> +#define ngx_regex_exec_n       "pcre_exec()"
> +
> +#endif
>  
>  ngx_int_t ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log);
>  
> 

Additional patch to make it possible to use modules compiled with 
different PCRE library variant:

# HG changeset patch
# User Maxim Dounin 
# Date 1639677470 -10800
#      Thu Dec 16 20:57:50 2021 +0300
# Node ID 86fbbba183881417965412791eaa840f8fb67005
# Parent  6915b92704ceca0030ea6d9e19f37e868a8d2303
PCRE2 and PCRE binary compatibility.

With this change, dynamic modules using nginx regex interface can be used
regardless of the variant of the PCRE library nginx was compiled with.

If a module is compiled with different PCRE library variant, in case of
ngx_regex_exec() errors it will report wrong function name in error
messages.  This is believed to be tolerable, given that fixing this will
require interface changes.

diff --git a/src/core/ngx_regex.c b/src/core/ngx_regex.c
--- a/src/core/ngx_regex.c
+++ b/src/core/ngx_regex.c
@@ -118,6 +118,7 @@ ngx_regex_compile(ngx_regex_compile_t *r
     char                   *p;
     u_char                  errstr[128];
     size_t                  erroff;
+    uint32_t                options;
     pcre2_code             *re;
     ngx_regex_elt_t        *elt;
     pcre2_general_context  *gctx;
@@ -152,11 +153,24 @@ ngx_regex_compile(ngx_regex_compile_t *r
         ngx_regex_malloc_done();
     }
 
+    options = 0;
+
+    if (rc->options & NGX_REGEX_CASELESS) {
+        options |= PCRE2_CASELESS;
+    }
+
+    if (rc->options & ~NGX_REGEX_CASELESS) {
+        rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                            "regex \"%V\" compilation failed: invalid options",
+                            &rc->pattern)
+                      - rc->err.data;
+        return NGX_ERROR;
+    }
+
     ngx_regex_malloc_init(rc->pool);
 
-    re = pcre2_compile(rc->pattern.data, rc->pattern.len,
-                       (uint32_t) rc->options, &errcode, &erroff,
-                       ngx_regex_compile_context);
+    re = pcre2_compile(rc->pattern.data, rc->pattern.len, options,
+                       &errcode, &erroff, ngx_regex_compile_context);
 
     /* ensure that there is no current pool */
     ngx_regex_malloc_done();
@@ -252,11 +266,32 @@ ngx_regex_compile(ngx_regex_compile_t *r
     char             *p;
     pcre             *re;
     const char       *errstr;
+    ngx_uint_t        options;
     ngx_regex_elt_t  *elt;
 
+#if (NGX_REGEX_CASELESS != PCRE_CASELESS)
+
+    options = 0;
+
+    if (rc->options & NGX_REGEX_CASELESS) {
+        options |= PCRE_CASELESS;
+    }
+
+    if (rc->options & ~NGX_REGEX_CASELESS) {
+        rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                            "regex \"%V\" compilation failed: invalid options",
+                            &rc->pattern)
+                      - rc->err.data;
+        return NGX_ERROR;
+    }
+
+#else
+    options = rc->options;
+#endif
+
     ngx_regex_malloc_init(rc->pool);
 
-    re = pcre_compile((const char *) rc->pattern.data, (int) rc->options,
+    re = pcre_compile((const char *) rc->pattern.data, (int) options,
                       &errstr, &erroff, NULL);
 
     /* ensure that there is no current pool */
@@ -413,6 +448,15 @@ failed:
     return rc;
 }
 
+#else
+
+ngx_int_t
+ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures, ngx_uint_t size)
+{
+    return pcre_exec(re->code, re->extra, (const char *) s->data, s->len,
+                     0, 0, captures, size);
+}
+
 #endif
 
 
diff --git a/src/core/ngx_regex.h b/src/core/ngx_regex.h
--- a/src/core/ngx_regex.h
+++ b/src/core/ngx_regex.h
@@ -19,7 +19,6 @@
 #include 
 
 #define NGX_REGEX_NO_MATCHED   PCRE2_ERROR_NOMATCH   /* -1 */
-#define NGX_REGEX_CASELESS     PCRE2_CASELESS
 
 typedef pcre2_code  ngx_regex_t;
 
@@ -28,7 +27,6 @@ typedef pcre2_code  ngx_regex_t;
 #include 
 
 #define NGX_REGEX_NO_MATCHED   PCRE_ERROR_NOMATCH   /* -1 */
-#define NGX_REGEX_CASELESS     PCRE_CASELESS
 
 typedef struct {
     pcre        *code;
@@ -38,10 +36,13 @@ typedef struct {
 #endif
 
 
+#define NGX_REGEX_CASELESS     0x00000001
+
+
 typedef struct {
     ngx_str_t     pattern;
     ngx_pool_t   *pool;
-    ngx_int_t     options;
+    ngx_uint_t    options;
 
     ngx_regex_t  *regex;
     int           captures;
@@ -61,19 +62,13 @@ typedef struct {
 void ngx_regex_init(void);
 ngx_int_t ngx_regex_compile(ngx_regex_compile_t *rc);
 
-#if (NGX_PCRE2)
-
 ngx_int_t ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures,
     ngx_uint_t size);
+
+#if (NGX_PCRE2)
 #define ngx_regex_exec_n       "pcre2_match()"
-
 #else
-
-#define ngx_regex_exec(re, s, captures, size)                                \
-    pcre_exec(re->code, re->extra, (const char *) (s)->data, (s)->len, 0, 0, \
-              captures, size)
 #define ngx_regex_exec_n       "pcre_exec()"
-
 #endif
 
 ngx_int_t ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log);

-- 
Maxim Dounin
http://mdounin.ru/

From fengyoulin at live.com  Fri Dec 17 05:59:40 2021
From: fengyoulin at live.com (=?iso-8859-1?q?Youlin_Feng?=)
Date: Fri, 17 Dec 2021 13:59:40 +0800
Subject: [PATCH] HTTP: keepalive_graceful_close support
Message-ID: 

# HG changeset patch
# User Youlin Feng 
# Date 1639718067 -28800
#      Fri Dec 17 13:14:27 2021 +0800
# Node ID 54db7272bb0c9040eaf657f92bb02c9147d927e6
# Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
HTTP: keepalive_graceful_close support.

Previously, keepalived connections will be suddenly closed during worker
process exiting, without notification to the client. The client may be
sending a request when the connection is closed, resulting in an error.

With "keepalive_graceful_close on;", the client will receive an
"Connection: close" response header in the last request, then the
connection can be closed gracefully.

diff -r a7a77549265e -r 54db7272bb0c src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c	Thu Nov 25 22:02:10 2021 +0300
+++ b/src/http/ngx_http_core_module.c	Fri Dec 17 13:14:27 2021 +0800
@@ -523,6 +523,13 @@
       offsetof(ngx_http_core_loc_conf_t, keepalive_disable),
       &ngx_http_core_keepalive_disable },
 
+    { ngx_string("keepalive_graceful_close"),
+      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG,
+      ngx_conf_set_flag_slot,
+      NGX_HTTP_LOC_CONF_OFFSET,
+      offsetof(ngx_http_core_loc_conf_t, keepalive_graceful_close),
+      NULL },
+
     { ngx_string("satisfy"),
       NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
       ngx_conf_set_enum_slot,
@@ -826,7 +833,7 @@
     if (!r->internal) {
         switch (r->headers_in.connection_type) {
         case 0:
-            r->keepalive = (r->http_version > NGX_HTTP_VERSION_10);
+            r->keepalive = (r->http_version > NGX_HTTP_VERSION_10) && !ngx_exiting;
             break;
 
         case NGX_HTTP_CONNECTION_CLOSE:
@@ -834,7 +841,7 @@
             break;
 
         case NGX_HTTP_CONNECTION_KEEP_ALIVE:
-            r->keepalive = 1;
+            r->keepalive = !ngx_exiting;
             break;
         }
 
@@ -3517,6 +3524,7 @@
     clcf->keepalive_timeout = NGX_CONF_UNSET_MSEC;
     clcf->keepalive_header = NGX_CONF_UNSET;
     clcf->keepalive_requests = NGX_CONF_UNSET_UINT;
+    clcf->keepalive_graceful_close = NGX_CONF_UNSET;
     clcf->lingering_close = NGX_CONF_UNSET_UINT;
     clcf->lingering_time = NGX_CONF_UNSET_MSEC;
     clcf->lingering_timeout = NGX_CONF_UNSET_MSEC;
@@ -3756,6 +3764,8 @@
                               prev->keepalive_header, 0);
     ngx_conf_merge_uint_value(conf->keepalive_requests,
                               prev->keepalive_requests, 1000);
+    ngx_conf_merge_value(conf->keepalive_graceful_close,
+                              prev->keepalive_graceful_close, 0);
     ngx_conf_merge_uint_value(conf->lingering_close,
                               prev->lingering_close, NGX_HTTP_LINGERING_ON);
     ngx_conf_merge_msec_value(conf->lingering_time,
diff -r a7a77549265e -r 54db7272bb0c src/http/ngx_http_core_module.h
--- a/src/http/ngx_http_core_module.h	Thu Nov 25 22:02:10 2021 +0300
+++ b/src/http/ngx_http_core_module.h	Fri Dec 17 13:14:27 2021 +0800
@@ -371,6 +371,8 @@
 
     time_t        keepalive_header;        /* keepalive_timeout */
 
+    ngx_flag_t    keepalive_graceful_close; /* keepalive_graceful_close */
+
     ngx_uint_t    keepalive_requests;      /* keepalive_requests */
     ngx_uint_t    keepalive_disable;       /* keepalive_disable */
     ngx_uint_t    satisfy;                 /* satisfy */
diff -r a7a77549265e -r 54db7272bb0c src/http/ngx_http_request.c
--- a/src/http/ngx_http_request.c	Thu Nov 25 22:02:10 2021 +0300
+++ b/src/http/ngx_http_request.c	Fri Dec 17 13:14:27 2021 +0800
@@ -2767,7 +2767,7 @@
     }
 
     if (!ngx_terminate
-         && !ngx_exiting
+         && (!ngx_exiting || clcf->keepalive_graceful_close)
          && r->keepalive
          && clcf->keepalive_timeout > 0)
     {
@@ -3252,7 +3252,7 @@
     r->http_state = NGX_HTTP_KEEPALIVE_STATE;
 #endif
 
-    c->idle = 1;
+    c->idle = !clcf->keepalive_graceful_close;
     ngx_reusable_connection(c, 1);
 
     ngx_add_timer(rev, clcf->keepalive_timeout);


From karetirk at gmail.com  Sun Dec 19 18:24:38 2021
From: karetirk at gmail.com (Kareti Ramakrishna MBA)
Date: Sun, 19 Dec 2021 23:54:38 +0530
Subject: nginx-quic, php able to access last set cookie only
In-Reply-To: 
References: 
 
Message-ID: 

I implemented nginx-quic using the steps at
https://quic.nginx.org/readme.html

The page is validating http3 quic at https://http3check.net and
https://gf.dev/http3-test

The page elements show h3 protocol in developer tools network tab.

in a test.php page, I have set 3 php cookies like this:



In test2.php in the same domain and same directory, I tried to access the
cookies :



It is showing only the last set cookie.

 array(1) { ["test3"]=> string(8) "content3" }

all the three cookies are showing in developer tools.

Javascript is able to read all the three cookies :



If I use nginx http2, php is able to access all the three cookies.

But, If I use nginx http3, php is able to access only the last cookie.

Please help
Thanks
Krishna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From gmm at csdoc.com  Mon Dec 20 18:05:54 2021
From: gmm at csdoc.com (Gena Makhomed)
Date: Mon, 20 Dec 2021 20:05:54 +0200
Subject: [PATCH] Contrib: vim syntax,
 update core and 3rd party module directives.
Message-ID: <48c37eaa-a317-eb92-d725-f7e8071664bc@csdoc.com>

# HG changeset patch
# User Gena Makhomed 
# Date 1640023368 -7200
#      Mon Dec 20 20:02:48 2021 +0200
# Node ID 4719937b5dcbc5345e1a4cbf4bf84d2da4e316f8
# Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
Contrib: vim syntax, update core and 3rd party module directives.

diff -r a7a77549265e -r 4719937b5dcb contrib/vim/syntax/nginx.vim
--- a/contrib/vim/syntax/nginx.vim	Thu Nov 25 22:02:10 2021 +0300
+++ b/contrib/vim/syntax/nginx.vim	Mon Dec 20 20:02:48 2021 +0200
@@ -152,6 +152,7 @@
  syn keyword ngxDirective contained auth_jwt_key_file
  syn keyword ngxDirective contained auth_jwt_key_request
  syn keyword ngxDirective contained auth_jwt_leeway
+syn keyword ngxDirective contained auth_jwt_require
  syn keyword ngxDirective contained auth_jwt_type
  syn keyword ngxDirective contained auth_request
  syn keyword ngxDirective contained auth_request_set
@@ -335,6 +336,10 @@
  syn keyword ngxDirective contained js_access
  syn keyword ngxDirective contained js_body_filter
  syn keyword ngxDirective contained js_content
+syn keyword ngxDirective contained js_fetch_ciphers
+syn keyword ngxDirective contained js_fetch_protocols
+syn keyword ngxDirective contained js_fetch_trusted_certificate
+syn keyword ngxDirective contained js_fetch_verify_depth
  syn keyword ngxDirective contained js_filter
  syn keyword ngxDirective contained js_header_filter
  syn keyword ngxDirective contained js_import
@@ -402,6 +407,7 @@
  syn keyword ngxDirective contained mp4_limit_rate
  syn keyword ngxDirective contained mp4_limit_rate_after
  syn keyword ngxDirective contained mp4_max_buffer_size
+syn keyword ngxDirective contained mp4_start_key_frame
  syn keyword ngxDirective contained msie_padding
  syn keyword ngxDirective contained msie_refresh
  syn keyword ngxDirective contained multi_accept
@@ -458,6 +464,7 @@
  syn keyword ngxDirective contained proxy_cookie_path
  syn keyword ngxDirective contained proxy_download_rate
  syn keyword ngxDirective contained proxy_force_ranges
+syn keyword ngxDirective contained proxy_half_close
  syn keyword ngxDirective contained proxy_headers_hash_bucket_size
  syn keyword ngxDirective contained proxy_headers_hash_max_size
  syn keyword ngxDirective contained proxy_hide_header
@@ -597,6 +604,7 @@
  syn keyword ngxDirective contained ssi_silent_errors
  syn keyword ngxDirective contained ssi_types
  syn keyword ngxDirective contained ssi_value_length
+syn keyword ngxDirective contained ssl_alpn
  syn keyword ngxDirective contained ssl_buffer_size
  syn keyword ngxDirective contained ssl_certificate
  syn keyword ngxDirective contained ssl_certificate_key
@@ -788,13 +796,18 @@
  syn keyword ngxDirectiveThirdParty contained auth_gss
  syn keyword ngxDirectiveThirdParty contained auth_gss_allow_basic_fallback
  syn keyword ngxDirectiveThirdParty contained auth_gss_authorized_principal
+syn keyword ngxDirectiveThirdParty contained 
auth_gss_authorized_principal_regex
+syn keyword ngxDirectiveThirdParty contained 
auth_gss_constrained_delegation
+syn keyword ngxDirectiveThirdParty contained auth_gss_delegate_credentials
  syn keyword ngxDirectiveThirdParty contained auth_gss_force_realm
  syn keyword ngxDirectiveThirdParty contained auth_gss_format_full
  syn keyword ngxDirectiveThirdParty contained auth_gss_keytab
  syn keyword ngxDirectiveThirdParty contained auth_gss_map_to_local
  syn keyword ngxDirectiveThirdParty contained auth_gss_realm
+syn keyword ngxDirectiveThirdParty contained auth_gss_service_ccache
  syn keyword ngxDirectiveThirdParty contained auth_gss_service_name

+
  " LDAP Authentication
  " https://github.com/kvspb/nginx-auth-ldap
  syn keyword ngxDirectiveThirdParty contained auth_ldap
@@ -969,7 +982,6 @@
  syn keyword ngxDirectiveThirdParty contained fancyindex_hide_symlinks
  syn keyword ngxDirectiveThirdParty contained fancyindex_ignore
  syn keyword ngxDirectiveThirdParty contained fancyindex_localtime
-syn keyword ngxDirectiveThirdParty contained fancyindex_name_length
  syn keyword ngxDirectiveThirdParty contained fancyindex_show_dotfiles
  syn keyword ngxDirectiveThirdParty contained fancyindex_show_path
  syn keyword ngxDirectiveThirdParty contained fancyindex_time_format
@@ -1059,7 +1071,9 @@
  syn keyword ngxDirectiveThirdParty contained nchan_pubsub
  syn keyword ngxDirectiveThirdParty contained nchan_pubsub_channel_id
  syn keyword ngxDirectiveThirdParty contained nchan_pubsub_location
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_cluster_check_interval
  syn keyword ngxDirectiveThirdParty contained nchan_redis_connect_timeout
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_discovered_ip_range_blacklist
  syn keyword ngxDirectiveThirdParty contained 
nchan_redis_fakesub_timer_interval
  syn keyword ngxDirectiveThirdParty contained 
nchan_redis_idle_channel_cache_timeout
  syn keyword ngxDirectiveThirdParty contained nchan_redis_namespace
@@ -1067,12 +1081,29 @@
  syn keyword ngxDirectiveThirdParty contained nchan_redis_optimize_target
  syn keyword ngxDirectiveThirdParty contained nchan_redis_pass
  syn keyword ngxDirectiveThirdParty contained nchan_redis_pass_inheritable
+syn keyword ngxDirectiveThirdParty contained nchan_redis_password
  syn keyword ngxDirectiveThirdParty contained nchan_redis_ping_interval
  syn keyword ngxDirectiveThirdParty contained 
nchan_redis_publish_msgpacked_max_size
  syn keyword ngxDirectiveThirdParty contained nchan_redis_server
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl_ciphers
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_ssl_client_certificate
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_ssl_client_certificate_key
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl_server_name
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_ssl_trusted_certificate
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_ssl_trusted_certificate_path
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_ssl_verify_certificate
  syn keyword ngxDirectiveThirdParty contained nchan_redis_storage_mode
  syn keyword ngxDirectiveThirdParty contained nchan_redis_subscribe_weights
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls_ciphers
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_tls_client_certificate
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls_server_name
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_tls_trusted_certificate
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_tls_trusted_certificate_path
+syn keyword ngxDirectiveThirdParty contained 
nchan_redis_tls_verify_certificate
  syn keyword ngxDirectiveThirdParty contained nchan_redis_url
+syn keyword ngxDirectiveThirdParty contained nchan_redis_username
  syn keyword ngxDirectiveThirdParty contained 
nchan_redis_wait_after_connecting
  syn keyword ngxDirectiveThirdParty contained nchan_shared_memory_size
  syn keyword ngxDirectiveThirdParty contained nchan_storage_engine
@@ -1385,6 +1416,7 @@
  syn keyword ngxDirectiveThirdParty contained lua_socket_send_lowat
  syn keyword ngxDirectiveThirdParty contained lua_socket_send_timeout
  syn keyword ngxDirectiveThirdParty contained lua_ssl_ciphers
+syn keyword ngxDirectiveThirdParty contained lua_ssl_conf_command
  syn keyword ngxDirectiveThirdParty contained lua_ssl_crl
  syn keyword ngxDirectiveThirdParty contained lua_ssl_protocols
  syn keyword ngxDirectiveThirdParty contained lua_ssl_trusted_certificate
@@ -1392,6 +1424,7 @@
  syn keyword ngxDirectiveThirdParty contained lua_thread_cache_max_entries
  syn keyword ngxDirectiveThirdParty contained 
lua_transform_underscores_in_response_headers
  syn keyword ngxDirectiveThirdParty contained lua_use_default_type
+syn keyword ngxDirectiveThirdParty contained lua_worker_thread_vm_pool_size
  syn keyword ngxDirectiveThirdParty contained rewrite_by_lua
  syn keyword ngxDirectiveThirdParty contained rewrite_by_lua_block
  syn keyword ngxDirectiveThirdParty contained rewrite_by_lua_file
@@ -1401,6 +1434,8 @@
  syn keyword ngxDirectiveThirdParty contained set_by_lua_file
  syn keyword ngxDirectiveThirdParty contained ssl_certificate_by_lua_block
  syn keyword ngxDirectiveThirdParty contained ssl_certificate_by_lua_file
+syn keyword ngxDirectiveThirdParty contained ssl_client_hello_by_lua_block
+syn keyword ngxDirectiveThirdParty contained ssl_client_hello_by_lua_file
  syn keyword ngxDirectiveThirdParty contained 
ssl_session_fetch_by_lua_block
  syn keyword ngxDirectiveThirdParty contained ssl_session_fetch_by_lua_file
  syn keyword ngxDirectiveThirdParty contained 
ssl_session_store_by_lua_block
@@ -1719,15 +1754,18 @@
  syn keyword ngxDirectiveThirdParty contained set_base32_padding
  syn keyword ngxDirectiveThirdParty contained set_decode_base32
  syn keyword ngxDirectiveThirdParty contained set_decode_base64
+syn keyword ngxDirectiveThirdParty contained set_decode_base64url
  syn keyword ngxDirectiveThirdParty contained set_decode_hex
  syn keyword ngxDirectiveThirdParty contained set_encode_base32
  syn keyword ngxDirectiveThirdParty contained set_encode_base64
+syn keyword ngxDirectiveThirdParty contained set_encode_base64url
  syn keyword ngxDirectiveThirdParty contained set_encode_hex
  syn keyword ngxDirectiveThirdParty contained set_escape_uri
  syn keyword ngxDirectiveThirdParty contained set_formatted_gmt_time
  syn keyword ngxDirectiveThirdParty contained set_formatted_local_time
  syn keyword ngxDirectiveThirdParty contained set_hashed_upstream
  syn keyword ngxDirectiveThirdParty contained set_hmac_sha1
+syn keyword ngxDirectiveThirdParty contained set_hmac_sha256
  syn keyword ngxDirectiveThirdParty contained set_if_empty
  syn keyword ngxDirectiveThirdParty contained set_local_today
  syn keyword ngxDirectiveThirdParty contained set_misc_base32_padding
@@ -1849,6 +1887,7 @@
  syn keyword ngxDirectiveThirdParty contained vod_open_file_thread_pool
  syn keyword ngxDirectiveThirdParty contained vod_output_buffer_pool
  syn keyword ngxDirectiveThirdParty contained vod_parse_hdlr_name
+syn keyword ngxDirectiveThirdParty contained vod_parse_udta_name
  syn keyword ngxDirectiveThirdParty contained vod_path_response_postfix
  syn keyword ngxDirectiveThirdParty contained vod_path_response_prefix
  syn keyword ngxDirectiveThirdParty contained vod_performance_counters


From xeioex at nginx.com  Tue Dec 21 17:45:59 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 21 Dec 2021 17:45:59 +0000
Subject: [njs] Added constructor flag for methods in external prototypes.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/3c28224a8f79
branches:  
changeset: 1768:3c28224a8f79
user:      Dmitry Volyntsev 
date:      Tue Dec 21 15:48:45 2021 +0000
description:
Added constructor flag for methods in external prototypes.

diffstat:

 src/njs.h        |  1 +
 src/njs_extern.c |  3 ++-
 2 files changed, 3 insertions(+), 1 deletions(-)

diffs (25 lines):

diff -r 659995ad4f97 -r 3c28224a8f79 src/njs.h
--- a/src/njs.h	Tue Dec 14 20:14:23 2021 +0000
+++ b/src/njs.h	Tue Dec 21 15:48:45 2021 +0000
@@ -137,6 +137,7 @@ struct njs_external_s {
         struct {
             njs_function_native_t   native;
             uint8_t                 magic8;
+            uint8_t                 ctor;
         } method;
 
         struct {
diff -r 659995ad4f97 -r 3c28224a8f79 src/njs_extern.c
--- a/src/njs_extern.c	Tue Dec 14 20:14:23 2021 +0000
+++ b/src/njs_extern.c	Tue Dec 21 15:48:45 2021 +0000
@@ -79,8 +79,9 @@ njs_external_add(njs_vm_t *vm, njs_arr_t
             function->object.extensible = 1;
             function->args_offset = 1;
             function->native = 1;
+            function->u.native = external->u.method.native;
             function->magic8 = external->u.method.magic8;
-            function->u.native = external->u.method.native;
+            function->ctor = external->u.method.ctor;
 
             njs_set_function(&prop->value, function);
 

From xeioex at nginx.com  Tue Dec 21 17:46:00 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 21 Dec 2021 17:46:00 +0000
Subject: [njs] Tests: moving backtraces test into a separate test suite.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/ce5cd29751a4
branches:  
changeset: 1769:ce5cd29751a4
user:      Dmitry Volyntsev 
date:      Tue Dec 21 15:49:05 2021 +0000
description:
Tests: moving backtraces test into a separate test suite.

diffstat:

 src/test/njs_unit_test.c |  115 +++++++++++++++++++++++++---------------------
 1 files changed, 63 insertions(+), 52 deletions(-)

diffs (267 lines):

diff -r 3c28224a8f79 -r ce5cd29751a4 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Tue Dec 21 15:48:45 2021 +0000
+++ b/src/test/njs_unit_test.c	Tue Dec 21 15:49:05 2021 +0000
@@ -21454,125 +21454,135 @@ static njs_unit_test_t  njs_shell_test[]
               "function(){}()" ENTER),
       njs_str("SyntaxError: Unexpected token \"(\" in 1") },
 
-    /* Backtraces */
-
-    { njs_str("function ff(o) {return o.a.a}" ENTER
-              "function f(o) {return ff(o)}" ENTER
-              "f({})" ENTER),
+    /* Exception in njs_vm_retval_string() */
+
+    { njs_str("var o = { toString: function() { return [1] } }" ENTER
+              "o" ENTER),
+      njs_str("TypeError: Cannot convert object to primitive value") },
+};
+
+
+static njs_unit_test_t  njs_backtraces_test[] =
+{
+    { njs_str("function ff(o) {return o.a.a};"
+              "function f(o) {return ff(o)};"
+              "f({})"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at ff (:1)\n"
               "    at f (:1)\n"
               "    at main (:1)\n") },
 
-    { njs_str("function ff(o) {return o.a.a}" ENTER
+    { njs_str("function ff(o) {return o.a.a};"
               "function f(o) {try {return ff(o)} "
-              "               finally {return o.a.a}}" ENTER
-              "f({})" ENTER),
+              "               finally {return o.a.a}};"
+              "f({})"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at f (:1)\n"
               "    at main (:1)\n") },
 
-    { njs_str("function f(ff, o) {return ff(o)}" ENTER
-              "f(function (o) {return o.a.a}, {})" ENTER),
+    { njs_str("function f(ff, o) {return ff(o)};"
+              "f(function (o) {return o.a.a}, {})"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at anonymous (:1)\n"
               "    at f (:1)\n"
               "    at main (:1)\n") },
 
     { njs_str("'str'.replace(/t/g,"
-              "              function(m) {return m.a.a})" ENTER),
+              "              function(m) {return m.a.a})"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at anonymous (:1)\n"
               "    at RegExp.prototype[Symbol.replace] (native)\n"
               "    at String.prototype.replace (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("function f(o) {return Object.keys(o)}" ENTER
-              "f()" ENTER),
+    { njs_str("function f(o) {return Object.keys(o)};"
+              "f()"),
       njs_str("TypeError: cannot convert undefined argument to object\n"
               "    at Object.keys (native)\n"
               "    at f (:1)\n"
               "    at main (:1)\n") },
 
-    { njs_str("[].concat({}.a.a)" ENTER),
+    { njs_str("[].concat({}.a.a)"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at Array.prototype.concat (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("''.repeat(-1)" ENTER),
+    { njs_str("''.repeat(-1)"),
       njs_str("RangeError\n"
               "    at String.prototype.repeat (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("Math.log({}.a.a)" ENTER),
+    { njs_str("Math.log({}.a.a)"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at Math.log (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("var bound = Math.max.bind(null, {toString(){return {}}}); bound(1)" ENTER),
+    { njs_str("var bound = Math.max.bind(null, {toString(){return {}}}); bound(1)"),
       njs_str("TypeError: Cannot convert object to primitive value\n"
               "    at Math.max (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("Object.prototype()" ENTER),
+    { njs_str("Object.prototype()"),
       njs_str("TypeError: (intermediate value)[\"prototype\"] is not a function\n"
                "    at main (:1)\n") },
 
-    { njs_str("eval()" ENTER),
+    { njs_str("eval()"),
       njs_str("InternalError: Not implemented\n"
               "    at eval (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("$shared.method({}.a.a)" ENTER),
+    { njs_str("$shared.method({}.a.a)"),
     /* FIXME: at $shared.method (native) */
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at $r.method (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("new Function(\n\n@)" ENTER),
+    { njs_str("new Function(\n\n@)"),
       njs_str("SyntaxError: Unexpected token \"@\" in 3") },
 
-    { njs_str("require()" ENTER),
+    { njs_str("require()"),
       njs_str("TypeError: missing path\n"
               "    at require (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("setTimeout()" ENTER),
+    { njs_str("setTimeout()"),
       njs_str("TypeError: too few arguments\n"
               "    at setTimeout (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("require('crypto').createHash('sha')" ENTER),
+    { njs_str("require('crypto').createHash('sha')"),
       njs_str("TypeError: not supported algorithm: \"sha\"\n"
               "    at crypto.createHash (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("var h = require('crypto').createHash('sha1')" ENTER
-              "h.update([])" ENTER),
+    { njs_str("var h = require('crypto').createHash('sha1');"
+              "h.update([])"),
       njs_str("TypeError: data argument \"array\" is not a string or Buffer-like object\n"
               "    at Hash.prototype.update (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("require('crypto').createHmac('sha1', [])" ENTER),
+    { njs_str("require('crypto').createHmac('sha1', [])"),
       njs_str("TypeError: key argument \"array\" is not a string or Buffer-like object\n"
               "    at crypto.createHmac (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("var h = require('crypto').createHmac('sha1', 'secret')" ENTER
-              "h.update([])" ENTER),
+    { njs_str("var h = require('crypto').createHmac('sha1', 'secret');"
+              "h.update([])"),
       njs_str("TypeError: data argument \"array\" is not a string or Buffer-like object\n"
               "    at Hmac.prototype.update (native)\n"
               "    at main (:1)\n") },
 
     { njs_str("function f(o) {function f_in(o) {return o.a.a};"
-              "               return f_in(o)}; f({})" ENTER),
+              "               return f_in(o)};"
+              "f({})"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at f_in (:1)\n"
               "    at f (:1)\n"
               "    at main (:1)\n") },
 
     { njs_str("function f(o) {var ff = function (o) {return o.a.a};"
-              "               return ff(o)}; f({})" ENTER),
+              "               return ff(o)};"
+              "f({})"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at anonymous (:1)\n"
               "    at f (:1)\n"
@@ -21595,65 +21605,59 @@ static njs_unit_test_t  njs_shell_test[]
               " 'realpath',"
               " 'realpathSync',"
               "]"
-              ".every(v=>{ try {fs[v]();} catch (e) { return e.stack.search(`fs.${v} `) >= 0}})" ENTER),
-      njs_str("true") },
-
-    { njs_str("parseInt({ toString: function() { return [1] } })" ENTER),
+              ".every(v=>{ try {fs[v]();} catch (e) { return e.stack.search(`fs.${v} `) >= 0}})"),
+      njs_str("true") },
+
+    { njs_str("parseInt({ toString: function() { return [1] } })"),
       njs_str("TypeError: Cannot convert object to primitive value\n"
               "    at parseInt (native)\n"
               "    at main (:1)\n") },
 
-    { njs_str("function f(n) { if (n == 0) { throw 'a'; } return f(n-1); }; f(2)" ENTER),
+    { njs_str("function f(n) { if (n == 0) { throw 'a'; } return f(n-1); }; f(2)"),
       njs_str("a") },
 
-    /* Exception in njs_vm_retval_string() */
-
-    { njs_str("var o = { toString: function() { return [1] } }" ENTER
-              "o" ENTER),
-      njs_str("TypeError: Cannot convert object to primitive value") },
-
     /* line numbers */
 
-    { njs_str("/**/(function(){throw Error();})()" ENTER),
+    { njs_str("/**/(function(){throw Error();})()"),
       njs_str("Error\n"
               "    at anonymous (:1)\n"
               "    at main (:1)\n") },
 
-    { njs_str("/***/(function(){throw Error();})()" ENTER),
+    { njs_str("/***/(function(){throw Error();})()"),
       njs_str("Error\n"
               "    at anonymous (:1)\n"
               "    at main (:1)\n") },
 
-    { njs_str("/*\n**/(function(){throw Error();})()" ENTER),
+    { njs_str("/*\n**/(function(){throw Error();})()"),
       njs_str("Error\n"
               "    at anonymous (:2)\n"
               "    at main (:2)\n") },
 
-    { njs_str("({})\n.a\n.a" ENTER),
+    { njs_str("({})\n.a\n.a"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at main (:3)\n") },
 
-    { njs_str("1\n+a" ENTER),
+    { njs_str("1\n+a"),
       njs_str("ReferenceError: \"a\" is not defined\n"
               "    at main (:2)\n") },
 
-    { njs_str("\n`\n${Object}\n${a}`" ENTER),
+    { njs_str("\n`\n${Object}\n${a}`"),
       njs_str("ReferenceError: \"a\" is not defined\n"
               "    at main (:4)\n") },
 
-    { njs_str("function log(v) {}\nlog({}\n.a\n.a)" ENTER),
+    { njs_str("function log(v) {}\nlog({}\n.a\n.a)"),
       njs_str("TypeError: cannot get property \"a\" of undefined\n"
               "    at main (:4)\n") },
 
-    { njs_str("\nfor (var i = 0;\n i < a;\n i++) { }\n" ENTER),
+    { njs_str("\nfor (var i = 0;\n i < a;\n i++) { }\n"),
       njs_str("ReferenceError: \"a\" is not defined\n"
               "    at main (:3)\n") },
 
-    { njs_str("\nfor (var i = 0;\n i < 5;\n a) {\n }" ENTER),
+    { njs_str("\nfor (var i = 0;\n i < 5;\n a) {\n }"),
       njs_str("ReferenceError: \"a\" is not defined\n"
               "    at main (:4)\n") },
 
-    { njs_str("Math\n.min(1,\na)" ENTER),
+    { njs_str("Math\n.min(1,\na)"),
       njs_str("ReferenceError: \"a\" is not defined\n"
               "    at Math.min (native)\n"
               "    at main (:3)\n") },
@@ -23392,8 +23396,15 @@ static njs_test_suite_t  njs_suites[] =
       { .externals = 1, .repeat = 1, .unsafe = 1 },
       njs_shell_test,
       njs_nitems(njs_shell_test),
+
       njs_interactive_test },
 
+    { njs_str("backtraces"),
+      { .backtrace = 1, .externals = 1, .repeat = 1, .unsafe = 1 },
+      njs_backtraces_test,
+      njs_nitems(njs_backtraces_test),
+      njs_unit_test },
+
     { njs_str("timezone"),
       { .repeat = 1, .unsafe = 1 },
       njs_tz_test,

From xeioex at nginx.com  Tue Dec 21 17:46:02 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 21 Dec 2021 17:46:02 +0000
Subject: [njs] Tests: moving module tests into separate test suites.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/1efb00924df8
branches:  
changeset: 1770:1efb00924df8
user:      Dmitry Volyntsev 
date:      Tue Dec 21 15:49:13 2021 +0000
description:
Tests: moving module tests into separate test suites.

diffstat:

 src/test/njs_unit_test.c |  2609 +++++++++++++++++++++++----------------------
 1 files changed, 1319 insertions(+), 1290 deletions(-)

diffs (truncated from 2647 to 1000 lines):

diff -r ce5cd29751a4 -r 1efb00924df8 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Tue Dec 21 15:49:05 2021 +0000
+++ b/src/test/njs_unit_test.c	Tue Dec 21 15:49:13 2021 +0000
@@ -17710,546 +17710,6 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("require.hasOwnProperty('length')"),
       njs_str("true") },
 
-    { njs_str("var fs = require('fs'); typeof fs"),
-      njs_str("object") },
-
-    { njs_str("var fs = require('fs'); Object.isExtensible(fs)"),
-      njs_str("true") },
-
-    { njs_str("require('fs') === require('fs')"),
-      njs_str("true") },
-
-    { njs_str("require('fs').a = 1; require('fs').a"),
-      njs_str("1") },
-
-    /* require('fs').readFile() */
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFile()"),
-      njs_str("TypeError: \"path\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "var path = Buffer.from('/broken'); path[3] = 0;"
-              "fs.readFile(path)"),
-      njs_str("TypeError: \"path\" must be a Buffer without null bytes") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFile('/njs_unknown_path')"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFile('/njs_unknown_path', 'utf8')"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFile('/njs_unknown_path', {flag:'xx'})"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFile('/njs_unknown_path', {flag:'xx'}, 1)"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFile('/njs_unknown_path', {flag:'xx'}, function () {})"),
-      njs_str("TypeError: Unknown file open flags: \"xx\"") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFile('/njs_unknown_path', {encoding:'ascii'}, function () {})"),
-      njs_str("TypeError: \"ascii\" encoding is not supported") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFile('/njs_unknown_path', 'ascii', function () {})"),
-      njs_str("TypeError: \"ascii\" encoding is not supported") },
-
-    /* require('fs').readFileSync() */
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFileSync()"),
-      njs_str("TypeError: \"path\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFileSync({})"),
-      njs_str("TypeError: \"path\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFileSync('/njs_unknown_path', {flag:'xx'})"),
-      njs_str("TypeError: Unknown file open flags: \"xx\"") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFileSync(Buffer.from('/njs_unknown_path'), {encoding:'ascii'})"),
-      njs_str("TypeError: \"ascii\" encoding is not supported") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFileSync('/njs_unknown_path', 'ascii')"),
-      njs_str("TypeError: \"ascii\" encoding is not supported") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.readFileSync('/njs_unknown_path', true)"),
-      njs_str("TypeError: Unknown options type: \"boolean\" (a string or object required)") },
-
-
-    /* require('fs').writeFile() */
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile()"),
-      njs_str("TypeError: \"path\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile({}, '', function () {})"),
-      njs_str("TypeError: \"path\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile('/njs_unknown_path')"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile('/njs_unknown_path', '')"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile('/njs_unknown_path', '', undefined)"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile('/njs_unknown_path', '', 'utf8')"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile('/njs_unknown_path', '', {flag:'xx'}, function () {})"),
-      njs_str("TypeError: Unknown file open flags: \"xx\"") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile('/njs_unknown_path', '', {encoding:'ascii'}, function () {})"),
-      njs_str("TypeError: \"ascii\" encoding is not supported") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile('/njs_unknown_path', '', 'ascii', function () {})"),
-      njs_str("TypeError: \"ascii\" encoding is not supported") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFile('/njs_unknown_path', '', true, function () {})"),
-      njs_str("TypeError: Unknown options type: \"boolean\" (a string or object required)") },
-
-    /* require('fs').writeFileSync() */
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFileSync()"),
-      njs_str("TypeError: \"path\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFileSync({}, '')"),
-      njs_str("TypeError: \"path\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFileSync('/njs_unknown_path', '', {flag:'xx'})"),
-      njs_str("TypeError: Unknown file open flags: \"xx\"") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFileSync('/njs_unknown_path', '', {encoding:'ascii'})"),
-      njs_str("TypeError: \"ascii\" encoding is not supported") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFileSync('/njs_unknown_path', '', 'ascii')"),
-      njs_str("TypeError: \"ascii\" encoding is not supported") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.writeFileSync('/njs_unknown_path', '', true)"),
-      njs_str("TypeError: Unknown options type: \"boolean\" (a string or object required)") },
-
-    /* require('fs').renameSync() */
-
-    { njs_str("var fs = require('fs');"
-              "fs.renameSync()"),
-      njs_str("TypeError: \"oldPath\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.renameSync('/njs_unknown_path')"),
-      njs_str("TypeError: \"newPath\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "[undefined, null, false, NaN, Symbol(), {}, Object('/njs_unknown_path')]"
-              ".map((x) => { try { fs.renameSync(x, '/njs_unknown_path'); } "
-              "              catch (e) { return (e instanceof TypeError); } })"
-              ".every((x) => x === true)"),
-      njs_str("true")},
-
-    { njs_str("var fs = require('fs');"
-              "[undefined, null, false, NaN, Symbol(), {}, Object('/njs_unknown_path')]"
-              ".map((x) => { try { fs.renameSync('/njs_unknown_path', x); } "
-              "              catch (e) { return (e instanceof TypeError); } })"
-              ".every((x) => x === true)"),
-      njs_str("true")},
-
-    /* require('fs').access() */
-
-    { njs_str("var fs = require('fs');"
-              "fs.access()"),
-      njs_str("TypeError: \"path\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.access('/njs_unknown_path')"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.access('/njs_unknown_path', fs.constants.F_OK)"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.access('/njs_unknown_path', 'fail', function () {})"),
-      njs_str("TypeError: \"mode\" must be a number") },
-
-    /* require('fs').accessSync() */
-
-    { njs_str("var fs = require('fs');"
-              "fs.accessSync()"),
-      njs_str("TypeError: \"path\" must be a string or Buffer") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.accessSync('/njs_unknown_path', 'fail')"),
-      njs_str("TypeError: \"mode\" must be a number") },
-
-    { njs_str("var "
-              "fs = require('fs'),"
-              "func = ["
-                "'access',"
-                "'accessSync',"
-                "'readFile',"
-                "'readFileSync',"
-                "'writeFile',"
-                "'writeFileSync',"
-                "'appendFile',"
-                "'appendFileSync',"
-                "'rename',"
-                "'renameSync',"
-                "'symlink',"
-                "'symlinkSync',"
-                "'unlink',"
-                "'unlinkSync',"
-                "'realpath',"
-                "'realpathSync',"
-                "'mkdir',"
-                "'mkdirSync',"
-                "'rmdir',"
-                "'rmdirSync',"
-                "'readdir',"
-                "'readdirSync',"
-              "],"
-              "test = (fname) =>"
-                "[undefined, null, false, NaN, Symbol(), {}, Object('/njs_unknown_path')]"
-                ".map((x) => { try { fs[fname](x); } "
-                "              catch (e) { return (e instanceof TypeError); } })"
-                ".every((x) => x === true);"
-              "func.map(test).every((x) => x)"),
-      njs_str("true")},
-
-    /* require('fs').promises */
-
-    { njs_str("var fs = require('fs');"
-              "typeof fs.promises"),
-      njs_str("object") },
-
-    { njs_str("var "
-              "fs = require('fs').promises,"
-              "func = ["
-                "'access',"
-                "'readFile',"
-                "'writeFile',"
-                "'appendFile',"
-                "'rename',"
-                "'symlink',"
-                "'unlink',"
-                "'realpath',"
-                "'mkdir',"
-                "'rmdir',"
-                "'readdir',"
-              "];"
-              "func.every((x) => typeof fs[x] == 'function')"),
-      njs_str("true")},
-
-    /* require('fs').constants */
-
-    { njs_str("var fs = require('fs');"
-              "typeof fs.constants"),
-      njs_str("object") },
-
-    { njs_str("var "
-              "fsc = require('fs').constants,"
-              "items = ["
-                "'F_OK',"
-                "'R_OK',"
-                "'W_OK',"
-                "'X_OK',"
-              "];"
-              "items.every((x) => typeof fsc[x] == 'number')"),
-      njs_str("true")},
-
-    /* require('fs').Dirent */
-
-    { njs_str("var fs = require('fs');"
-              "typeof fs.Dirent"),
-      njs_str("function") },
-
-    { njs_str("var fs = require('fs');"
-              "fs.Dirent('file', 123)"),
-      njs_str("TypeError: the Dirent constructor must be called with new") },
-
-    { njs_str("var fs = require('fs');"
-              "var e = new fs.Dirent('file', 123); [e.name, e.type]"),
-      njs_str("file,123") },
-
-    { njs_str("var "
-              "fs = require('fs'),"
-              "e = new fs.Dirent('file', 0),"
-              "func = ["
-                "'isDirectory',"
-                "'isFile',"
-                "'isBlockDevice',"
-                "'isCharacterDevice',"
-                "'isSymbolicLink',"
-                "'isFIFO',"
-                "'isSocket',"
-              "];"
-              "func.every((x) => typeof e[x] == 'function')"),
-      njs_str("true")},
-
-    /* require('crypto').createHash() */
-
-    { njs_str("var h = require('crypto').createHash('sha1');"
-              "[Object.prototype.toString.call(h), njs.dump(h),h]"),
-      njs_str("[object Hash],Hash {},[object Hash]") },
-
-    { njs_str("var h = require('crypto').createHash('sha1');"
-              "var Hash = h.constructor; "
-              "Hash('sha1').update('AB').digest('hex')"),
-      njs_str("06d945942aa26a61be18c3e22bf19bbca8dd2b5d") },
-
-    { njs_str("var h = require('crypto').createHash('sha1');"
-              "h.constructor.name"),
-      njs_str("Hash") },
-
-    { njs_str("var hash = require('crypto').createHash.bind(undefined, 'md5');"
-              "['hex', 'base64', 'base64url'].map(e => {"
-              "   var h = hash().update('AB').digest().toString(e);"
-              "   var h2 = hash().update(Buffer.from('XABX').subarray(1,3)).digest(e);"
-              "   var h3 = hash().update('A').update('B').digest(e);"
-              "   if (h !== h2) {throw new Error(`digest().toString($e):$h != digest($e):$h2`)};"
-              "   if (h !== h3) {throw new Error(`digest().toString($e):$h != update('A').update('B').digest($e):$h3`)};"
-              "   return h;"
-              "})"),
-      njs_str("b86fc6b051f63d73de262d4c34e3a0a9,"
-              "uG/GsFH2PXPeJi1MNOOgqQ==,"
-              "uG_GsFH2PXPeJi1MNOOgqQ") },
-
-    { njs_str("var hash = require('crypto').createHash.bind(undefined, 'sha1');"
-              "['hex', 'base64', 'base64url'].map(e => {"
-              "   var h = hash().update('4142', 'hex').digest().toString(e);"
-              "   var h2 = hash().update(Buffer.from('XABX').subarray(1,3)).digest(e);"
-              "   var h3 = hash().update('A').update('B').digest(e);"
-              "   if (h !== h2) {throw new Error(`digest().toString($e):$h != digest($e):$h2`)};"
-              "   if (h !== h3) {throw new Error(`digest().toString($e):$h != update('A').update('B').digest($e):$h3`)};"
-              "   return h;"
-              "})"),
-      njs_str("06d945942aa26a61be18c3e22bf19bbca8dd2b5d,"
-              "BtlFlCqiamG+GMPiK/GbvKjdK10=,"
-              "BtlFlCqiamG-GMPiK_GbvKjdK10") },
-
-    { njs_str("var hash = require('crypto').createHash.bind(undefined, 'sha1');"
-              "['hex', 'base64', 'base64url'].every(e => {"
-              "   var h = hash().digest(e);"
-              "   var h2 = hash().update('').digest(e);"
-              "   if (h !== h2) {throw new Error(`digest($e):$h != update('').digest($e):$h2`)};"
-              "   return true;"
-              "})"),
-      njs_str("true") },
-
-    { njs_str("var hash = require('crypto').createHash.bind(undefined, 'sha1');"
-              "["
-              " ['AB'],"
-              " ['4142', 'hex'],"
-              " ['QUI=', 'base64'],"
-              " ['QUI', 'base64url']"
-              "].every(args => {"
-              "        return hash().update(args[0], args[1]).digest('hex') === '06d945942aa26a61be18c3e22bf19bbca8dd2b5d';"
-              "})"),
-      njs_str("true") },
-
-    { njs_str("var hash = require('crypto').createHash.bind(undefined, 'sha256');"
-              "['hex', 'base64', 'base64url'].map(e => {"
-              "   var h = hash().update('AB').digest().toString(e);"
-              "   var h2 = hash().update(Buffer.from('XABX').subarray(1,3)).digest(e);"
-              "   var h3 = hash().update('A').update('B').digest(e);"
-              "   if (h !== h2) {throw new Error(`digest().toString($e):$h != digest($e):$h2`)};"
-              "   if (h !== h3) {throw new Error(`digest().toString($e):$h != update('A').update('B').digest($e):$h3`)};"
-              "   return h;"
-              "})"),
-      njs_str("38164fbd17603d73f696b8b4d72664d735bb6a7c88577687fd2ae33fd6964153,"
-              "OBZPvRdgPXP2lri01yZk1zW7anyIV3aH/SrjP9aWQVM=,"
-              "OBZPvRdgPXP2lri01yZk1zW7anyIV3aH_SrjP9aWQVM") },
-
-    { njs_str("var hash = require('crypto').createHash;"
-              "njs.dump(['', 'abc'.repeat(100)].map(v => {"
-              "    return ['md5', 'sha1', 'sha256'].map(h => {"
-              "        return hash(h).update(v).digest('hex');"
-              "     })"
-              "}))"),
-      njs_str("[['d41d8cd98f00b204e9800998ecf8427e',"
-              "'da39a3ee5e6b4b0d3255bfef95601890afd80709',"
-              "'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'],"
-              "['f571117acbd8153c8dc3c81b8817773a',"
-              "'c95466320eaae6d19ee314ae4f135b12d45ced9a',"
-              "'d9f5aeb06abebb3be3f38adec9a2e3b94228d52193be923eb4e24c9b56ee0930']]") },
-
-    { njs_str("var h = require('crypto').createHash()"),
-      njs_str("TypeError: algorithm must be a string") },
-
-    { njs_str("var h = require('crypto').createHash([])"),
-      njs_str("TypeError: algorithm must be a string") },
-
-    { njs_str("var h = require('crypto').createHash('sha512')"),
-      njs_str("TypeError: not supported algorithm: \"sha512\"") },
-
-    { njs_str("var h = require('crypto').createHash('sha1');"
-              "h.update()"),
-      njs_str("TypeError: data argument \"undefined\" is not a string or Buffer-like object") },
-
-    { njs_str("var h = require('crypto').createHash('sha1');"
-              "h.update({})"),
-      njs_str("TypeError: data argument \"object\" is not a string or Buffer-like object") },
-
-    { njs_str("var h = require('crypto').createHash('sha1');"
-              "h.update('A').digest('latin1')"),
-      njs_str("TypeError: Unknown digest encoding: \"latin1\"") },
-
-    { njs_str("require('crypto').createHash('sha1').digest() instanceof Buffer"),
-      njs_str("true") },
-
-    { njs_str("var h = require('crypto').createHash('sha1');"
-              "h.update('A').digest('hex'); h.digest('hex')"),
-      njs_str("Error: Digest already called") },
-
-    { njs_str("var h = require('crypto').createHash('sha1');"
-              "h.update('A').digest('hex'); h.update('B')"),
-      njs_str("Error: Digest already called") },
-
-    { njs_str("typeof require('crypto').createHash('md5')"),
-      njs_str("object") },
-
-    /* require('crypto').createHmac() */
-
-    { njs_str("var h = require('crypto').createHmac('sha1', '');"
-              "[Object.prototype.toString.call(h), njs.dump(h),h]"),
-      njs_str("[object Hmac],Hmac {},[object Hmac]") },
-
-    { njs_str("var hmac = require('crypto').createHmac.bind(undefined, 'md5', '');"
-              "['hex', 'base64', 'base64url'].map(e => {"
-              "   var h = hmac().update('AB').digest().toString(e);"
-              "   var h2 = hmac().update(Buffer.from('XABX').subarray(1,3)).digest(e);"
-              "   var h3 = hmac().update('A').update('B').digest(e);"
-              "   if (h !== h2) {throw new Error(`digest().toString($e):$h != digest($e):$h2`)};"
-              "   if (h !== h3) {throw new Error(`digest().toString($e):$h != update('A').update('B').digest($e):$h3`)};"
-              "   return h;"
-              "})"),
-      njs_str("9e0e9e545ef63d41dfb653daecf8ebc7,"
-              "ng6eVF72PUHftlPa7Pjrxw==,"
-              "ng6eVF72PUHftlPa7Pjrxw") },
-
-    { njs_str("var hmac = require('crypto').createHmac.bind(undefined, 'sha1', '');"
-              "['hex', 'base64', 'base64url'].map(e => {"
-              "   var h = hmac().update('AB').digest().toString(e);"
-              "   var h2 = hmac().update(Buffer.from('XABX').subarray(1,3)).digest(e);"
-              "   var h3 = hmac().update('A').update('B').digest(e);"
-              "   if (h !== h2) {throw new Error(`digest().toString($e):$h != digest($e):$h2`)};"
-              "   if (h !== h3) {throw new Error(`digest().toString($e):$h != update('A').update('B').digest($e):$h3`)};"
-              "   return h;"
-              "})"),
-      njs_str("d32c0b6637cc2dfe4670f3fe48ef4434123c4810,"
-              "0ywLZjfMLf5GcPP+SO9ENBI8SBA=,"
-              "0ywLZjfMLf5GcPP-SO9ENBI8SBA") },
-
-    { njs_str("var hash = require('crypto').createHmac.bind(undefined, 'sha1', '');"
-              "["
-              " ['AB'],"
-              " ['4142', 'hex'],"
-              " ['QUI=', 'base64'],"
-              " ['QUI', 'base64url']"
-              "].every(args => {"
-              "        return hash().update(args[0], args[1]).digest('hex') === 'd32c0b6637cc2dfe4670f3fe48ef4434123c4810';"
-              "})"),
-      njs_str("true") },
-
-    { njs_str("var hmac = require('crypto').createHmac.bind(undefined, 'sha256', '');"
-              "['hex', 'base64', 'base64url'].map(e => {"
-              "   var h = hmac().update('AB').digest().toString(e);"
-              "   var h2 = hmac().update(Buffer.from('AB')).digest(e);"
-              "   var h3 = hmac().update('A').update('B').digest(e);"
-              "   if (h !== h2) {throw new Error(`digest().toString($e):$h != digest($e):$h2`)};"
-              "   if (h !== h3) {throw new Error(`digest().toString($e):$h != update('A').update('B').digest($e):$h3`)};"
-              "   return h;"
-              "})"),
-      njs_str("d53400095496267cf02e5dbd4b0bf9fbfb5f36f311ea7d9809af5487421743e3,"
-              "1TQACVSWJnzwLl29Swv5+/tfNvMR6n2YCa9Uh0IXQ+M=,"
-              "1TQACVSWJnzwLl29Swv5-_tfNvMR6n2YCa9Uh0IXQ-M") },
-
-    { njs_str("var hmac = require('crypto').createHmac;"
-              "njs.dump(['', 'abc'.repeat(100)].map(v => {"
-              "    return ['md5', 'sha1', 'sha256'].map(h => {"
-              "        return hmac(h, Buffer.from('secret')).update(v).digest('hex');"
-              "     })"
-              "}))"),
-      njs_str("[['5c8db03f04cec0f43bcb060023914190',"
-              "'25af6174a0fcecc4d346680a72b7ce644b9a88e8',"
-              "'f9e66e179b6747ae54108f82f8ade8b3c25d76fd30afde6c395822c530196169'],"
-              "['91eb74a225cdd3bbfccc34396c6e3ac5',"
-              "'0aac71e3a813a7acc4a809cfdedb2ecba04ffc5e',"
-              "'8660d2d51d6f20f61d5aadfb6c43df7fd05fc2fc4967d8aec1846f3d9ec03987']]") },
-
-    { njs_str("var h = require('crypto').createHmac('sha1', '');"
-              "var Hmac = h.constructor; "
-              "Hmac('sha1', '').digest('hex')"),
-      njs_str("fbdb1d1b18aa6c08324b7d64b71fb76370690e1d") },
-
-    { njs_str("var h = require('crypto').createHmac('sha1', '');"
-              "h.constructor.name"),
-      njs_str("Hmac") },
-
-    { njs_str("require('crypto').createHmac('sha1', '').digest() instanceof Buffer"),
-      njs_str("true") },
-
-    { njs_str("var h = require('crypto').createHmac('sha256', 'A'.repeat(64));"
-              "h.update('AB').digest('hex')"),
-      njs_str("ee9dce43b12eb3e865614ad9c1a8d4fad4b6eac2b64647bd24cd192888d3f367") },
-
-    { njs_str("var h = require('crypto').createHmac('sha256', 'A'.repeat(100));"
-              "h.update('AB').digest('hex')"),
-      njs_str("5647b6c429701ff512f0f18232b4507065d2376ca8899a816a0a6e721bf8ddcc") },
-
-    { njs_str("var h = require('crypto').createHmac()"),
-      njs_str("TypeError: algorithm must be a string") },
-
-    { njs_str("var h = require('crypto').createHmac([])"),
-      njs_str("TypeError: algorithm must be a string") },
-
-    { njs_str("var h = require('crypto').createHmac('sha512', '')"),
-      njs_str("TypeError: not supported algorithm: \"sha512\"") },
-
-    { njs_str("var h = require('crypto').createHmac('sha1', [])"),
-      njs_str("TypeError: key argument \"array\" is not a string or Buffer-like object") },
-
-    { njs_str("var h = require('crypto').createHmac('sha1', 'secret key');"
-              "h.update('A').digest('hex'); h.digest('hex')"),
-      njs_str("Error: Digest already called") },
-
-    { njs_str("var h = require('crypto').createHmac('sha1', 'secret key');"
-              "h.update('A').digest('hex'); h.update('B')"),
-      njs_str("Error: Digest already called") },
-
-    { njs_str("typeof require('crypto').createHmac('md5', 'a')"),
-      njs_str("object") },
-
-    { njs_str("var cr = require('crypto'); var h = cr.createHash('sha1');"
-              "h.update.call(cr.createHmac('sha1', 's'), '')"),
-      njs_str("TypeError: \"this\" is not a hash object") },
-
     /* setTimeout(). */
 
     { njs_str("setTimeout()"),
@@ -18614,8 +18074,1296 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("var t = \"123\"; t = parseInt(t); t"),
       njs_str("123") },
 
-    /* Query String */
-
+    /* TextEncoder. */
+
+    { njs_str("var en = new TextEncoder(); typeof en.encode()"),
+      njs_str("object") },
+
+    { njs_str("var en = new TextEncoder(); en.encode()"),
+      njs_str("") },
+
+    { njs_str("var en = new TextEncoder(); var res = en.encode('?'); res"),
+      njs_str("206,177") },
+
+    { njs_str("var en = new TextEncoder(); var res = en.encode('?1?'); res[2]"),
+      njs_str("49") },
+
+    { njs_str("var en = new TextEncoder(); en.encode(String.bytesFrom([0xCE]))"),
+      njs_str("239,191,189") },
+
+    { njs_str("var en = new TextEncoder();"
+              "en.encode(String.bytesFrom([0xCE, 0xB1, 0xCE]))"),
+      njs_str("206,177,239,191,189") },
+
+    { njs_str("var en = new TextEncoder();"
+              "en.encode(String.bytesFrom([0xCE, 0xCE, 0xB1]))"),
+      njs_str("239,191,189,206,177") },
+
+    { njs_str("var en = new TextEncoder(); en.encoding"),
+      njs_str("utf-8") },
+
+    { njs_str("TextEncoder.prototype.encode.apply({}, [])"),
+      njs_str("TypeError: \"this\" is not a TextEncoder") },
+
+    { njs_str("var en = new TextEncoder();"
+              "var utf8 = new Uint8Array(5);"
+              "var res = en.encodeInto('?????', utf8); njs.dump(res)"),
+      njs_str("{read:2,written:4}") },
+
+    { njs_str("var en = new TextEncoder();"
+              "var utf8 = new Uint8Array(10);"
+              "var res = en.encodeInto('?????', utf8); njs.dump(res)"),
+      njs_str("{read:5,written:10}") },
+
+    { njs_str("var en = new TextEncoder();"
+              "var utf8 = new Uint8Array(10);"
+              "en.encodeInto('?????', utf8.subarray(2)); utf8[0]"),
+      njs_str("0") },
+
+    { njs_str("var str = String.bytesFrom([0xCE]);"
+              "var en = new TextEncoder();"
+              "var utf8 = new Uint8Array(3);"
+              "var res = en.encodeInto(str, utf8); "
+              "[njs.dump(res), utf8]"),
+      njs_str("{read:1,written:3},239,191,189") },
+
+    { njs_str("var str = String.bytesFrom([0xCE]);"
+              "var en = new TextEncoder();"
+              "var utf8 = new Uint8Array(5);"
+              "en.encodeInto(str, utf8); utf8"),
+      njs_str("239,191,189,0,0") },
+
+    { njs_str("var str = String.bytesFrom([0xCE, 0xB1, 0xCE]);"
+              "var en = new TextEncoder();"
+              "var utf8 = new Uint8Array(5);"
+              "var res = en.encodeInto(str, utf8);"
+              "[njs.dump(res), utf8]"),
+      njs_str("{read:2,written:5},206,177,239,191,189") },
+
+    { njs_str("var str = String.bytesFrom([0xCE, 0xCE, 0xB1]);"
+              "var en = new TextEncoder();"
+              "var utf8 = new Uint8Array(5);"
+              "var res = en.encodeInto(str, utf8);"
+              "[njs.dump(res), utf8]"),
+      njs_str("{read:2,written:5},239,191,189,206,177") },
+
+    { njs_str("TextEncoder.prototype.encodeInto.apply({}, [])"),
+      njs_str("TypeError: \"this\" is not a TextEncoder") },
+
+    { njs_str("(new TextEncoder()).encodeInto('', 0.12) "),
+      njs_str("TypeError: The \"destination\" argument must be an instance of Uint8Array") },
+
+    /* TextDecoder. */
+
+    { njs_str("var de = new TextDecoder();"
+              "var u8arr = new Uint8Array([240, 160, 174, 183]);"
+              "var u16arr = new Uint16Array(u8arr.buffer);"
+              "var u32arr = new Uint32Array(u8arr.buffer);"
+              "[u8arr, u16arr, u32arr].map(v=>de.decode(v)).join(',')"),
+      njs_str("?,?,?") },
+
+    { njs_str("var de = new TextDecoder();"
+              "[new Uint8Array([240, 160]), "
+              " new Uint8Array([174]), "
+              " new Uint8Array([183])].map(v=>de.decode(v, {stream: 1}))[2]"),
+      njs_str("?") },
+
+    { njs_str("var de = new TextDecoder();"
+              "de.decode(new Uint8Array([240, 160]), {stream: 1});"
+              "de.decode(new Uint8Array([174]), {stream: 1});"
+              "de.decode(new Uint8Array([183]))"),
+      njs_str("?") },
+
+    { njs_str("var de = new TextDecoder();"
+              "de.decode(new Uint8Array([240, 160]), {stream: 1});"
+              "de.decode()"),
+      njs_str("?") },
+
+    { njs_str("var de = new TextDecoder('utf-8', {fatal: true});"
+              "de.decode(new Uint8Array([240, 160]))"),
+      njs_str("TypeError: The encoded data was not valid") },
+
+    { njs_str("var de = new TextDecoder('utf-8', {fatal: false});"
+              "de.decode(new Uint8Array([240, 160]))"),
+      njs_str("?") },
+
+    { njs_str("var en = new TextEncoder();"
+              "var de = new TextDecoder('utf-8', {ignoreBOM: true});"
+              "en.encode(de.decode(new Uint8Array([239, 187, 191, 50])))"),
+      njs_str("239,187,191,50") },
+
+    { njs_str("var en = new TextEncoder();"
+              "var de = new TextDecoder('utf-8', {ignoreBOM: false});"
+              "en.encode(de.decode(new Uint8Array([239, 187, 191, 50])))"),
+      njs_str("50") },
+
+    { njs_str("var en = new TextEncoder(); var de = new TextDecoder();"
+              "en.encode(de.decode(new Uint8Array([239, 187, 191, 50])))"),
+      njs_str("50") },
+
+    { njs_str("var de = new TextDecoder(); de.decode('')"),
+      njs_str("TypeError: The \"input\" argument must be an instance of TypedArray") },
+
+    { njs_str("var de = new TextDecoder({})"),
+      njs_str("RangeError: The \"[object Object]\" encoding is not supported") },
+
+    { njs_str("var de = new TextDecoder('foo')"),
+      njs_str("RangeError: The \"foo\" encoding is not supported") },
+
+    { njs_str("var de = new TextDecoder(); de.encoding"),
+      njs_str("utf-8") },
+
+    { njs_str("var de = new TextDecoder(); de.fatal"),
+      njs_str("false") },
+
+    { njs_str("var de = new TextDecoder(); de.ignoreBOM"),
+      njs_str("false") },
+
+    { njs_str("TextDecoder.prototype.decode.apply({}, new Uint8Array([1]))"),
+      njs_str("TypeError: \"this\" is not a TextDecoder") },
+
+    { njs_str("var de = new TextDecoder();"
+              "var buf = new Uint32Array([1,2,3]).buffer;"
+              "var en = new TextEncoder();"
+              "njs.dump(new Uint32Array(en.encode(de.decode(buf)).buffer))"),
+      njs_str("Uint32Array [1,2,3]") },
+
+    { njs_str("var de = new TextDecoder();"
+              "var buf = new Uint32Array([1,2,3]).subarray(1,2);"
+              "var en = new TextEncoder();"
+              "njs.dump(new Uint32Array(en.encode(de.decode(buf)).buffer))"),
+      njs_str("Uint32Array [2]") },
+
+    /* let */
+
+    { njs_str("let x"),
+      njs_str("undefined") },
+
+    { njs_str("let x = 123; x"),
+      njs_str("123") },
+
+    { njs_str("let x = [123]; x"),
+      njs_str("123") },
+
+    { njs_str("let x = () => x; x()"),
+      njs_str("[object Function]") },
+
+    { njs_str("let x = (() => x)()"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("x; let x"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("x; let x = 123"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("let x = x + 123"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("let x = (x, 1)"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("let x = x"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("let x; var x"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("var x; let x"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("let x; function x() {}"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("function x() {} let x"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("function x() {let x; var x}"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("function x() {var x; let x}"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("var x = function f() {let f}"),
+      njs_str("undefined") },
+
+    { njs_str("let a; let x = 1;"
+              "{let x = 2; a = x}"
+              "[x, a]"),
+      njs_str("1,2") },
+
+    { njs_str("let a; let x = 1;"
+              "if (true) {let x = 2; a = x}"
+              "[x, a]"),
+      njs_str("1,2") },
+
+    { njs_str("var a = 5, b = 10, arr = [];"
+              "{let a = 4; var b = 1; arr.push(a); arr.push(b)}"
+              "arr.push(a); arr.push(b); arr"),
+      njs_str("4,1,5,1") },
+
+    { njs_str("function func() {return x}"
+              "let x = 123;"
+              "func()"),
+      njs_str("123") },
+
+    { njs_str("function func() {return x}"
+              "func();"
+              "let x = 123"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("function func() {return () => x}"
+              "let x = 123;"
+              "func()()"),
+      njs_str("123") },
+
+    { njs_str("function func() {x = x + 1; let x}"),
+      njs_str("undefined") },
+
+    { njs_str("function func() {return () => x}"
+              "func()();"
+              "let x = 123;"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("var arr = [];"
+              ""
+              "for (var i = 0; i < 10; i++) {"
+              "    let x = i;"
+              ""
+              "    arr.push( (n) => {x += n; return x} );"
+              "}"
+              ""
+              "["
+              "    arr[0](2), arr[1](1), arr[2](4), arr[3](7), arr[4](0),"
+              "    arr[5](1), arr[6](2), arr[7](5), arr[8](8), arr[9](10)"
+              "]"),
+      njs_str("2,2,6,10,4,6,8,12,16,19") },
+
+    { njs_str("var arr = [];"
+              ""
+              "for (let i = 0; i < 10; i++) {"
+              "    arr.push( (n) => {i += n; return i} );"
+              "}"
+              ""
+              "["
+              "    arr[0](2), arr[1](1), arr[2](4), arr[3](7), arr[4](0),"
+              "    arr[5](1), arr[6](2), arr[7](5), arr[8](8), arr[9](10)"
+              "]"),
+      njs_str("2,2,6,10,4,6,8,12,16,19") },
+
+    { njs_str("for (let i = 0; i < 1; i++) {"
+              "    let i = i + 2;"
+              "}"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("let arr = [], res = [];"
+              "for (let i = 0, f = function() { return i }; i < 5; i++) {"
+              "    arr.push(f);"
+              "}"
+              "for (let i = 0; i < 5; i++) {"
+              "    res.push(arr[i]());"
+              "} res"),
+      njs_str("0,0,0,0,0") },
+
+    { njs_str("let arr = [], res = [];"
+              "for (let i = 0; arr.push(() => i), i < 10; i++) {}"
+              "for (let k = 0; k < 10; k++) {res.push(arr[k]())}"
+              "res"),
+      njs_str("0,1,2,3,4,5,6,7,8,9") },
+
+    { njs_str("let res = [];"
+              "for (let n in [1,2,3]) {res.push(n)}"
+              "res"),
+      njs_str("0,1,2") },
+
+    { njs_str("let arr = [], res = [];"
+              ""
+              "for (let n in [1,2,3]) {"
+              "    arr.push(() => n);"
+              "}"
+              ""
+              "for (let n in arr) {"
+              "    res.push(arr[n]());"
+              "}"
+              "res"),
+      njs_str("0,1,2") },
+
+    { njs_str("let arr = [];"
+              ""
+              "for (let n in [1,2,3]) {"
+              "    let n = 1;"
+              "    arr.push(n);"
+              "}"
+              "arr"),
+      njs_str("1,1,1") },
+
+    { njs_str("for (let n in [1,2,3]) {"
+              "    let n = n + 1;"
+              "}"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("for (let n in [1,2,3]) {}"
+              "n"),
+      njs_str("ReferenceError: \"n\" is not defined") },
+
+    { njs_str("for (let n in [1,n,3]) {}"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("(function() {"
+              "function f() {return x + 1}"
+              "function abc() {f()};"
+              "abc();"
+              "let x;"
+              "}())"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("function func() {var x = 1; {let x = x + 1} } func()"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("if (false) let x = 1"),
+      njs_str("SyntaxError: let declaration cannot appear in a single-statement context in 1") },
+
+    { njs_str("while (false) let x = 1"),
+      njs_str("SyntaxError: let declaration cannot appear in a single-statement context in 1") },
+
+    { njs_str("for (;;) let x = 1"),
+      njs_str("SyntaxError: let declaration cannot appear in a single-statement context in 1") },
+
+    { njs_str("try {} catch (e) {let e}"),
+      njs_str("SyntaxError: \"e\" has already been declared in 1") },
+
+    { njs_str("let arr = [], x = 2;"
+              "switch(true) {default: let x = 1; arr.push(x)}"
+              "arr.push(x); arr"),
+      njs_str("1,2") },
+
+    { njs_str("switch(true) {case false: let x = 1; default: x = 2}"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("let res;"
+              "switch(true) {case true: let x = 1; default: x = 2; res = x} res"),
+      njs_str("2") },
+
+    { njs_str("let null"),
+      njs_str("SyntaxError: Unexpected token \"null\" in 1") },
+
+    { njs_str("let continue"),
+      njs_str("SyntaxError: Unexpected token \"continue\" in 1") },
+
+    { njs_str("let undefined"),
+      njs_str("SyntaxError: \"undefined\" has already been declared in 1") },
+
+    { njs_str("let a = 1; globalThis.a"),
+      njs_str("undefined") },
+
+    { njs_str("if (false) {x = 2} else {x = 1} let x;"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("let let"),
+      njs_str("SyntaxError: Unexpected token \"let\" in 1") },
+
+    { njs_str("let null"),
+      njs_str("SyntaxError: Unexpected token \"null\" in 1") },
+
+    { njs_str("function let() {}"),
+      njs_str("SyntaxError: Unexpected token \"let\" in 1") },
+
+    { njs_str("function static() {}"),
+      njs_str("SyntaxError: Unexpected token \"static\" in 1") },
+
+    /* const */
+
+    { njs_str("const x"),
+      njs_str("SyntaxError: missing initializer in const declaration") },
+
+    { njs_str("const x = 1; x"),
+      njs_str("1") },
+
+    { njs_str("const x = 1; x = 1"),
+      njs_str("TypeError: assignment to constant variable") },
+
+    { njs_str("function abc() {const x}"),
+      njs_str("SyntaxError: missing initializer in const declaration") },
+
+    { njs_str("const x = [123]; x"),
+      njs_str("123") },
+
+    { njs_str("const x = () => x; x()"),
+      njs_str("[object Function]") },
+
+    { njs_str("const x = (() => x)()"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("x; const x = 123"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("const x = x + 123"),
+      njs_str("ReferenceError: cannot access variable before initialization") },
+
+    { njs_str("const x; var x"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("const x; let x"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("let x; const x"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("const x = 1; function x() {}"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("function x() {} const x = 1"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+
+    { njs_str("function x() {const x; var x}"),
+      njs_str("SyntaxError: \"x\" has already been declared in 1") },
+

From xeioex at nginx.com  Tue Dec 21 17:46:04 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 21 Dec 2021 17:46:04 +0000
Subject: [njs] Renaming njs_module_t -> njs_mod_t.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/bea31b350b5e
branches:  
changeset: 1771:bea31b350b5e
user:      Dmitry Volyntsev 
date:      Tue Dec 21 15:49:21 2021 +0000
description:
Renaming njs_module_t -> njs_mod_t.

diffstat:

 src/njs_builtin.c   |   6 ++--
 src/njs_generator.c |   4 +-
 src/njs_module.c    |  54 ++++++++++++++++++++++++++--------------------------
 src/njs_module.h    |   2 +-
 4 files changed, 33 insertions(+), 33 deletions(-)

diffs (234 lines):

diff -r 1efb00924df8 -r bea31b350b5e src/njs_builtin.c
--- a/src/njs_builtin.c	Tue Dec 21 15:49:13 2021 +0000
+++ b/src/njs_builtin.c	Tue Dec 21 15:49:21 2021 +0000
@@ -135,8 +135,8 @@ njs_int_t
 njs_builtin_objects_create(njs_vm_t *vm)
 {
     njs_int_t                  ret;
+    njs_mod_t                  *module;
     njs_uint_t                 i;
-    njs_module_t               *module;
     njs_object_t               *object, *string_object;
     njs_function_t             *constructor;
     njs_vm_shared_t            *shared;
@@ -237,7 +237,7 @@ njs_builtin_objects_create(njs_vm_t *vm)
     for (p = njs_module_init; *p != NULL; p++) {
         obj = *p;
 
-        module = njs_mp_zalloc(vm->mem_pool, sizeof(njs_module_t));
+        module = njs_mp_zalloc(vm->mem_pool, sizeof(njs_mod_t));
         if (njs_slow_path(module == NULL)) {
             return NJS_ERROR;
         }
@@ -760,9 +760,9 @@ njs_builtin_match_native_function(njs_vm
 {
     uint8_t                 magic8;
     njs_int_t               ret;
+    njs_mod_t               *module;
     njs_uint_t              i, n;
     njs_value_t             value;
-    njs_module_t            *module;
     njs_lvlhsh_each_t       lhe;
     njs_function_name_t     *fn;
     njs_function_native_t   native;
diff -r 1efb00924df8 -r bea31b350b5e src/njs_generator.c
--- a/src/njs_generator.c	Tue Dec 21 15:49:13 2021 +0000
+++ b/src/njs_generator.c	Tue Dec 21 15:49:21 2021 +0000
@@ -4649,13 +4649,13 @@ static njs_int_t
 njs_generate_import_statement_end(njs_vm_t *vm, njs_generator_t *generator,
     njs_parser_node_t *node)
 {
-    njs_module_t              *module;
+    njs_mod_t                 *module;
     njs_parser_node_t         *expr;
     njs_vmcode_object_copy_t  *copy;
 
     expr = node->right;
 
-    module = (njs_module_t *) expr->index;
+    module = (njs_mod_t *) expr->index;
 
     njs_generate_code(generator, njs_vmcode_object_copy_t, copy,
                       NJS_VMCODE_OBJECT_COPY, 2, node);
diff -r 1efb00924df8 -r bea31b350b5e src/njs_module.c
--- a/src/njs_module.c	Tue Dec 21 15:49:13 2021 +0000
+++ b/src/njs_module.c	Tue Dec 21 15:49:21 2021 +0000
@@ -37,19 +37,19 @@ static njs_int_t njs_module_absolute_pat
 static njs_bool_t njs_module_realpath_equal(const njs_str_t *path1,
     const njs_str_t *path2);
 static njs_int_t njs_module_read(njs_vm_t *vm, int fd, njs_str_t *body);
-static njs_module_t *njs_module_find(njs_vm_t *vm, njs_str_t *name,
+static njs_mod_t *njs_module_find(njs_vm_t *vm, njs_str_t *name,
     njs_bool_t local);
-static njs_module_t *njs_module_add(njs_vm_t *vm, njs_str_t *name);
-static njs_int_t njs_module_insert(njs_parser_t *parser, njs_module_t *module);
+static njs_mod_t *njs_module_add(njs_vm_t *vm, njs_str_t *name);
+static njs_int_t njs_module_insert(njs_parser_t *parser, njs_mod_t *module);
 
 
 njs_int_t
 njs_module_load(njs_vm_t *vm)
 {
-    njs_int_t     ret;
-    njs_uint_t    i;
-    njs_value_t   *value;
-    njs_module_t  **item, *module;
+    njs_int_t    ret;
+    njs_mod_t    **item, *module;
+    njs_uint_t   i;
+    njs_value_t  *value;
 
     if (vm->modules == NULL) {
         return NJS_OK;
@@ -82,8 +82,8 @@ njs_module_load(njs_vm_t *vm)
 void
 njs_module_reset(njs_vm_t *vm)
 {
+    njs_mod_t           **item, *module;
     njs_uint_t          i;
-    njs_module_t        **item, *module;
     njs_lvlhsh_query_t  lhq;
 
     if (vm->modules == NULL) {
@@ -117,7 +117,7 @@ njs_parser_module(njs_parser_t *parser, 
 {
     njs_int_t          ret;
     njs_str_t          name, text;
-    njs_module_t       *module;
+    njs_mod_t          *module;
     njs_module_temp_t  *temp;
     njs_module_info_t  info;
 
@@ -217,7 +217,7 @@ njs_int_t
 njs_parser_module_lambda_after(njs_parser_t *parser, njs_lexer_token_t *token,
     njs_queue_link_t *current)
 {
-    njs_module_t       *module;
+    njs_mod_t          *module;
     njs_module_temp_t  *temp;
 
     temp = (njs_module_temp_t *) parser->target;
@@ -261,7 +261,7 @@ njs_parser_module_after(njs_parser_t *pa
     njs_queue_link_t *current)
 {
     njs_int_t          ret;
-    njs_module_t       *module;
+    njs_mod_t          *module;
     njs_parser_node_t  *node;
 
     node = njs_parser_node_new(parser, 0);
@@ -271,7 +271,7 @@ njs_parser_module_after(njs_parser_t *pa
 
     node->left = parser->node;
 
-    module = (njs_module_t *) parser->target;
+    module = (njs_mod_t *) parser->target;
 
     if (module->index == 0) {
         ret = njs_module_insert(parser, module);
@@ -453,7 +453,7 @@ njs_module_realpath_equal(const njs_str_
 static njs_int_t
 njs_module_hash_test(njs_lvlhsh_query_t *lhq, void *data)
 {
-    njs_module_t  *module;
+    njs_mod_t  *module;
 
     module = data;
 
@@ -475,11 +475,11 @@ const njs_lvlhsh_proto_t  njs_modules_ha
 };
 
 
-static njs_module_t *
+static njs_mod_t *
 njs_module_find(njs_vm_t *vm, njs_str_t *name, njs_bool_t local)
 {
     njs_int_t           ret;
-    njs_module_t        *module, *shared;
+    njs_mod_t           *shared, *module;
     njs_object_t        *object;
     njs_lvlhsh_query_t  lhq;
 
@@ -498,13 +498,13 @@ njs_module_find(njs_vm_t *vm, njs_str_t 
             return shared;
         }
 
-        module = njs_mp_alloc(vm->mem_pool, sizeof(njs_module_t));
+        module = njs_mp_alloc(vm->mem_pool, sizeof(njs_mod_t));
         if (njs_slow_path(module == NULL)) {
             njs_memory_error(vm);
             return NULL;
         }
 
-        memcpy(module, shared, sizeof(njs_module_t));
+        memcpy(module, shared, sizeof(njs_mod_t));
         object = &module->object;
 
         object->__proto__ = &vm->prototypes[NJS_OBJ_TYPE_OBJECT].object;
@@ -528,14 +528,14 @@ njs_module_find(njs_vm_t *vm, njs_str_t 
 }
 
 
-static njs_module_t *
+static njs_mod_t *
 njs_module_add(njs_vm_t *vm, njs_str_t *name)
 {
     njs_int_t           ret;
-    njs_module_t        *module;
+    njs_mod_t           *module;
     njs_lvlhsh_query_t  lhq;
 
-    module = njs_mp_zalloc(vm->mem_pool, sizeof(njs_module_t));
+    module = njs_mp_zalloc(vm->mem_pool, sizeof(njs_mod_t));
     if (njs_slow_path(module == NULL)) {
         njs_memory_error(vm);
         return NULL;
@@ -569,10 +569,10 @@ njs_module_add(njs_vm_t *vm, njs_str_t *
 
 
 static njs_int_t
-njs_module_insert(njs_parser_t *parser, njs_module_t *module)
+njs_module_insert(njs_parser_t *parser, njs_mod_t *module)
 {
     njs_vm_t            *vm;
-    njs_module_t        **value;
+    njs_mod_t           **value;
     njs_parser_scope_t  *scope;
 
     scope = njs_parser_global_scope(parser);
@@ -583,7 +583,7 @@ njs_module_insert(njs_parser_t *parser, 
     scope->items++;
 
     if (vm->modules == NULL) {
-        vm->modules = njs_arr_create(vm->mem_pool, 4, sizeof(njs_module_t *));
+        vm->modules = njs_arr_create(vm->mem_pool, 4, sizeof(njs_mod_t *));
         if (njs_slow_path(vm->modules == NULL)) {
             return NJS_ERROR;
         }
@@ -604,10 +604,10 @@ njs_int_t
 njs_module_require(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t unused)
 {
-    njs_int_t     ret;
-    njs_str_t     name;
-    njs_value_t   *path;
-    njs_module_t  *module;
+    njs_int_t    ret;
+    njs_str_t    name;
+    njs_mod_t    *module;
+    njs_value_t  *path;
 
     if (nargs < 2) {
         njs_type_error(vm, "missing path");
diff -r 1efb00924df8 -r bea31b350b5e src/njs_module.h
--- a/src/njs_module.h	Tue Dec 21 15:49:13 2021 +0000
+++ b/src/njs_module.h	Tue Dec 21 15:49:21 2021 +0000
@@ -13,7 +13,7 @@ typedef struct {
     njs_object_t                object;
     njs_index_t                 index;
     njs_function_t              function;
-} njs_module_t;
+} njs_mod_t;
 
 
 njs_int_t njs_module_load(njs_vm_t *vm);

From xeioex at nginx.com  Tue Dec 21 17:46:06 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 21 Dec 2021 17:46:06 +0000
Subject: [njs] Refactor modules using external prototypes.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/4d4657128baf
branches:  
changeset: 1772:4d4657128baf
user:      Dmitry Volyntsev 
date:      Tue Dec 21 17:42:26 2021 +0000
description:
Refactor modules using external prototypes.

diffstat:

 auto/init                     |     3 +
 auto/make                     |    45 +-
 auto/module                   |     6 +
 auto/modules                  |    34 +
 auto/options                  |     2 +
 auto/sources                  |     4 -
 configure                     |     1 +
 external/njs_crypto.c         |   650 ++++++++
 external/njs_fs.c             |  3094 +++++++++++++++++++++++++++++++++++++++++
 external/njs_query_string.c   |   971 ++++++++++++
 external/njs_webcrypto.c      |    13 +-
 external/njs_webcrypto.h      |    15 -
 nginx/config                  |     3 +-
 nginx/config.make             |     2 +-
 nginx/ngx_http_js_module.c    |     1 +
 nginx/ngx_js.c                |    16 +-
 nginx/ngx_js.h                |     3 +
 nginx/ngx_stream_js_module.c  |     1 +
 src/njs.h                     |     9 +
 src/njs_buffer.c              |   128 +-
 src/njs_buffer.h              |     1 -
 src/njs_builtin.c             |   118 +-
 src/njs_crypto.c              |   693 ---------
 src/njs_crypto.h              |    16 -
 src/njs_fs.c                  |  2946 ---------------------------------------
 src/njs_fs.h                  |    16 -
 src/njs_main.h                |     4 -
 src/njs_module.c              |    60 +-
 src/njs_module.h              |     4 +-
 src/njs_query_string.c        |   924 ------------
 src/njs_query_string.h        |    12 -
 src/njs_shell.c               |    41 +-
 src/njs_value.h               |     2 -
 src/njs_vm.c                  |    25 +-
 src/njs_vm.h                  |     4 -
 src/test/njs_externals_test.c |    14 -
 src/test/njs_unit_test.c      |    12 +-
 37 files changed, 5042 insertions(+), 4851 deletions(-)

diffs (truncated from 10539 to 1000 lines):

diff -r bea31b350b5e -r 4d4657128baf auto/init
--- a/auto/init	Tue Dec 21 15:49:21 2021 +0000
+++ b/auto/init	Tue Dec 21 17:42:26 2021 +0000
@@ -20,6 +20,9 @@ NJS_AUTOCONF_ERR=$NJS_BUILD_DIR/autoconf
 NJS_AUTO_CONFIG_H=$NJS_BUILD_DIR/njs_auto_config.h
 NJS_MAKEFILE=$NJS_BUILD_DIR/Makefile
 
+NJS_LIB_MODULES=
+NJS_LIB_INCS="src $NJS_BUILD_DIR"
+
 test -d $NJS_BUILD_DIR || mkdir $NJS_BUILD_DIR
 
 > $NJS_AUTOCONF_ERR
diff -r bea31b350b5e -r 4d4657128baf auto/make
--- a/auto/make	Tue Dec 21 15:49:21 2021 +0000
+++ b/auto/make	Tue Dec 21 17:42:26 2021 +0000
@@ -7,11 +7,44 @@
 echo "creating $NJS_MAKEFILE"
 
 mkdir -p $NJS_BUILD_DIR/src
+mkdir -p $NJS_BUILD_DIR/build
+mkdir -p $NJS_BUILD_DIR/external
 mkdir -p $NJS_BUILD_DIR/test
 
+njs_modules_c=$NJS_BUILD_DIR/njs_modules.c
+
+NJS_LIB_SRCS="$NJS_LIB_SRCS $njs_modules_c"
+
+njs_incs=`echo $NJS_LIB_INCS \
+        | sed -e "s# *\([^ ]*\)#$njs_regex_cont-I\1#g"`
 njs_objs=`echo $NJS_LIB_SRCS \
         | sed -e "s# *\([^ ]*\.\)c#$NJS_BUILD_DIR/\1o$njs_regex_cont#g"`
 
+cat << END                                    > $njs_modules_c
+
+#include 
+
+END
+
+for mod in $NJS_LIB_MODULES
+do
+    echo "extern njs_module_t  $mod;"         >> $njs_modules_c
+done
+
+echo                                          >> $njs_modules_c
+echo 'njs_module_t *njs_modules[] = {'        >> $njs_modules_c
+
+for mod in $NJS_LIB_MODULES
+do
+    echo "    &$mod,"                         >> $njs_modules_c
+done
+
+cat << END                                    >> $njs_modules_c
+    NULL
+};
+
+END
+
 cat << END > $NJS_MAKEFILE
 
 # This file is auto-generated by configure
@@ -28,8 +61,7 @@ NPM = npm
 
 default: njs
 
-NJS_LIB_INCS = -Isrc -I$NJS_BUILD_DIR
-
+NJS_LIB_INCS = $njs_incs
 NJS_LIB_OBJS = $njs_objs
 
 libnjs: $NJS_BUILD_DIR/libnjs.a
@@ -46,9 +78,8 @@ END
 
 for njs_src in $NJS_LIB_SRCS
 do
-    fname=$(basename $njs_src)
-    njs_obj="src/${fname%.c}.o"
-    njs_dep="src/${fname%.c}.dep"
+    njs_obj="${njs_src%.c}.o"
+    njs_dep="${njs_src%.c}.dep"
     njs_dep_flags=`njs_gen_dep_flags $njs_dep $njs_obj`
     njs_dep_post=`njs_gen_dep_post $njs_dep $njs_obj`
     cat << END >> $NJS_MAKEFILE
@@ -73,7 +104,7 @@ cat << END >> $NJS_MAKEFILE
 
 $NJS_BUILD_DIR/njs: \\
 	$NJS_BUILD_DIR/libnjs.a \\
-	src/njs_shell.c external/njs_webcrypto.h external/njs_webcrypto.c
+	src/njs_shell.c
 	\$(NJS_LINK) -o $NJS_BUILD_DIR/njs \$(NJS_CFLAGS) \\
 		$NJS_LIB_AUX_CFLAGS \$(NJS_LIB_INCS) -Injs \\
 		src/njs_shell.c \\
@@ -143,7 +174,7 @@ njs_dep_post=`njs_gen_dep_post $njs_dep 
 cat << END >> $NJS_MAKEFILE
 
 $NJS_BUILD_DIR/$njs_externals_obj: \\
-    $njs_src external/njs_webcrypto.h external/njs_webcrypto.c
+    $njs_src external/njs_webcrypto.c
 	\$(NJS_CC) -c \$(NJS_CFLAGS) $NJS_LIB_AUX_CFLAGS \\
 		\$(NJS_LIB_INCS) -Injs \\
 		-o $NJS_BUILD_DIR/$njs_externals_obj \\
diff -r bea31b350b5e -r 4d4657128baf auto/module
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/auto/module	Tue Dec 21 17:42:26 2021 +0000
@@ -0,0 +1,6 @@
+# Copyright (C) Dmitry Volyntsev
+# Copyright (C) NGINX, Inc.
+
+NJS_LIB_MODULES="$NJS_LIB_MODULES $njs_module_name"
+NJS_LIB_SRCS="$NJS_LIB_SRCS $njs_module_srcs"
+NJS_LIB_INCS="$NJS_LIB_INCS $njs_module_incs"
diff -r bea31b350b5e -r 4d4657128baf auto/modules
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/auto/modules	Tue Dec 21 17:42:26 2021 +0000
@@ -0,0 +1,34 @@
+# Copyright (C) Dmitry Volyntsev
+# Copyright (C) NGINX, Inc.
+
+njs_module_name=njs_buffer_module
+njs_module_incs=
+njs_module_srcs=src/njs_buffer.c
+
+. auto/module
+
+njs_module_name=njs_crypto_module
+njs_module_incs=
+njs_module_srcs=external/njs_crypto.c
+
+. auto/module
+
+if [ $NJS_WEBCRYPTO = YES -a $NJS_HAVE_OPENSSL = YES ]; then
+	njs_module_name=njs_webcrypto_module
+	njs_module_incs=
+	njs_module_srcs=external/njs_webcrypto.c
+
+	. auto/module
+fi
+
+njs_module_name=njs_fs_module
+njs_module_incs=
+njs_module_srcs=external/njs_fs.c
+
+. auto/module
+
+njs_module_name=njs_query_string_module
+njs_module_incs=
+njs_module_srcs=external/njs_query_string.c
+
+. auto/module
diff -r bea31b350b5e -r 4d4657128baf auto/options
--- a/auto/options	Tue Dec 21 15:49:21 2021 +0000
+++ b/auto/options	Tue Dec 21 17:42:26 2021 +0000
@@ -11,6 +11,7 @@ NJS_DEBUG_MEMORY=NO
 NJS_ADDRESS_SANITIZER=NO
 NJS_TEST262=YES
 
+NJS_WEBCRYPTO=YES
 NJS_TRY_PCRE2=YES
 
 NJS_CONFIGURE_OPTIONS=
@@ -33,6 +34,7 @@ do
         --debug-memory=*)                NJS_DEBUG_MEMORY="$value"           ;;
         --test262=*)                     NJS_TEST262="$value"                ;;
 
+        --no-webcrypto)                  NJS_WEBCRYPTO=NO                    ;;
         --no-pcre2)                      NJS_TRY_PCRE2=NO                    ;;
 
         --help)
diff -r bea31b350b5e -r 4d4657128baf auto/sources
--- a/auto/sources	Tue Dec 21 15:49:21 2021 +0000
+++ b/auto/sources	Tue Dec 21 17:42:26 2021 +0000
@@ -42,8 +42,6 @@ NJS_LIB_SRCS=" \
    src/njs_timer.c \
    src/njs_module.c \
    src/njs_event.c \
-   src/njs_fs.c \
-   src/njs_crypto.c \
    src/njs_extern.c \
    src/njs_variable.c \
    src/njs_builtin.c \
@@ -55,9 +53,7 @@ NJS_LIB_SRCS=" \
    src/njs_array_buffer.c \
    src/njs_typed_array.c \
    src/njs_promise.c \
-   src/njs_query_string.c \
    src/njs_encoding.c \
-   src/njs_buffer.c \
    src/njs_iterator.c \
    src/njs_scope.c \
    src/njs_async.c \
diff -r bea31b350b5e -r 4d4657128baf configure
--- a/configure	Tue Dec 21 15:49:21 2021 +0000
+++ b/configure	Tue Dec 21 17:42:26 2021 +0000
@@ -35,6 +35,7 @@ NJS_LIB_AUX_CFLAGS="$NJS_PCRE_CFLAGS"
 NJS_LIBS="$NJS_LIBRT"
 NJS_LIB_AUX_LIBS="$NJS_PCRE_LIB $NJS_OPENSSL_LIB"
 
+. auto/modules
 . auto/make
 
 . auto/expect
diff -r bea31b350b5e -r 4d4657128baf external/njs_crypto.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/external/njs_crypto.c	Tue Dec 21 17:42:26 2021 +0000
@@ -0,0 +1,650 @@
+
+/*
+ * Copyright (C) Dmitry Volyntsev
+ * Copyright (C) NGINX, Inc.
+ */
+
+
+#include 
+
+
+typedef void (*njs_hash_init)(void *ctx);
+typedef void (*njs_hash_update)(void *ctx, const void *data, size_t size);
+typedef void (*njs_hash_final)(u_char *result, void *ctx);
+
+typedef njs_int_t (*njs_digest_encode)(njs_vm_t *vm, njs_value_t *value,
+    const njs_str_t *src);
+
+
+typedef struct {
+    njs_str_t           name;
+
+    size_t              size;
+    njs_hash_init       init;
+    njs_hash_update     update;
+    njs_hash_final      final;
+} njs_hash_alg_t;
+
+typedef struct {
+    union {
+        njs_md5_t       md5;
+        njs_sha1_t      sha1;
+        njs_sha2_t      sha2;
+    } u;
+
+    njs_hash_alg_t      *alg;
+} njs_digest_t;
+
+typedef struct {
+    u_char              opad[64];
+
+    union {
+        njs_md5_t       md5;
+        njs_sha1_t      sha1;
+        njs_sha2_t      sha2;
+    } u;
+
+    njs_hash_alg_t      *alg;
+} njs_hmac_t;
+
+
+typedef struct {
+    njs_str_t             name;
+
+    njs_digest_encode     encode;
+} njs_crypto_enc_t;
+
+
+static njs_hash_alg_t *njs_crypto_algorithm(njs_vm_t *vm,
+    const njs_value_t *value);
+static njs_crypto_enc_t *njs_crypto_encoding(njs_vm_t *vm,
+    const njs_value_t *value);
+static njs_int_t njs_buffer_digest(njs_vm_t *vm, njs_value_t *value,
+    const njs_str_t *src);
+static njs_int_t njs_crypto_create_hash(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t unused);
+static njs_int_t njs_hash_prototype_update(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t hmac);
+static njs_int_t njs_hash_prototype_digest(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t hmac);
+static njs_int_t njs_crypto_create_hmac(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t unused);
+
+static njs_int_t njs_crypto_init(njs_vm_t *vm);
+
+
+static njs_hash_alg_t njs_hash_algorithms[] = {
+
+   {
+     njs_str("md5"),
+     16,
+     (njs_hash_init) njs_md5_init,
+     (njs_hash_update) njs_md5_update,
+     (njs_hash_final) njs_md5_final
+   },
+
+   {
+     njs_str("sha1"),
+     20,
+     (njs_hash_init) njs_sha1_init,
+     (njs_hash_update) njs_sha1_update,
+     (njs_hash_final) njs_sha1_final
+   },
+
+   {
+     njs_str("sha256"),
+     32,
+     (njs_hash_init) njs_sha2_init,
+     (njs_hash_update) njs_sha2_update,
+     (njs_hash_final) njs_sha2_final
+   },
+
+   {
+    njs_null_str,
+    0,
+    NULL,
+    NULL,
+    NULL
+   }
+
+};
+
+
+static njs_crypto_enc_t njs_encodings[] = {
+
+   {
+     njs_str("buffer"),
+     njs_buffer_digest
+   },
+
+   {
+     njs_str("hex"),
+     njs_string_hex
+   },
+
+   {
+     njs_str("base64"),
+     njs_string_base64
+   },
+
+   {
+     njs_str("base64url"),
+     njs_string_base64url
+   },
+
+   {
+    njs_null_str,
+    NULL
+   }
+};
+
+
+static njs_external_t  njs_ext_crypto_hash[] = {
+
+    {
+        .flags = NJS_EXTERN_PROPERTY | NJS_EXTERN_SYMBOL,
+        .name.symbol = NJS_SYMBOL_TO_STRING_TAG,
+        .u.property = {
+            .value = "Hash",
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("update"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_hash_prototype_update,
+            .magic8 = 0,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("digest"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_hash_prototype_digest,
+            .magic8 = 0,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("constructor"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_crypto_create_hash,
+        }
+    },
+};
+
+
+static njs_external_t  njs_ext_crypto_hmac[] = {
+
+    {
+        .flags = NJS_EXTERN_PROPERTY | NJS_EXTERN_SYMBOL,
+        .name.symbol = NJS_SYMBOL_TO_STRING_TAG,
+        .u.property = {
+            .value = "Hmac",
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("update"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_hash_prototype_update,
+            .magic8 = 1,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("digest"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_hash_prototype_digest,
+            .magic8 = 1,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("constructor"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_crypto_create_hmac,
+            .magic8 = 0,
+        }
+    },
+};
+
+
+static njs_external_t  njs_ext_crypto_crypto[] = {
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("createHash"),
+        .writable = 1,
+        .configurable = 1,
+        .enumerable = 1,
+        .u.method = {
+            .native = njs_crypto_create_hash,
+            .magic8 = 0,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("createHmac"),
+        .writable = 1,
+        .configurable = 1,
+        .enumerable = 1,
+        .u.method = {
+            .native = njs_crypto_create_hmac,
+            .magic8 = 0,
+        }
+    },
+};
+
+
+static njs_int_t    njs_crypto_hash_proto_id;
+static njs_int_t    njs_crypto_hmac_proto_id;
+
+
+njs_module_t  njs_crypto_module = {
+    .name = njs_str("crypto"),
+    .init = njs_crypto_init,
+};
+
+
+static njs_int_t
+njs_crypto_create_hash(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    njs_digest_t    *dgst;
+    njs_hash_alg_t  *alg;
+
+    alg = njs_crypto_algorithm(vm, njs_arg(args, nargs, 1));
+    if (njs_slow_path(alg == NULL)) {
+        return NJS_ERROR;
+    }
+
+    dgst = njs_mp_alloc(vm->mem_pool, sizeof(njs_digest_t));
+    if (njs_slow_path(dgst == NULL)) {
+        njs_memory_error(vm);
+        return NJS_ERROR;
+    }
+
+    dgst->alg = alg;
+
+    alg->init(&dgst->u);
+
+    return njs_vm_external_create(vm, &vm->retval, njs_crypto_hash_proto_id,
+                                  dgst, 0);
+}
+
+
+static njs_int_t
+njs_hash_prototype_update(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t hmac)
+{
+    njs_str_t                    data;
+    njs_int_t                    ret;
+    njs_hmac_t                   *ctx;
+    njs_value_t                  *this, dst;
+    njs_digest_t                 *dgst;
+    njs_typed_array_t            *array;
+    const njs_value_t            *value;
+    njs_array_buffer_t           *buffer;
+    const njs_buffer_encoding_t  *encoding;
+
+    this = njs_argument(args, 0);
+
+    if (!hmac) {
+        dgst = njs_vm_external(vm, njs_crypto_hash_proto_id, this);
+        if (njs_slow_path(dgst == NULL)) {
+            njs_type_error(vm, "\"this\" is not a hash object");
+            return NJS_ERROR;
+        }
+
+        if (njs_slow_path(dgst->alg == NULL)) {
+            njs_error(vm, "Digest already called");
+            return NJS_ERROR;
+        }
+
+        ctx = NULL;
+
+    } else {
+        ctx = njs_vm_external(vm, njs_crypto_hmac_proto_id, this);
+        if (njs_slow_path(ctx == NULL)) {
+            njs_type_error(vm, "\"this\" is not a hmac object");
+            return NJS_ERROR;
+        }
+
+        if (njs_slow_path(ctx->alg == NULL)) {
+            njs_error(vm, "Digest already called");
+            return NJS_ERROR;
+        }
+
+        dgst = NULL;
+    }
+
+    value = njs_arg(args, nargs, 1);
+
+    switch (value->type) {
+    case NJS_STRING:
+        encoding = njs_buffer_encoding(vm, njs_arg(args, nargs, 2));
+        if (njs_slow_path(encoding == NULL)) {
+            return NJS_ERROR;
+        }
+
+        ret = njs_buffer_decode_string(vm, value, &dst, encoding);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_ERROR;
+        }
+
+        njs_string_get(&dst, &data);
+        break;
+
+    case NJS_TYPED_ARRAY:
+    case NJS_DATA_VIEW:
+        array = njs_typed_array(value);
+        buffer = array->buffer;
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
+        data.start = &buffer->u.u8[array->offset];
+        data.length = array->byte_length;
+        break;
+
+    default:
+        njs_type_error(vm, "data argument \"%s\" is not a string "
+                       "or Buffer-like object", njs_type_string(value->type));
+
+        return NJS_ERROR;
+    }
+
+    if (!hmac) {
+        dgst->alg->update(&dgst->u, data.start, data.length);
+
+    } else {
+        ctx->alg->update(&ctx->u, data.start, data.length);
+    }
+
+    vm->retval = *this;
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_hash_prototype_digest(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t hmac)
+{
+    njs_str_t         str;
+    njs_hmac_t        *ctx;
+    njs_value_t       *this;
+    njs_digest_t      *dgst;
+    njs_hash_alg_t    *alg;
+    njs_crypto_enc_t  *enc;
+    u_char            hash1[32], digest[32];
+
+    this = njs_argument(args, 0);
+
+    if (!hmac) {
+        dgst = njs_vm_external(vm, njs_crypto_hash_proto_id, this);
+        if (njs_slow_path(dgst == NULL)) {
+            njs_type_error(vm, "\"this\" is not a hash object");
+            return NJS_ERROR;
+        }
+
+        if (njs_slow_path(dgst->alg == NULL)) {
+            goto exception;
+        }
+
+        ctx = NULL;
+
+    } else {
+        ctx = njs_vm_external(vm, njs_crypto_hmac_proto_id, this);
+        if (njs_slow_path(ctx == NULL)) {
+            njs_type_error(vm, "\"this\" is not a hmac object");
+            return NJS_ERROR;
+        }
+
+        if (njs_slow_path(ctx->alg == NULL)) {
+            goto exception;
+        }
+
+        dgst = NULL;
+    }
+
+    enc = njs_crypto_encoding(vm, njs_arg(args, nargs, 1));
+    if (njs_slow_path(enc == NULL)) {
+        return NJS_ERROR;
+    }
+
+    if (!hmac) {
+        alg = dgst->alg;
+        alg->final(digest, &dgst->u);
+        dgst->alg = NULL;
+
+    } else {
+        alg = ctx->alg;
+        alg->final(hash1, &ctx->u);
+
+        alg->init(&ctx->u);
+        alg->update(&ctx->u, ctx->opad, 64);
+        alg->update(&ctx->u, hash1, alg->size);
+        alg->final(digest, &ctx->u);
+        ctx->alg = NULL;
+    }
+
+    str.start = digest;
+    str.length = alg->size;
+
+    return enc->encode(vm, &vm->retval, &str);
+
+exception:
+
+    njs_error(vm, "Digest already called");
+    return NJS_ERROR;
+}
+
+
+static njs_int_t
+njs_crypto_create_hmac(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    njs_str_t           key;
+    njs_uint_t          i;
+    njs_hmac_t          *ctx;
+    njs_hash_alg_t      *alg;
+    njs_typed_array_t   *array;
+    const njs_value_t   *value;
+    njs_array_buffer_t  *buffer;
+    u_char              digest[32], key_buf[64];
+
+    alg = njs_crypto_algorithm(vm, njs_arg(args, nargs, 1));
+    if (njs_slow_path(alg == NULL)) {
+        return NJS_ERROR;
+    }
+
+    value = njs_arg(args, nargs, 2);
+
+    switch (value->type) {
+    case NJS_STRING:
+        njs_string_get(value, &key);
+        break;
+
+    case NJS_TYPED_ARRAY:
+    case NJS_DATA_VIEW:
+        array = njs_typed_array(value);
+        buffer = array->buffer;
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
+        key.start = &buffer->u.u8[array->offset];
+        key.length = array->byte_length;
+        break;
+
+    default:
+        njs_type_error(vm, "key argument \"%s\" is not a string "
+                       "or Buffer-like object", njs_type_string(value->type));
+
+        return NJS_ERROR;
+    }
+
+    ctx = njs_mp_alloc(vm->mem_pool, sizeof(njs_hmac_t));
+    if (njs_slow_path(ctx == NULL)) {
+        njs_memory_error(vm);
+        return NJS_ERROR;
+    }
+
+    ctx->alg = alg;
+
+    if (key.length > sizeof(key_buf)) {
+        alg->init(&ctx->u);
+        alg->update(&ctx->u, key.start, key.length);
+        alg->final(digest, &ctx->u);
+
+        memcpy(key_buf, digest, alg->size);
+        njs_explicit_memzero(key_buf + alg->size, sizeof(key_buf) - alg->size);
+
+    } else {
+        memcpy(key_buf, key.start, key.length);
+        njs_explicit_memzero(key_buf + key.length,
+                             sizeof(key_buf) - key.length);
+    }
+
+    for (i = 0; i < 64; i++) {
+        ctx->opad[i] = key_buf[i] ^ 0x5c;
+    }
+
+    for (i = 0; i < 64; i++) {
+         key_buf[i] ^= 0x36;
+    }
+
+    alg->init(&ctx->u);
+    alg->update(&ctx->u, key_buf, 64);
+
+    return njs_vm_external_create(vm, &vm->retval, njs_crypto_hmac_proto_id,
+                                  ctx, 0);
+}
+
+
+static njs_hash_alg_t *
+njs_crypto_algorithm(njs_vm_t *vm, const njs_value_t *value)
+{
+    njs_str_t       name;
+    njs_hash_alg_t  *e;
+
+    if (njs_slow_path(!njs_is_string(value))) {
+        njs_type_error(vm, "algorithm must be a string");
+        return NULL;
+    }
+
+    njs_string_get(value, &name);
+
+    for (e = &njs_hash_algorithms[0]; e->name.length != 0; e++) {
+        if (njs_strstr_eq(&name, &e->name)) {
+            return e;
+        }
+    }
+
+    njs_type_error(vm, "not supported algorithm: \"%V\"", &name);
+
+    return NULL;
+}
+
+
+static njs_crypto_enc_t *
+njs_crypto_encoding(njs_vm_t *vm, const njs_value_t *value)
+{
+    njs_str_t         name;
+    njs_crypto_enc_t  *e;
+
+    if (njs_slow_path(!njs_is_string(value))) {
+        if (njs_is_defined(value)) {
+            njs_type_error(vm, "encoding must be a string");
+            return NULL;
+        }
+
+        return &njs_encodings[0];
+    }
+
+    njs_string_get(value, &name);
+
+    for (e = &njs_encodings[1]; e->name.length != 0; e++) {
+        if (njs_strstr_eq(&name, &e->name)) {
+            return e;
+        }
+    }
+
+    njs_type_error(vm, "Unknown digest encoding: \"%V\"", &name);
+
+    return NULL;
+}
+
+
+static njs_int_t
+njs_buffer_digest(njs_vm_t *vm, njs_value_t *value, const njs_str_t *src)
+{
+    return njs_buffer_new(vm, value, src->start, src->length);
+}
+
+
+static njs_int_t
+njs_crypto_init(njs_vm_t *vm)
+{
+    njs_int_t           ret, proto_id;
+    njs_mod_t           *module;
+    njs_opaque_value_t  value;
+
+    njs_crypto_hash_proto_id =
+                     njs_vm_external_prototype(vm, njs_ext_crypto_hash,
+                                               njs_nitems(njs_ext_crypto_hash));
+    if (njs_slow_path(njs_crypto_hash_proto_id < 0)) {
+        return NJS_ERROR;
+    }
+
+    njs_crypto_hmac_proto_id =
+                     njs_vm_external_prototype(vm, njs_ext_crypto_hmac,
+                                               njs_nitems(njs_ext_crypto_hmac));
+    if (njs_slow_path(njs_crypto_hmac_proto_id < 0)) {
+        return NJS_ERROR;
+    }
+
+    proto_id = njs_vm_external_prototype(vm, njs_ext_crypto_crypto,
+                                         njs_nitems(njs_ext_crypto_crypto));
+    if (njs_slow_path(proto_id < 0)) {
+        return NJS_ERROR;
+    }
+
+    ret = njs_vm_external_create(vm, njs_value_arg(&value), proto_id, NULL, 1);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
+    module = njs_module_add(vm, &njs_str_value("crypto"), 1);
+    if (njs_slow_path(module == NULL)) {
+        return NJS_ERROR;
+    }
+
+    njs_value_assign(&module->value, &value);
+    module->function.native = 1;
+
+    return NJS_OK;
+}
diff -r bea31b350b5e -r 4d4657128baf external/njs_fs.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/external/njs_fs.c	Tue Dec 21 17:42:26 2021 +0000
@@ -0,0 +1,3094 @@
+
+/*
+ * Copyright (C) Dmitry Volyntsev
+ * Copyright (C) NGINX, Inc.
+ */
+
+
+#include 
+
+#include 
+
+#if (NJS_SOLARIS)
+
+#define DT_DIR         0
+#define DT_REG         1
+#define DT_CHR         2
+#define DT_LNK         3
+#define DT_BLK         4
+#define DT_FIFO        5
+#define DT_SOCK        6
+#define NJS_DT_INVALID 0xffffffff
+
+#define njs_dentry_type(_dentry)                                             \
+    (NJS_DT_INVALID)
+
+#else
+
+#define NJS_DT_INVALID 0xffffffff
+
+#define njs_dentry_type(_dentry)                                             \
+    ((_dentry)->d_type)
+
+#endif
+
+
+#define njs_fs_magic(calltype, mode)                                         \
+    (((mode) << 2) | calltype)
+
+#define njs_fs_magic2(field, type)                                           \
+    (((type) << 4) | field)
+
+
+typedef enum {
+    NJS_FS_DIRECT,
+    NJS_FS_PROMISE,
+    NJS_FS_CALLBACK,
+} njs_fs_calltype_t;
+
+
+typedef enum {
+    NJS_FS_TRUNC,
+    NJS_FS_APPEND,
+} njs_fs_writemode_t;
+
+
+typedef enum {
+    NJS_FS_STAT,
+    NJS_FS_LSTAT,
+} njs_fs_statmode_t;
+
+
+typedef struct {
+    njs_str_t       name;
+    int             value;
+} njs_fs_entry_t;
+
+
+typedef enum {
+    NJS_FTW_PHYS = 1,
+    NJS_FTW_MOUNT = 2,
+    NJS_FTW_DEPTH = 8,
+} njs_ftw_flags_t;
+
+
+typedef enum {
+    NJS_FTW_F,
+    NJS_FTW_D,
+    NJS_FTW_DNR,
+    NJS_FTW_NS,
+    NJS_FTW_SL,
+    NJS_FTW_DP,
+    NJS_FTW_SLN,
+} njs_ftw_type_t;
+
+
+typedef struct {
+    long tv_sec;
+    long tv_nsec;
+} njs_timespec_t;
+
+
+typedef struct {
+    uint64_t        st_dev;
+    uint64_t        st_mode;
+    uint64_t        st_nlink;
+    uint64_t        st_uid;
+    uint64_t        st_gid;
+    uint64_t        st_rdev;
+    uint64_t        st_ino;
+    uint64_t        st_size;
+    uint64_t        st_blksize;
+    uint64_t        st_blocks;
+    njs_timespec_t  st_atim;
+    njs_timespec_t  st_mtim;
+    njs_timespec_t  st_ctim;
+    njs_timespec_t  st_birthtim;
+} njs_stat_t;
+
+
+typedef enum {
+    NJS_FS_STAT_DEV,
+    NJS_FS_STAT_INO,
+    NJS_FS_STAT_MODE,
+    NJS_FS_STAT_NLINK,
+    NJS_FS_STAT_UID,
+    NJS_FS_STAT_GID,
+    NJS_FS_STAT_RDEV,
+    NJS_FS_STAT_SIZE,
+    NJS_FS_STAT_BLKSIZE,
+    NJS_FS_STAT_BLOCKS,
+    NJS_FS_STAT_ATIME,
+    NJS_FS_STAT_BIRTHTIME,
+    NJS_FS_STAT_CTIME,
+    NJS_FS_STAT_MTIME,
+} njs_stat_prop_t;
+
+
+typedef njs_int_t (*njs_file_tree_walk_cb_t)(const char *, const struct stat *,
+     njs_ftw_type_t);
+
+
+static njs_int_t njs_fs_access(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t calltype);
+static njs_int_t njs_fs_mkdir(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t calltype);
+static njs_int_t njs_fs_read(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t calltype);
+static njs_int_t njs_fs_readdir(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t calltype);
+static njs_int_t njs_fs_realpath(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t calltype);

From richagaur586 at gmail.com  Wed Dec 22 05:35:14 2021
From: richagaur586 at gmail.com (Richa Gaur)
Date: Wed, 22 Dec 2021 11:05:14 +0530
Subject: Fwd: Delivery Status Notification (Failure)
In-Reply-To: <61c1e799.1c69fb81.f3b36.5e68.GMR@mx.google.com>
References: 
 CACcy-86tz2-tb6qdOntguTT99FCFri-P7W3bU2N8XdYvrtD9yA@mail.gmail.com
 <61c1e799.1c69fb81.f3b36.5e68.GMR@mx.google.com>
Message-ID: 

Hello,

I am using nginx as an L4 load balancer. I need to expose some stream level
metrics like throughput etc. I am using nginx-module-stream-sts
 for calculating metrics.

However, this module operates at nginx-stream-log-phase which is called
just before closing the connection. In case, if the underlying connection
is persistent and several requests are being made on the same connection,
the metrics are calculated at the end which does not depict the true
picture.

I tried looking at the stream-module code and based on preliminary
observations it looks like the stream phases are called for each stream
session and not for each tcp request (analogous to http request). So, even
if I register the handler at an earlier phase, let's say
nginx-stream-content-phase, it would be called only once in the connection
lifecycle instead of being called for every request.

Please let me know if my understanding is correct and if there is any
mechanism by which I can calculate metrics for each TCP request and show
real time data.

Any help would be appreciated.

Thanks and Regards,
Richa Gaur
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From xeioex at nginx.com  Wed Dec 22 16:55:05 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 22 Dec 2021 16:55:05 +0000
Subject: [njs] Fixed OpenSSL detection support when Address Sanitizer is
 enabled.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/31db917a742c
branches:  
changeset: 1773:31db917a742c
user:      Dmitry Volyntsev 
date:      Wed Dec 22 16:54:05 2021 +0000
description:
Fixed OpenSSL detection support when Address Sanitizer is enabled.

diffstat:

 auto/openssl |  4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diffs (14 lines):

diff -r 4d4657128baf -r 31db917a742c auto/openssl
--- a/auto/openssl	Tue Dec 21 17:42:26 2021 +0000
+++ b/auto/openssl	Wed Dec 22 16:54:05 2021 +0000
@@ -18,7 +18,9 @@ njs_feature_libs="-lcrypto"
 njs_feature_test="#include 
 
                   int main() {
-                      EVP_CIPHER_CTX_new();
+                      EVP_CIPHER_CTX  *ctx;
+                      ctx = EVP_CIPHER_CTX_new();
+                      EVP_CIPHER_CTX_free(ctx);
                       return 0;
                  }"
 . auto/feature

From xeioex at nginx.com  Wed Dec 22 16:55:07 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 22 Dec 2021 16:55:07 +0000
Subject: [njs] Added missed static declarations.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/9b8ec2a9e9b2
branches:  
changeset: 1774:9b8ec2a9e9b2
user:      Zhidao HONG 
date:      Wed Dec 22 10:50:16 2021 +0800
description:
Added missed static declarations.

diffstat:

 src/njs_module.c |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (21 lines):

diff -r 31db917a742c -r 9b8ec2a9e9b2 src/njs_module.c
--- a/src/njs_module.c	Wed Dec 22 16:54:05 2021 +0000
+++ b/src/njs_module.c	Wed Dec 22 10:50:16 2021 +0800
@@ -218,7 +218,7 @@ fail:
 }
 
 
-njs_int_t
+static njs_int_t
 njs_parser_module_lambda_after(njs_parser_t *parser, njs_lexer_token_t *token,
     njs_queue_link_t *current)
 {
@@ -261,7 +261,7 @@ njs_parser_module_lambda_after(njs_parse
 }
 
 
-njs_int_t
+static njs_int_t
 njs_parser_module_after(njs_parser_t *parser, njs_lexer_token_t *token,
     njs_queue_link_t *current)
 {

From xeioex at nginx.com  Wed Dec 22 16:55:09 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 22 Dec 2021 16:55:09 +0000
Subject: [njs] Added missed njs_parser_failed() in
 njs_parser_module_lambda_after().
Message-ID: 

details:   https://hg.nginx.org/njs/rev/6906e8a60d3a
branches:  
changeset: 1775:6906e8a60d3a
user:      Zhidao HONG 
date:      Wed Dec 22 10:50:16 2021 +0800
description:
Added missed njs_parser_failed() in njs_parser_module_lambda_after().

diffstat:

 src/njs_module.c |  2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diffs (12 lines):

diff -r 9b8ec2a9e9b2 -r 6906e8a60d3a src/njs_module.c
--- a/src/njs_module.c	Wed Dec 22 10:50:16 2021 +0800
+++ b/src/njs_module.c	Wed Dec 22 10:50:16 2021 +0800
@@ -245,6 +245,8 @@ njs_parser_module_lambda_after(njs_parse
         if (temp->text.start != NULL) {
             njs_mp_free(parser->vm->mem_pool, temp->text.start);
         }
+
+        return njs_parser_failed(parser);
     }
 
     module->function.args_offset = 1;

From xeioex at nginx.com  Wed Dec 22 16:55:11 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 22 Dec 2021 16:55:11 +0000
Subject: [njs] Fixed WebCrypto sign() and verify() methods with OpenSSL 3.0.1.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/dd2386828c1c
branches:  
changeset: 1776:dd2386828c1c
user:      Dmitry Volyntsev 
date:      Wed Dec 22 16:54:13 2021 +0000
description:
Fixed WebCrypto sign() and verify() methods with OpenSSL 3.0.1.

Since 3.0.1 EVP_DigestSignFinal() expects siglen to contain the
actual buffer size for signature buffer.

diffstat:

 external/njs_webcrypto.c |  2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diffs (12 lines):

diff -r 6906e8a60d3a -r dd2386828c1c external/njs_webcrypto.c
--- a/external/njs_webcrypto.c	Wed Dec 22 10:50:16 2021 +0800
+++ b/external/njs_webcrypto.c	Wed Dec 22 16:54:13 2021 +0000
@@ -2077,6 +2077,8 @@ njs_ext_sign(njs_vm_t *vm, njs_value_t *
             dst = (u_char *) &m[0];
         }
 
+        outlen = olen;
+
         ret = EVP_DigestSignFinal(mctx, dst, &outlen);
         if (njs_slow_path(ret <= 0 || olen != outlen)) {
             njs_webcrypto_error(vm, "EVP_DigestSignFinal() failed");

From xeioex at nginx.com  Wed Dec 22 16:55:13 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 22 Dec 2021 16:55:13 +0000
Subject: [njs] Tests: improved test262 output when a test fails.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/8843d18bee30
branches:  
changeset: 1777:8843d18bee30
user:      Dmitry Volyntsev 
date:      Wed Dec 22 16:54:27 2021 +0000
description:
Tests: improved test262 output when a test fails.

diffstat:

 test/report  |  1 +
 test/setup   |  3 +++
 test/test262 |  6 +++---
 3 files changed, 7 insertions(+), 3 deletions(-)

diffs (54 lines):

diff -r dd2386828c1c -r 8843d18bee30 test/report
--- a/test/report	Wed Dec 22 16:54:13 2021 +0000
+++ b/test/report	Wed Dec 22 16:54:27 2021 +0000
@@ -9,6 +9,7 @@ if [ $njs_passed -ne $njs_total ]; then
 fi
 
 if [ -n "$njs_failed_list" ]; then
+    printf "\n"
     for t in $njs_failed_list; do
         printf "$t FAILED\n"
     done
diff -r dd2386828c1c -r 8843d18bee30 test/setup
--- a/test/setup	Wed Dec 22 16:54:13 2021 +0000
+++ b/test/setup	Wed Dec 22 16:54:27 2021 +0000
@@ -17,6 +17,9 @@ passed() {
 failed() {
     njs_failed_list="$njs_failed_list $1"
     verbose " FAILED\n"
+    printf "$1:\n"
+    cat $2
+    printf "\n"
 }
 
 verbose "Test dir: $NJS_TEST_DIR\n"
diff -r dd2386828c1c -r 8843d18bee30 test/test262
--- a/test/test262	Wed Dec 22 16:54:13 2021 +0000
+++ b/test/test262	Wed Dec 22 16:54:27 2021 +0000
@@ -34,7 +34,7 @@ END
 
         elif [ $njs_async = yes ]; then
             if [ "$njs_out" != 'Test262:AsyncTestComplete' ]; then
-                failed $njs_test
+                failed $njs_test $njs_log
 
             else
                 passed $njs_test
@@ -42,7 +42,7 @@ END
 
         else
             if [ -n "$njs_out" ]; then
-                failed $njs_test
+                failed $njs_test $njs_log
 
             else
                 passed $njs_test
@@ -54,7 +54,7 @@ END
             passed $njs_test
 
         else
-            failed $njs_test
+            failed $njs_test $njs_log
         fi
     fi
 

From xeioex at nginx.com  Wed Dec 22 17:22:56 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 22 Dec 2021 17:22:56 +0000
Subject: [njs] Tests: fixed fs/methods.t.ts on RHEL8.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/82518ae9f209
branches:  
changeset: 1778:82518ae9f209
user:      Dmitry Volyntsev 
date:      Wed Dec 22 17:22:14 2021 +0000
description:
Tests: fixed fs/methods.t.ts on RHEL8.

diffstat:

 test/fs/methods.t.js |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (21 lines):

diff -r 8843d18bee30 -r 82518ae9f209 test/fs/methods.t.js
--- a/test/fs/methods.t.js	Wed Dec 22 16:54:27 2021 +0000
+++ b/test/fs/methods.t.js	Wed Dec 22 17:22:14 2021 +0000
@@ -135,7 +135,7 @@ let read_tests = () => [
 
           return data.compare(params.expected) == 0;
       } },
-    { args: ["/proc/cpuinfo"], slice:[0,9], expected: Buffer.from("processor"),
+    { args: ["/proc/cpuinfo"],
       check: (data, params) => {
 
           if (data.error) {
@@ -147,7 +147,7 @@ let read_tests = () => [
               return true;
           }
 
-          return data.compare(params.expected) == 0;
+          return /processor/.test(data);
       } },
 ];
 

From xeioex at nginx.com  Thu Dec 23 13:31:45 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 23 Dec 2021 13:31:45 +0000
Subject: [njs] Improved njs_ftw().
Message-ID: 

details:   https://hg.nginx.org/njs/rev/9e1fd062a1d8
branches:  
changeset: 1779:9e1fd062a1d8
user:      Dmitry Volyntsev 
date:      Thu Dec 23 13:30:44 2021 +0000
description:
Improved njs_ftw().

Replacing strcpy() with memcpy() as the former is considered
insecure.

Found by Clang static analyzer.

diffstat:

 external/njs_fs.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r 82518ae9f209 -r 9e1fd062a1d8 external/njs_fs.c
--- a/external/njs_fs.c	Wed Dec 22 17:22:14 2021 +0000
+++ b/external/njs_fs.c	Thu Dec 23 13:30:44 2021 +0000
@@ -2153,7 +2153,7 @@ njs_ftw(char *path, njs_file_tree_walk_c
             }
 
             path[base] = '/';
-            strcpy(path + base + 1, d_name);
+            memcpy(&path[base + 1], d_name, length + sizeof("\0"));
 
             if (fd_limit != 0) {
                 ret = njs_ftw(path, cb, fd_limit - 1, flags, &trace);

From mdounin at mdounin.ru  Thu Dec 23 14:52:50 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 23 Dec 2021 14:52:50 +0000
Subject: [nginx] Contrib: vim syntax,
 update core and 3rd party module directives.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/76aea0ad78e5
branches:  
changeset: 7976:76aea0ad78e5
user:      Gena Makhomed 
date:      Mon Dec 20 20:02:48 2021 +0200
description:
Contrib: vim syntax, update core and 3rd party module directives.

diffstat:

 contrib/vim/syntax/nginx.vim |  40 +++++++++++++++++++++++++++++++++++++++-
 1 files changed, 39 insertions(+), 1 deletions(-)

diffs (162 lines):

diff -r a7a77549265e -r 76aea0ad78e5 contrib/vim/syntax/nginx.vim
--- a/contrib/vim/syntax/nginx.vim	Thu Nov 25 22:02:10 2021 +0300
+++ b/contrib/vim/syntax/nginx.vim	Mon Dec 20 20:02:48 2021 +0200
@@ -152,6 +152,7 @@ syn keyword ngxDirective contained auth_
 syn keyword ngxDirective contained auth_jwt_key_file
 syn keyword ngxDirective contained auth_jwt_key_request
 syn keyword ngxDirective contained auth_jwt_leeway
+syn keyword ngxDirective contained auth_jwt_require
 syn keyword ngxDirective contained auth_jwt_type
 syn keyword ngxDirective contained auth_request
 syn keyword ngxDirective contained auth_request_set
@@ -335,6 +336,10 @@ syn keyword ngxDirective contained ip_ha
 syn keyword ngxDirective contained js_access
 syn keyword ngxDirective contained js_body_filter
 syn keyword ngxDirective contained js_content
+syn keyword ngxDirective contained js_fetch_ciphers
+syn keyword ngxDirective contained js_fetch_protocols
+syn keyword ngxDirective contained js_fetch_trusted_certificate
+syn keyword ngxDirective contained js_fetch_verify_depth
 syn keyword ngxDirective contained js_filter
 syn keyword ngxDirective contained js_header_filter
 syn keyword ngxDirective contained js_import
@@ -402,6 +407,7 @@ syn keyword ngxDirective contained mp4_b
 syn keyword ngxDirective contained mp4_limit_rate
 syn keyword ngxDirective contained mp4_limit_rate_after
 syn keyword ngxDirective contained mp4_max_buffer_size
+syn keyword ngxDirective contained mp4_start_key_frame
 syn keyword ngxDirective contained msie_padding
 syn keyword ngxDirective contained msie_refresh
 syn keyword ngxDirective contained multi_accept
@@ -458,6 +464,7 @@ syn keyword ngxDirective contained proxy
 syn keyword ngxDirective contained proxy_cookie_path
 syn keyword ngxDirective contained proxy_download_rate
 syn keyword ngxDirective contained proxy_force_ranges
+syn keyword ngxDirective contained proxy_half_close
 syn keyword ngxDirective contained proxy_headers_hash_bucket_size
 syn keyword ngxDirective contained proxy_headers_hash_max_size
 syn keyword ngxDirective contained proxy_hide_header
@@ -597,6 +604,7 @@ syn keyword ngxDirective contained ssi_m
 syn keyword ngxDirective contained ssi_silent_errors
 syn keyword ngxDirective contained ssi_types
 syn keyword ngxDirective contained ssi_value_length
+syn keyword ngxDirective contained ssl_alpn
 syn keyword ngxDirective contained ssl_buffer_size
 syn keyword ngxDirective contained ssl_certificate
 syn keyword ngxDirective contained ssl_certificate_key
@@ -788,11 +796,15 @@ syn keyword ngxDirectiveThirdParty conta
 syn keyword ngxDirectiveThirdParty contained auth_gss
 syn keyword ngxDirectiveThirdParty contained auth_gss_allow_basic_fallback
 syn keyword ngxDirectiveThirdParty contained auth_gss_authorized_principal
+syn keyword ngxDirectiveThirdParty contained auth_gss_authorized_principal_regex
+syn keyword ngxDirectiveThirdParty contained auth_gss_constrained_delegation
+syn keyword ngxDirectiveThirdParty contained auth_gss_delegate_credentials
 syn keyword ngxDirectiveThirdParty contained auth_gss_force_realm
 syn keyword ngxDirectiveThirdParty contained auth_gss_format_full
 syn keyword ngxDirectiveThirdParty contained auth_gss_keytab
 syn keyword ngxDirectiveThirdParty contained auth_gss_map_to_local
 syn keyword ngxDirectiveThirdParty contained auth_gss_realm
+syn keyword ngxDirectiveThirdParty contained auth_gss_service_ccache
 syn keyword ngxDirectiveThirdParty contained auth_gss_service_name
 
 " LDAP Authentication
@@ -969,7 +981,6 @@ syn keyword ngxDirectiveThirdParty conta
 syn keyword ngxDirectiveThirdParty contained fancyindex_hide_symlinks
 syn keyword ngxDirectiveThirdParty contained fancyindex_ignore
 syn keyword ngxDirectiveThirdParty contained fancyindex_localtime
-syn keyword ngxDirectiveThirdParty contained fancyindex_name_length
 syn keyword ngxDirectiveThirdParty contained fancyindex_show_dotfiles
 syn keyword ngxDirectiveThirdParty contained fancyindex_show_path
 syn keyword ngxDirectiveThirdParty contained fancyindex_time_format
@@ -1059,7 +1070,9 @@ syn keyword ngxDirectiveThirdParty conta
 syn keyword ngxDirectiveThirdParty contained nchan_pubsub
 syn keyword ngxDirectiveThirdParty contained nchan_pubsub_channel_id
 syn keyword ngxDirectiveThirdParty contained nchan_pubsub_location
+syn keyword ngxDirectiveThirdParty contained nchan_redis_cluster_check_interval
 syn keyword ngxDirectiveThirdParty contained nchan_redis_connect_timeout
+syn keyword ngxDirectiveThirdParty contained nchan_redis_discovered_ip_range_blacklist
 syn keyword ngxDirectiveThirdParty contained nchan_redis_fakesub_timer_interval
 syn keyword ngxDirectiveThirdParty contained nchan_redis_idle_channel_cache_timeout
 syn keyword ngxDirectiveThirdParty contained nchan_redis_namespace
@@ -1067,12 +1080,29 @@ syn keyword ngxDirectiveThirdParty conta
 syn keyword ngxDirectiveThirdParty contained nchan_redis_optimize_target
 syn keyword ngxDirectiveThirdParty contained nchan_redis_pass
 syn keyword ngxDirectiveThirdParty contained nchan_redis_pass_inheritable
+syn keyword ngxDirectiveThirdParty contained nchan_redis_password
 syn keyword ngxDirectiveThirdParty contained nchan_redis_ping_interval
 syn keyword ngxDirectiveThirdParty contained nchan_redis_publish_msgpacked_max_size
 syn keyword ngxDirectiveThirdParty contained nchan_redis_server
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl_ciphers
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl_client_certificate
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl_client_certificate_key
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl_server_name
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl_trusted_certificate
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl_trusted_certificate_path
+syn keyword ngxDirectiveThirdParty contained nchan_redis_ssl_verify_certificate
 syn keyword ngxDirectiveThirdParty contained nchan_redis_storage_mode
 syn keyword ngxDirectiveThirdParty contained nchan_redis_subscribe_weights
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls_ciphers
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls_client_certificate
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls_server_name
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls_trusted_certificate
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls_trusted_certificate_path
+syn keyword ngxDirectiveThirdParty contained nchan_redis_tls_verify_certificate
 syn keyword ngxDirectiveThirdParty contained nchan_redis_url
+syn keyword ngxDirectiveThirdParty contained nchan_redis_username
 syn keyword ngxDirectiveThirdParty contained nchan_redis_wait_after_connecting
 syn keyword ngxDirectiveThirdParty contained nchan_shared_memory_size
 syn keyword ngxDirectiveThirdParty contained nchan_storage_engine
@@ -1385,6 +1415,7 @@ syn keyword ngxDirectiveThirdParty conta
 syn keyword ngxDirectiveThirdParty contained lua_socket_send_lowat
 syn keyword ngxDirectiveThirdParty contained lua_socket_send_timeout
 syn keyword ngxDirectiveThirdParty contained lua_ssl_ciphers
+syn keyword ngxDirectiveThirdParty contained lua_ssl_conf_command
 syn keyword ngxDirectiveThirdParty contained lua_ssl_crl
 syn keyword ngxDirectiveThirdParty contained lua_ssl_protocols
 syn keyword ngxDirectiveThirdParty contained lua_ssl_trusted_certificate
@@ -1392,6 +1423,7 @@ syn keyword ngxDirectiveThirdParty conta
 syn keyword ngxDirectiveThirdParty contained lua_thread_cache_max_entries
 syn keyword ngxDirectiveThirdParty contained lua_transform_underscores_in_response_headers
 syn keyword ngxDirectiveThirdParty contained lua_use_default_type
+syn keyword ngxDirectiveThirdParty contained lua_worker_thread_vm_pool_size
 syn keyword ngxDirectiveThirdParty contained rewrite_by_lua
 syn keyword ngxDirectiveThirdParty contained rewrite_by_lua_block
 syn keyword ngxDirectiveThirdParty contained rewrite_by_lua_file
@@ -1401,6 +1433,8 @@ syn keyword ngxDirectiveThirdParty conta
 syn keyword ngxDirectiveThirdParty contained set_by_lua_file
 syn keyword ngxDirectiveThirdParty contained ssl_certificate_by_lua_block
 syn keyword ngxDirectiveThirdParty contained ssl_certificate_by_lua_file
+syn keyword ngxDirectiveThirdParty contained ssl_client_hello_by_lua_block
+syn keyword ngxDirectiveThirdParty contained ssl_client_hello_by_lua_file
 syn keyword ngxDirectiveThirdParty contained ssl_session_fetch_by_lua_block
 syn keyword ngxDirectiveThirdParty contained ssl_session_fetch_by_lua_file
 syn keyword ngxDirectiveThirdParty contained ssl_session_store_by_lua_block
@@ -1719,15 +1753,18 @@ syn keyword ngxDirectiveThirdParty conta
 syn keyword ngxDirectiveThirdParty contained set_base32_padding
 syn keyword ngxDirectiveThirdParty contained set_decode_base32
 syn keyword ngxDirectiveThirdParty contained set_decode_base64
+syn keyword ngxDirectiveThirdParty contained set_decode_base64url
 syn keyword ngxDirectiveThirdParty contained set_decode_hex
 syn keyword ngxDirectiveThirdParty contained set_encode_base32
 syn keyword ngxDirectiveThirdParty contained set_encode_base64
+syn keyword ngxDirectiveThirdParty contained set_encode_base64url
 syn keyword ngxDirectiveThirdParty contained set_encode_hex
 syn keyword ngxDirectiveThirdParty contained set_escape_uri
 syn keyword ngxDirectiveThirdParty contained set_formatted_gmt_time
 syn keyword ngxDirectiveThirdParty contained set_formatted_local_time
 syn keyword ngxDirectiveThirdParty contained set_hashed_upstream
 syn keyword ngxDirectiveThirdParty contained set_hmac_sha1
+syn keyword ngxDirectiveThirdParty contained set_hmac_sha256
 syn keyword ngxDirectiveThirdParty contained set_if_empty
 syn keyword ngxDirectiveThirdParty contained set_local_today
 syn keyword ngxDirectiveThirdParty contained set_misc_base32_padding
@@ -1849,6 +1886,7 @@ syn keyword ngxDirectiveThirdParty conta
 syn keyword ngxDirectiveThirdParty contained vod_open_file_thread_pool
 syn keyword ngxDirectiveThirdParty contained vod_output_buffer_pool
 syn keyword ngxDirectiveThirdParty contained vod_parse_hdlr_name
+syn keyword ngxDirectiveThirdParty contained vod_parse_udta_name
 syn keyword ngxDirectiveThirdParty contained vod_path_response_postfix
 syn keyword ngxDirectiveThirdParty contained vod_path_response_prefix
 syn keyword ngxDirectiveThirdParty contained vod_performance_counters

From mdounin at mdounin.ru  Thu Dec 23 14:52:56 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 23 Dec 2021 17:52:56 +0300
Subject: [PATCH] Contrib: vim syntax,
 update core and 3rd party module directives.
In-Reply-To: <48c37eaa-a317-eb92-d725-f7e8071664bc@csdoc.com>
References: <48c37eaa-a317-eb92-d725-f7e8071664bc@csdoc.com>
Message-ID: 

Hello!

On Mon, Dec 20, 2021 at 08:05:54PM +0200, Gena Makhomed wrote:

> # HG changeset patch
> # User Gena Makhomed 
> # Date 1640023368 -7200
> #      Mon Dec 20 20:02:48 2021 +0200
> # Node ID 4719937b5dcbc5345e1a4cbf4bf84d2da4e316f8
> # Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
> Contrib: vim syntax, update core and 3rd party module directives.
> 

[...]

> +syn keyword ngxDirectiveThirdParty contained auth_gss_delegate_credentials
>   syn keyword ngxDirectiveThirdParty contained auth_gss_force_realm
>   syn keyword ngxDirectiveThirdParty contained auth_gss_format_full
>   syn keyword ngxDirectiveThirdParty contained auth_gss_keytab
>   syn keyword ngxDirectiveThirdParty contained auth_gss_map_to_local
>   syn keyword ngxDirectiveThirdParty contained auth_gss_realm
> +syn keyword ngxDirectiveThirdParty contained auth_gss_service_ccache
>   syn keyword ngxDirectiveThirdParty contained auth_gss_service_name
> 
> +

Looks like a stray empty line.  Otherwise looks good, committed 
with the empty line removed.

Thanks.

-- 
Maxim Dounin
http://mdounin.ru/

From guillaume.bilic at gmail.com  Thu Dec 23 16:19:26 2021
From: guillaume.bilic at gmail.com (Guillaume Bilic)
Date: Thu, 23 Dec 2021 17:19:26 +0100
Subject: [nginx-quic] fastcgi cookie param is overwritten resulting in getting
 only last cookie
Message-ID: 

Hi all,



Using nginx-quic (1.21.4), cookies are parsed individually by http3 code :



*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse literal done
"number1=this+is+the+first+one"*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field lri done
static[5] "number1=this+is+the+first+one"*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 static[5] lookup
"cookie":""*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field
representation done*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 header: "cookie:
number1=this+is+the+first+one"*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field
representation*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field lri*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse prefix int 5*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse prefix int 24*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse literal huff:1,
len:24*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse literal done
"number2=this+is+the+second+one"*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field lri done
static[5] "number2=this+is+the+second+one"*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 static[5] lookup
"cookie":""*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field
representation done*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 header: "cookie:
number2=this+is+the+second+one"*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field
representation*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field lri*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse prefix int 5*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse prefix int 23*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse literal huff:1,
len:23*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse literal done
"number3=this+is+the+third+one"*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field lri done
static[5] "number3=this+is+the+third+one"*

2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 static[5] lookup
"cookie":""

2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse field representation
done

2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 parse headers done

2021/12/23 14:29:37 [debug] 32322#0: *3576 http3 header: "cookie:
number3=this+is+the+third+one ?





But then the fastcgi param HTTP_COOKIE is passed for each cookie, resulting
in overwriting it and keeping only the last one :



*2021/12/23 14:29:37 [debug] 32322#0: *3576 fastcgi param: "HTTP_COOKIE:
number1=this+is+the+first+one"*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 fastcgi param: "HTTP_COOKIE:
number2=this+is+the+second+one"*

*2021/12/23 14:29:37 [debug] 32322#0: *3576 fastcgi param: "HTTP_COOKIE:
number3=this+is+the+third+one ?*



The HTTP_COOKIE param should be the whole cookie header.

Http2 code handles cookie header in a dedicated function
? ngx_http_v2_construct_cookie_header ? and then processes other headers.

There doesn?t seem to be the case of http3 code which process cookie the
same way of others headers.



Regards,

Guillaume
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From ru at nginx.com  Thu Dec 23 22:49:23 2021
From: ru at nginx.com (Ruslan Ermilov)
Date: Thu, 23 Dec 2021 22:49:23 +0000
Subject: [nginx] Moved Huffman coding out of HTTP/2.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/336084ff943b
branches:  
changeset: 7977:336084ff943b
user:      Ruslan Ermilov 
date:      Tue Dec 21 07:54:16 2021 +0300
description:
Moved Huffman coding out of HTTP/2.

ngx_http_v2_huff_decode.c and ngx_http_v2_huff_encode.c are renamed
to ngx_http_huff_decode.c and ngx_http_huff_encode.c.

diffstat:

 auto/modules                            |     7 +-
 auto/sources                            |     3 +
 src/http/modules/ngx_http_grpc_module.c |    16 +-
 src/http/ngx_http.h                     |     8 +
 src/http/ngx_http_huff_decode.c         |  2714 +++++++++++++++++++++++++++++++
 src/http/ngx_http_huff_encode.c         |   254 ++
 src/http/v2/ngx_http_v2.c               |     8 +-
 src/http/v2/ngx_http_v2.h               |     6 -
 src/http/v2/ngx_http_v2_encode.c        |     2 +-
 src/http/v2/ngx_http_v2_huff_decode.c   |  2714 -------------------------------
 src/http/v2/ngx_http_v2_huff_encode.c   |   254 --
 11 files changed, 2997 insertions(+), 2989 deletions(-)

diffs (truncated from 6083 to 1000 lines):

diff -r 76aea0ad78e5 -r 336084ff943b auto/modules
--- a/auto/modules	Mon Dec 20 20:02:48 2021 +0200
+++ b/auto/modules	Tue Dec 21 07:54:16 2021 +0300
@@ -102,6 +102,11 @@ if [ $HTTP = YES ]; then
     fi
 
 
+    if [ $HTTP_V2 = YES ]; then
+        HTTP_SRCS="$HTTP_SRCS $HTTP_HUFF_SRCS"
+    fi
+
+
     # the module order is important
     #     ngx_http_static_module
     #     ngx_http_gzip_static_module
@@ -414,8 +419,6 @@ if [ $HTTP = YES ]; then
         ngx_module_srcs="src/http/v2/ngx_http_v2.c \
                          src/http/v2/ngx_http_v2_table.c \
                          src/http/v2/ngx_http_v2_encode.c \
-                         src/http/v2/ngx_http_v2_huff_decode.c \
-                         src/http/v2/ngx_http_v2_huff_encode.c \
                          src/http/v2/ngx_http_v2_module.c"
         ngx_module_libs=
         ngx_module_link=$HTTP_V2
diff -r 76aea0ad78e5 -r 336084ff943b auto/sources
--- a/auto/sources	Mon Dec 20 20:02:48 2021 +0200
+++ b/auto/sources	Tue Dec 21 07:54:16 2021 +0300
@@ -255,3 +255,6 @@ NGX_WIN32_RC="src/os/win32/nginx.rc"
 
 
 HTTP_FILE_CACHE_SRCS=src/http/ngx_http_file_cache.c
+
+HTTP_HUFF_SRCS="src/http/ngx_http_huff_decode.c
+                src/http/ngx_http_huff_encode.c"
diff -r 76aea0ad78e5 -r 336084ff943b src/http/modules/ngx_http_grpc_module.c
--- a/src/http/modules/ngx_http_grpc_module.c	Mon Dec 20 20:02:48 2021 +0200
+++ b/src/http/modules/ngx_http_grpc_module.c	Tue Dec 21 07:54:16 2021 +0300
@@ -3180,10 +3180,10 @@ ngx_http_grpc_parse_fragment(ngx_http_re
             ctx->field_rest -= size;
 
             if (ctx->field_huffman) {
-                if (ngx_http_v2_huff_decode(&ctx->field_state, p, size,
-                                            &ctx->field_end,
-                                            ctx->field_rest == 0,
-                                            r->connection->log)
+                if (ngx_http_huff_decode(&ctx->field_state, p, size,
+                                         &ctx->field_end,
+                                         ctx->field_rest == 0,
+                                         r->connection->log)
                     != NGX_OK)
                 {
                     ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
@@ -3289,10 +3289,10 @@ ngx_http_grpc_parse_fragment(ngx_http_re
             ctx->field_rest -= size;
 
             if (ctx->field_huffman) {
-                if (ngx_http_v2_huff_decode(&ctx->field_state, p, size,
-                                            &ctx->field_end,
-                                            ctx->field_rest == 0,
-                                            r->connection->log)
+                if (ngx_http_huff_decode(&ctx->field_state, p, size,
+                                         &ctx->field_end,
+                                         ctx->field_rest == 0,
+                                         r->connection->log)
                     != NGX_OK)
                 {
                     ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
diff -r 76aea0ad78e5 -r 336084ff943b src/http/ngx_http.h
--- a/src/http/ngx_http.h	Mon Dec 20 20:02:48 2021 +0200
+++ b/src/http/ngx_http.h	Tue Dec 21 07:54:16 2021 +0300
@@ -167,6 +167,14 @@ ngx_uint_t  ngx_http_degraded(ngx_http_r
 #endif
 
 
+#if (NGX_HTTP_V2)
+ngx_int_t ngx_http_huff_decode(u_char *state, u_char *src, size_t len,
+    u_char **dst, ngx_uint_t last, ngx_log_t *log);
+size_t ngx_http_huff_encode(u_char *src, size_t len, u_char *dst,
+    ngx_uint_t lower);
+#endif
+
+
 extern ngx_module_t  ngx_http_module;
 
 extern ngx_str_t  ngx_http_html_default_types[];
diff -r 76aea0ad78e5 -r 336084ff943b src/http/ngx_http_huff_decode.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/src/http/ngx_http_huff_decode.c	Tue Dec 21 07:54:16 2021 +0300
@@ -0,0 +1,2714 @@
+
+/*
+ * Copyright (C) Nginx, Inc.
+ * Copyright (C) Valentin V. Bartenev
+ */
+
+
+#include 
+#include 
+#include 
+
+
+typedef struct {
+    u_char  next;
+    u_char  emit;
+    u_char  sym;
+    u_char  ending;
+} ngx_http_huff_decode_code_t;
+
+
+static ngx_inline ngx_int_t ngx_http_huff_decode_bits(u_char *state,
+    u_char *ending, ngx_uint_t bits, u_char **dst);
+
+
+static ngx_http_huff_decode_code_t  ngx_http_huff_decode_codes[256][16] =
+{
+    /* 0 */
+    {
+        {0x04, 0x00, 0x00, 0x00}, {0x05, 0x00, 0x00, 0x00},
+        {0x07, 0x00, 0x00, 0x00}, {0x08, 0x00, 0x00, 0x00},
+        {0x0b, 0x00, 0x00, 0x00}, {0x0c, 0x00, 0x00, 0x00},
+        {0x10, 0x00, 0x00, 0x00}, {0x13, 0x00, 0x00, 0x00},
+        {0x19, 0x00, 0x00, 0x00}, {0x1c, 0x00, 0x00, 0x00},
+        {0x20, 0x00, 0x00, 0x00}, {0x23, 0x00, 0x00, 0x00},
+        {0x2a, 0x00, 0x00, 0x00}, {0x31, 0x00, 0x00, 0x00},
+        {0x39, 0x00, 0x00, 0x00}, {0x40, 0x00, 0x00, 0x01}
+    },
+    {
+        {0x00, 0x01, 0x30, 0x01}, {0x00, 0x01, 0x31, 0x01},
+        {0x00, 0x01, 0x32, 0x01}, {0x00, 0x01, 0x61, 0x01},
+        {0x00, 0x01, 0x63, 0x01}, {0x00, 0x01, 0x65, 0x01},
+        {0x00, 0x01, 0x69, 0x01}, {0x00, 0x01, 0x6f, 0x01},
+        {0x00, 0x01, 0x73, 0x01}, {0x00, 0x01, 0x74, 0x01},
+        {0x0d, 0x00, 0x00, 0x00}, {0x0e, 0x00, 0x00, 0x00},
+        {0x11, 0x00, 0x00, 0x00}, {0x12, 0x00, 0x00, 0x00},
+        {0x14, 0x00, 0x00, 0x00}, {0x15, 0x00, 0x00, 0x00}
+    },
+    {
+        {0x01, 0x01, 0x30, 0x00}, {0x16, 0x01, 0x30, 0x01},
+        {0x01, 0x01, 0x31, 0x00}, {0x16, 0x01, 0x31, 0x01},
+        {0x01, 0x01, 0x32, 0x00}, {0x16, 0x01, 0x32, 0x01},
+        {0x01, 0x01, 0x61, 0x00}, {0x16, 0x01, 0x61, 0x01},
+        {0x01, 0x01, 0x63, 0x00}, {0x16, 0x01, 0x63, 0x01},
+        {0x01, 0x01, 0x65, 0x00}, {0x16, 0x01, 0x65, 0x01},
+        {0x01, 0x01, 0x69, 0x00}, {0x16, 0x01, 0x69, 0x01},
+        {0x01, 0x01, 0x6f, 0x00}, {0x16, 0x01, 0x6f, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x30, 0x00}, {0x09, 0x01, 0x30, 0x00},
+        {0x17, 0x01, 0x30, 0x00}, {0x28, 0x01, 0x30, 0x01},
+        {0x02, 0x01, 0x31, 0x00}, {0x09, 0x01, 0x31, 0x00},
+        {0x17, 0x01, 0x31, 0x00}, {0x28, 0x01, 0x31, 0x01},
+        {0x02, 0x01, 0x32, 0x00}, {0x09, 0x01, 0x32, 0x00},
+        {0x17, 0x01, 0x32, 0x00}, {0x28, 0x01, 0x32, 0x01},
+        {0x02, 0x01, 0x61, 0x00}, {0x09, 0x01, 0x61, 0x00},
+        {0x17, 0x01, 0x61, 0x00}, {0x28, 0x01, 0x61, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x30, 0x00}, {0x06, 0x01, 0x30, 0x00},
+        {0x0a, 0x01, 0x30, 0x00}, {0x0f, 0x01, 0x30, 0x00},
+        {0x18, 0x01, 0x30, 0x00}, {0x1f, 0x01, 0x30, 0x00},
+        {0x29, 0x01, 0x30, 0x00}, {0x38, 0x01, 0x30, 0x01},
+        {0x03, 0x01, 0x31, 0x00}, {0x06, 0x01, 0x31, 0x00},
+        {0x0a, 0x01, 0x31, 0x00}, {0x0f, 0x01, 0x31, 0x00},
+        {0x18, 0x01, 0x31, 0x00}, {0x1f, 0x01, 0x31, 0x00},
+        {0x29, 0x01, 0x31, 0x00}, {0x38, 0x01, 0x31, 0x01}
+    },
+    /* 5 */
+    {
+        {0x03, 0x01, 0x32, 0x00}, {0x06, 0x01, 0x32, 0x00},
+        {0x0a, 0x01, 0x32, 0x00}, {0x0f, 0x01, 0x32, 0x00},
+        {0x18, 0x01, 0x32, 0x00}, {0x1f, 0x01, 0x32, 0x00},
+        {0x29, 0x01, 0x32, 0x00}, {0x38, 0x01, 0x32, 0x01},
+        {0x03, 0x01, 0x61, 0x00}, {0x06, 0x01, 0x61, 0x00},
+        {0x0a, 0x01, 0x61, 0x00}, {0x0f, 0x01, 0x61, 0x00},
+        {0x18, 0x01, 0x61, 0x00}, {0x1f, 0x01, 0x61, 0x00},
+        {0x29, 0x01, 0x61, 0x00}, {0x38, 0x01, 0x61, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x63, 0x00}, {0x09, 0x01, 0x63, 0x00},
+        {0x17, 0x01, 0x63, 0x00}, {0x28, 0x01, 0x63, 0x01},
+        {0x02, 0x01, 0x65, 0x00}, {0x09, 0x01, 0x65, 0x00},
+        {0x17, 0x01, 0x65, 0x00}, {0x28, 0x01, 0x65, 0x01},
+        {0x02, 0x01, 0x69, 0x00}, {0x09, 0x01, 0x69, 0x00},
+        {0x17, 0x01, 0x69, 0x00}, {0x28, 0x01, 0x69, 0x01},
+        {0x02, 0x01, 0x6f, 0x00}, {0x09, 0x01, 0x6f, 0x00},
+        {0x17, 0x01, 0x6f, 0x00}, {0x28, 0x01, 0x6f, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x63, 0x00}, {0x06, 0x01, 0x63, 0x00},
+        {0x0a, 0x01, 0x63, 0x00}, {0x0f, 0x01, 0x63, 0x00},
+        {0x18, 0x01, 0x63, 0x00}, {0x1f, 0x01, 0x63, 0x00},
+        {0x29, 0x01, 0x63, 0x00}, {0x38, 0x01, 0x63, 0x01},
+        {0x03, 0x01, 0x65, 0x00}, {0x06, 0x01, 0x65, 0x00},
+        {0x0a, 0x01, 0x65, 0x00}, {0x0f, 0x01, 0x65, 0x00},
+        {0x18, 0x01, 0x65, 0x00}, {0x1f, 0x01, 0x65, 0x00},
+        {0x29, 0x01, 0x65, 0x00}, {0x38, 0x01, 0x65, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x69, 0x00}, {0x06, 0x01, 0x69, 0x00},
+        {0x0a, 0x01, 0x69, 0x00}, {0x0f, 0x01, 0x69, 0x00},
+        {0x18, 0x01, 0x69, 0x00}, {0x1f, 0x01, 0x69, 0x00},
+        {0x29, 0x01, 0x69, 0x00}, {0x38, 0x01, 0x69, 0x01},
+        {0x03, 0x01, 0x6f, 0x00}, {0x06, 0x01, 0x6f, 0x00},
+        {0x0a, 0x01, 0x6f, 0x00}, {0x0f, 0x01, 0x6f, 0x00},
+        {0x18, 0x01, 0x6f, 0x00}, {0x1f, 0x01, 0x6f, 0x00},
+        {0x29, 0x01, 0x6f, 0x00}, {0x38, 0x01, 0x6f, 0x01}
+    },
+    {
+        {0x01, 0x01, 0x73, 0x00}, {0x16, 0x01, 0x73, 0x01},
+        {0x01, 0x01, 0x74, 0x00}, {0x16, 0x01, 0x74, 0x01},
+        {0x00, 0x01, 0x20, 0x01}, {0x00, 0x01, 0x25, 0x01},
+        {0x00, 0x01, 0x2d, 0x01}, {0x00, 0x01, 0x2e, 0x01},
+        {0x00, 0x01, 0x2f, 0x01}, {0x00, 0x01, 0x33, 0x01},
+        {0x00, 0x01, 0x34, 0x01}, {0x00, 0x01, 0x35, 0x01},
+        {0x00, 0x01, 0x36, 0x01}, {0x00, 0x01, 0x37, 0x01},
+        {0x00, 0x01, 0x38, 0x01}, {0x00, 0x01, 0x39, 0x01}
+    },
+    /* 10 */
+    {
+        {0x02, 0x01, 0x73, 0x00}, {0x09, 0x01, 0x73, 0x00},
+        {0x17, 0x01, 0x73, 0x00}, {0x28, 0x01, 0x73, 0x01},
+        {0x02, 0x01, 0x74, 0x00}, {0x09, 0x01, 0x74, 0x00},
+        {0x17, 0x01, 0x74, 0x00}, {0x28, 0x01, 0x74, 0x01},
+        {0x01, 0x01, 0x20, 0x00}, {0x16, 0x01, 0x20, 0x01},
+        {0x01, 0x01, 0x25, 0x00}, {0x16, 0x01, 0x25, 0x01},
+        {0x01, 0x01, 0x2d, 0x00}, {0x16, 0x01, 0x2d, 0x01},
+        {0x01, 0x01, 0x2e, 0x00}, {0x16, 0x01, 0x2e, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x73, 0x00}, {0x06, 0x01, 0x73, 0x00},
+        {0x0a, 0x01, 0x73, 0x00}, {0x0f, 0x01, 0x73, 0x00},
+        {0x18, 0x01, 0x73, 0x00}, {0x1f, 0x01, 0x73, 0x00},
+        {0x29, 0x01, 0x73, 0x00}, {0x38, 0x01, 0x73, 0x01},
+        {0x03, 0x01, 0x74, 0x00}, {0x06, 0x01, 0x74, 0x00},
+        {0x0a, 0x01, 0x74, 0x00}, {0x0f, 0x01, 0x74, 0x00},
+        {0x18, 0x01, 0x74, 0x00}, {0x1f, 0x01, 0x74, 0x00},
+        {0x29, 0x01, 0x74, 0x00}, {0x38, 0x01, 0x74, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x20, 0x00}, {0x09, 0x01, 0x20, 0x00},
+        {0x17, 0x01, 0x20, 0x00}, {0x28, 0x01, 0x20, 0x01},
+        {0x02, 0x01, 0x25, 0x00}, {0x09, 0x01, 0x25, 0x00},
+        {0x17, 0x01, 0x25, 0x00}, {0x28, 0x01, 0x25, 0x01},
+        {0x02, 0x01, 0x2d, 0x00}, {0x09, 0x01, 0x2d, 0x00},
+        {0x17, 0x01, 0x2d, 0x00}, {0x28, 0x01, 0x2d, 0x01},
+        {0x02, 0x01, 0x2e, 0x00}, {0x09, 0x01, 0x2e, 0x00},
+        {0x17, 0x01, 0x2e, 0x00}, {0x28, 0x01, 0x2e, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x20, 0x00}, {0x06, 0x01, 0x20, 0x00},
+        {0x0a, 0x01, 0x20, 0x00}, {0x0f, 0x01, 0x20, 0x00},
+        {0x18, 0x01, 0x20, 0x00}, {0x1f, 0x01, 0x20, 0x00},
+        {0x29, 0x01, 0x20, 0x00}, {0x38, 0x01, 0x20, 0x01},
+        {0x03, 0x01, 0x25, 0x00}, {0x06, 0x01, 0x25, 0x00},
+        {0x0a, 0x01, 0x25, 0x00}, {0x0f, 0x01, 0x25, 0x00},
+        {0x18, 0x01, 0x25, 0x00}, {0x1f, 0x01, 0x25, 0x00},
+        {0x29, 0x01, 0x25, 0x00}, {0x38, 0x01, 0x25, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x2d, 0x00}, {0x06, 0x01, 0x2d, 0x00},
+        {0x0a, 0x01, 0x2d, 0x00}, {0x0f, 0x01, 0x2d, 0x00},
+        {0x18, 0x01, 0x2d, 0x00}, {0x1f, 0x01, 0x2d, 0x00},
+        {0x29, 0x01, 0x2d, 0x00}, {0x38, 0x01, 0x2d, 0x01},
+        {0x03, 0x01, 0x2e, 0x00}, {0x06, 0x01, 0x2e, 0x00},
+        {0x0a, 0x01, 0x2e, 0x00}, {0x0f, 0x01, 0x2e, 0x00},
+        {0x18, 0x01, 0x2e, 0x00}, {0x1f, 0x01, 0x2e, 0x00},
+        {0x29, 0x01, 0x2e, 0x00}, {0x38, 0x01, 0x2e, 0x01}
+    },
+    /* 15 */
+    {
+        {0x01, 0x01, 0x2f, 0x00}, {0x16, 0x01, 0x2f, 0x01},
+        {0x01, 0x01, 0x33, 0x00}, {0x16, 0x01, 0x33, 0x01},
+        {0x01, 0x01, 0x34, 0x00}, {0x16, 0x01, 0x34, 0x01},
+        {0x01, 0x01, 0x35, 0x00}, {0x16, 0x01, 0x35, 0x01},
+        {0x01, 0x01, 0x36, 0x00}, {0x16, 0x01, 0x36, 0x01},
+        {0x01, 0x01, 0x37, 0x00}, {0x16, 0x01, 0x37, 0x01},
+        {0x01, 0x01, 0x38, 0x00}, {0x16, 0x01, 0x38, 0x01},
+        {0x01, 0x01, 0x39, 0x00}, {0x16, 0x01, 0x39, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x2f, 0x00}, {0x09, 0x01, 0x2f, 0x00},
+        {0x17, 0x01, 0x2f, 0x00}, {0x28, 0x01, 0x2f, 0x01},
+        {0x02, 0x01, 0x33, 0x00}, {0x09, 0x01, 0x33, 0x00},
+        {0x17, 0x01, 0x33, 0x00}, {0x28, 0x01, 0x33, 0x01},
+        {0x02, 0x01, 0x34, 0x00}, {0x09, 0x01, 0x34, 0x00},
+        {0x17, 0x01, 0x34, 0x00}, {0x28, 0x01, 0x34, 0x01},
+        {0x02, 0x01, 0x35, 0x00}, {0x09, 0x01, 0x35, 0x00},
+        {0x17, 0x01, 0x35, 0x00}, {0x28, 0x01, 0x35, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x2f, 0x00}, {0x06, 0x01, 0x2f, 0x00},
+        {0x0a, 0x01, 0x2f, 0x00}, {0x0f, 0x01, 0x2f, 0x00},
+        {0x18, 0x01, 0x2f, 0x00}, {0x1f, 0x01, 0x2f, 0x00},
+        {0x29, 0x01, 0x2f, 0x00}, {0x38, 0x01, 0x2f, 0x01},
+        {0x03, 0x01, 0x33, 0x00}, {0x06, 0x01, 0x33, 0x00},
+        {0x0a, 0x01, 0x33, 0x00}, {0x0f, 0x01, 0x33, 0x00},
+        {0x18, 0x01, 0x33, 0x00}, {0x1f, 0x01, 0x33, 0x00},
+        {0x29, 0x01, 0x33, 0x00}, {0x38, 0x01, 0x33, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x34, 0x00}, {0x06, 0x01, 0x34, 0x00},
+        {0x0a, 0x01, 0x34, 0x00}, {0x0f, 0x01, 0x34, 0x00},
+        {0x18, 0x01, 0x34, 0x00}, {0x1f, 0x01, 0x34, 0x00},
+        {0x29, 0x01, 0x34, 0x00}, {0x38, 0x01, 0x34, 0x01},
+        {0x03, 0x01, 0x35, 0x00}, {0x06, 0x01, 0x35, 0x00},
+        {0x0a, 0x01, 0x35, 0x00}, {0x0f, 0x01, 0x35, 0x00},
+        {0x18, 0x01, 0x35, 0x00}, {0x1f, 0x01, 0x35, 0x00},
+        {0x29, 0x01, 0x35, 0x00}, {0x38, 0x01, 0x35, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x36, 0x00}, {0x09, 0x01, 0x36, 0x00},
+        {0x17, 0x01, 0x36, 0x00}, {0x28, 0x01, 0x36, 0x01},
+        {0x02, 0x01, 0x37, 0x00}, {0x09, 0x01, 0x37, 0x00},
+        {0x17, 0x01, 0x37, 0x00}, {0x28, 0x01, 0x37, 0x01},
+        {0x02, 0x01, 0x38, 0x00}, {0x09, 0x01, 0x38, 0x00},
+        {0x17, 0x01, 0x38, 0x00}, {0x28, 0x01, 0x38, 0x01},
+        {0x02, 0x01, 0x39, 0x00}, {0x09, 0x01, 0x39, 0x00},
+        {0x17, 0x01, 0x39, 0x00}, {0x28, 0x01, 0x39, 0x01}
+    },
+    /* 20 */
+    {
+        {0x03, 0x01, 0x36, 0x00}, {0x06, 0x01, 0x36, 0x00},
+        {0x0a, 0x01, 0x36, 0x00}, {0x0f, 0x01, 0x36, 0x00},
+        {0x18, 0x01, 0x36, 0x00}, {0x1f, 0x01, 0x36, 0x00},
+        {0x29, 0x01, 0x36, 0x00}, {0x38, 0x01, 0x36, 0x01},
+        {0x03, 0x01, 0x37, 0x00}, {0x06, 0x01, 0x37, 0x00},
+        {0x0a, 0x01, 0x37, 0x00}, {0x0f, 0x01, 0x37, 0x00},
+        {0x18, 0x01, 0x37, 0x00}, {0x1f, 0x01, 0x37, 0x00},
+        {0x29, 0x01, 0x37, 0x00}, {0x38, 0x01, 0x37, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x38, 0x00}, {0x06, 0x01, 0x38, 0x00},
+        {0x0a, 0x01, 0x38, 0x00}, {0x0f, 0x01, 0x38, 0x00},
+        {0x18, 0x01, 0x38, 0x00}, {0x1f, 0x01, 0x38, 0x00},
+        {0x29, 0x01, 0x38, 0x00}, {0x38, 0x01, 0x38, 0x01},
+        {0x03, 0x01, 0x39, 0x00}, {0x06, 0x01, 0x39, 0x00},
+        {0x0a, 0x01, 0x39, 0x00}, {0x0f, 0x01, 0x39, 0x00},
+        {0x18, 0x01, 0x39, 0x00}, {0x1f, 0x01, 0x39, 0x00},
+        {0x29, 0x01, 0x39, 0x00}, {0x38, 0x01, 0x39, 0x01}
+    },
+    {
+        {0x1a, 0x00, 0x00, 0x00}, {0x1b, 0x00, 0x00, 0x00},
+        {0x1d, 0x00, 0x00, 0x00}, {0x1e, 0x00, 0x00, 0x00},
+        {0x21, 0x00, 0x00, 0x00}, {0x22, 0x00, 0x00, 0x00},
+        {0x24, 0x00, 0x00, 0x00}, {0x25, 0x00, 0x00, 0x00},
+        {0x2b, 0x00, 0x00, 0x00}, {0x2e, 0x00, 0x00, 0x00},
+        {0x32, 0x00, 0x00, 0x00}, {0x35, 0x00, 0x00, 0x00},
+        {0x3a, 0x00, 0x00, 0x00}, {0x3d, 0x00, 0x00, 0x00},
+        {0x41, 0x00, 0x00, 0x00}, {0x44, 0x00, 0x00, 0x01}
+    },
+    {
+        {0x00, 0x01, 0x3d, 0x01}, {0x00, 0x01, 0x41, 0x01},
+        {0x00, 0x01, 0x5f, 0x01}, {0x00, 0x01, 0x62, 0x01},
+        {0x00, 0x01, 0x64, 0x01}, {0x00, 0x01, 0x66, 0x01},
+        {0x00, 0x01, 0x67, 0x01}, {0x00, 0x01, 0x68, 0x01},
+        {0x00, 0x01, 0x6c, 0x01}, {0x00, 0x01, 0x6d, 0x01},
+        {0x00, 0x01, 0x6e, 0x01}, {0x00, 0x01, 0x70, 0x01},
+        {0x00, 0x01, 0x72, 0x01}, {0x00, 0x01, 0x75, 0x01},
+        {0x26, 0x00, 0x00, 0x00}, {0x27, 0x00, 0x00, 0x00}
+    },
+    {
+        {0x01, 0x01, 0x3d, 0x00}, {0x16, 0x01, 0x3d, 0x01},
+        {0x01, 0x01, 0x41, 0x00}, {0x16, 0x01, 0x41, 0x01},
+        {0x01, 0x01, 0x5f, 0x00}, {0x16, 0x01, 0x5f, 0x01},
+        {0x01, 0x01, 0x62, 0x00}, {0x16, 0x01, 0x62, 0x01},
+        {0x01, 0x01, 0x64, 0x00}, {0x16, 0x01, 0x64, 0x01},
+        {0x01, 0x01, 0x66, 0x00}, {0x16, 0x01, 0x66, 0x01},
+        {0x01, 0x01, 0x67, 0x00}, {0x16, 0x01, 0x67, 0x01},
+        {0x01, 0x01, 0x68, 0x00}, {0x16, 0x01, 0x68, 0x01}
+    },
+    /* 25 */
+    {
+        {0x02, 0x01, 0x3d, 0x00}, {0x09, 0x01, 0x3d, 0x00},
+        {0x17, 0x01, 0x3d, 0x00}, {0x28, 0x01, 0x3d, 0x01},
+        {0x02, 0x01, 0x41, 0x00}, {0x09, 0x01, 0x41, 0x00},
+        {0x17, 0x01, 0x41, 0x00}, {0x28, 0x01, 0x41, 0x01},
+        {0x02, 0x01, 0x5f, 0x00}, {0x09, 0x01, 0x5f, 0x00},
+        {0x17, 0x01, 0x5f, 0x00}, {0x28, 0x01, 0x5f, 0x01},
+        {0x02, 0x01, 0x62, 0x00}, {0x09, 0x01, 0x62, 0x00},
+        {0x17, 0x01, 0x62, 0x00}, {0x28, 0x01, 0x62, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x3d, 0x00}, {0x06, 0x01, 0x3d, 0x00},
+        {0x0a, 0x01, 0x3d, 0x00}, {0x0f, 0x01, 0x3d, 0x00},
+        {0x18, 0x01, 0x3d, 0x00}, {0x1f, 0x01, 0x3d, 0x00},
+        {0x29, 0x01, 0x3d, 0x00}, {0x38, 0x01, 0x3d, 0x01},
+        {0x03, 0x01, 0x41, 0x00}, {0x06, 0x01, 0x41, 0x00},
+        {0x0a, 0x01, 0x41, 0x00}, {0x0f, 0x01, 0x41, 0x00},
+        {0x18, 0x01, 0x41, 0x00}, {0x1f, 0x01, 0x41, 0x00},
+        {0x29, 0x01, 0x41, 0x00}, {0x38, 0x01, 0x41, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x5f, 0x00}, {0x06, 0x01, 0x5f, 0x00},
+        {0x0a, 0x01, 0x5f, 0x00}, {0x0f, 0x01, 0x5f, 0x00},
+        {0x18, 0x01, 0x5f, 0x00}, {0x1f, 0x01, 0x5f, 0x00},
+        {0x29, 0x01, 0x5f, 0x00}, {0x38, 0x01, 0x5f, 0x01},
+        {0x03, 0x01, 0x62, 0x00}, {0x06, 0x01, 0x62, 0x00},
+        {0x0a, 0x01, 0x62, 0x00}, {0x0f, 0x01, 0x62, 0x00},
+        {0x18, 0x01, 0x62, 0x00}, {0x1f, 0x01, 0x62, 0x00},
+        {0x29, 0x01, 0x62, 0x00}, {0x38, 0x01, 0x62, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x64, 0x00}, {0x09, 0x01, 0x64, 0x00},
+        {0x17, 0x01, 0x64, 0x00}, {0x28, 0x01, 0x64, 0x01},
+        {0x02, 0x01, 0x66, 0x00}, {0x09, 0x01, 0x66, 0x00},
+        {0x17, 0x01, 0x66, 0x00}, {0x28, 0x01, 0x66, 0x01},
+        {0x02, 0x01, 0x67, 0x00}, {0x09, 0x01, 0x67, 0x00},
+        {0x17, 0x01, 0x67, 0x00}, {0x28, 0x01, 0x67, 0x01},
+        {0x02, 0x01, 0x68, 0x00}, {0x09, 0x01, 0x68, 0x00},
+        {0x17, 0x01, 0x68, 0x00}, {0x28, 0x01, 0x68, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x64, 0x00}, {0x06, 0x01, 0x64, 0x00},
+        {0x0a, 0x01, 0x64, 0x00}, {0x0f, 0x01, 0x64, 0x00},
+        {0x18, 0x01, 0x64, 0x00}, {0x1f, 0x01, 0x64, 0x00},
+        {0x29, 0x01, 0x64, 0x00}, {0x38, 0x01, 0x64, 0x01},
+        {0x03, 0x01, 0x66, 0x00}, {0x06, 0x01, 0x66, 0x00},
+        {0x0a, 0x01, 0x66, 0x00}, {0x0f, 0x01, 0x66, 0x00},
+        {0x18, 0x01, 0x66, 0x00}, {0x1f, 0x01, 0x66, 0x00},
+        {0x29, 0x01, 0x66, 0x00}, {0x38, 0x01, 0x66, 0x01}
+    },
+    /* 30 */
+    {
+        {0x03, 0x01, 0x67, 0x00}, {0x06, 0x01, 0x67, 0x00},
+        {0x0a, 0x01, 0x67, 0x00}, {0x0f, 0x01, 0x67, 0x00},
+        {0x18, 0x01, 0x67, 0x00}, {0x1f, 0x01, 0x67, 0x00},
+        {0x29, 0x01, 0x67, 0x00}, {0x38, 0x01, 0x67, 0x01},
+        {0x03, 0x01, 0x68, 0x00}, {0x06, 0x01, 0x68, 0x00},
+        {0x0a, 0x01, 0x68, 0x00}, {0x0f, 0x01, 0x68, 0x00},
+        {0x18, 0x01, 0x68, 0x00}, {0x1f, 0x01, 0x68, 0x00},
+        {0x29, 0x01, 0x68, 0x00}, {0x38, 0x01, 0x68, 0x01}
+    },
+    {
+        {0x01, 0x01, 0x6c, 0x00}, {0x16, 0x01, 0x6c, 0x01},
+        {0x01, 0x01, 0x6d, 0x00}, {0x16, 0x01, 0x6d, 0x01},
+        {0x01, 0x01, 0x6e, 0x00}, {0x16, 0x01, 0x6e, 0x01},
+        {0x01, 0x01, 0x70, 0x00}, {0x16, 0x01, 0x70, 0x01},
+        {0x01, 0x01, 0x72, 0x00}, {0x16, 0x01, 0x72, 0x01},
+        {0x01, 0x01, 0x75, 0x00}, {0x16, 0x01, 0x75, 0x01},
+        {0x00, 0x01, 0x3a, 0x01}, {0x00, 0x01, 0x42, 0x01},
+        {0x00, 0x01, 0x43, 0x01}, {0x00, 0x01, 0x44, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x6c, 0x00}, {0x09, 0x01, 0x6c, 0x00},
+        {0x17, 0x01, 0x6c, 0x00}, {0x28, 0x01, 0x6c, 0x01},
+        {0x02, 0x01, 0x6d, 0x00}, {0x09, 0x01, 0x6d, 0x00},
+        {0x17, 0x01, 0x6d, 0x00}, {0x28, 0x01, 0x6d, 0x01},
+        {0x02, 0x01, 0x6e, 0x00}, {0x09, 0x01, 0x6e, 0x00},
+        {0x17, 0x01, 0x6e, 0x00}, {0x28, 0x01, 0x6e, 0x01},
+        {0x02, 0x01, 0x70, 0x00}, {0x09, 0x01, 0x70, 0x00},
+        {0x17, 0x01, 0x70, 0x00}, {0x28, 0x01, 0x70, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x6c, 0x00}, {0x06, 0x01, 0x6c, 0x00},
+        {0x0a, 0x01, 0x6c, 0x00}, {0x0f, 0x01, 0x6c, 0x00},
+        {0x18, 0x01, 0x6c, 0x00}, {0x1f, 0x01, 0x6c, 0x00},
+        {0x29, 0x01, 0x6c, 0x00}, {0x38, 0x01, 0x6c, 0x01},
+        {0x03, 0x01, 0x6d, 0x00}, {0x06, 0x01, 0x6d, 0x00},
+        {0x0a, 0x01, 0x6d, 0x00}, {0x0f, 0x01, 0x6d, 0x00},
+        {0x18, 0x01, 0x6d, 0x00}, {0x1f, 0x01, 0x6d, 0x00},
+        {0x29, 0x01, 0x6d, 0x00}, {0x38, 0x01, 0x6d, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x6e, 0x00}, {0x06, 0x01, 0x6e, 0x00},
+        {0x0a, 0x01, 0x6e, 0x00}, {0x0f, 0x01, 0x6e, 0x00},
+        {0x18, 0x01, 0x6e, 0x00}, {0x1f, 0x01, 0x6e, 0x00},
+        {0x29, 0x01, 0x6e, 0x00}, {0x38, 0x01, 0x6e, 0x01},
+        {0x03, 0x01, 0x70, 0x00}, {0x06, 0x01, 0x70, 0x00},
+        {0x0a, 0x01, 0x70, 0x00}, {0x0f, 0x01, 0x70, 0x00},
+        {0x18, 0x01, 0x70, 0x00}, {0x1f, 0x01, 0x70, 0x00},
+        {0x29, 0x01, 0x70, 0x00}, {0x38, 0x01, 0x70, 0x01}
+    },
+    /* 35 */
+    {
+        {0x02, 0x01, 0x72, 0x00}, {0x09, 0x01, 0x72, 0x00},
+        {0x17, 0x01, 0x72, 0x00}, {0x28, 0x01, 0x72, 0x01},
+        {0x02, 0x01, 0x75, 0x00}, {0x09, 0x01, 0x75, 0x00},
+        {0x17, 0x01, 0x75, 0x00}, {0x28, 0x01, 0x75, 0x01},
+        {0x01, 0x01, 0x3a, 0x00}, {0x16, 0x01, 0x3a, 0x01},
+        {0x01, 0x01, 0x42, 0x00}, {0x16, 0x01, 0x42, 0x01},
+        {0x01, 0x01, 0x43, 0x00}, {0x16, 0x01, 0x43, 0x01},
+        {0x01, 0x01, 0x44, 0x00}, {0x16, 0x01, 0x44, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x72, 0x00}, {0x06, 0x01, 0x72, 0x00},
+        {0x0a, 0x01, 0x72, 0x00}, {0x0f, 0x01, 0x72, 0x00},
+        {0x18, 0x01, 0x72, 0x00}, {0x1f, 0x01, 0x72, 0x00},
+        {0x29, 0x01, 0x72, 0x00}, {0x38, 0x01, 0x72, 0x01},
+        {0x03, 0x01, 0x75, 0x00}, {0x06, 0x01, 0x75, 0x00},
+        {0x0a, 0x01, 0x75, 0x00}, {0x0f, 0x01, 0x75, 0x00},
+        {0x18, 0x01, 0x75, 0x00}, {0x1f, 0x01, 0x75, 0x00},
+        {0x29, 0x01, 0x75, 0x00}, {0x38, 0x01, 0x75, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x3a, 0x00}, {0x09, 0x01, 0x3a, 0x00},
+        {0x17, 0x01, 0x3a, 0x00}, {0x28, 0x01, 0x3a, 0x01},
+        {0x02, 0x01, 0x42, 0x00}, {0x09, 0x01, 0x42, 0x00},
+        {0x17, 0x01, 0x42, 0x00}, {0x28, 0x01, 0x42, 0x01},
+        {0x02, 0x01, 0x43, 0x00}, {0x09, 0x01, 0x43, 0x00},
+        {0x17, 0x01, 0x43, 0x00}, {0x28, 0x01, 0x43, 0x01},
+        {0x02, 0x01, 0x44, 0x00}, {0x09, 0x01, 0x44, 0x00},
+        {0x17, 0x01, 0x44, 0x00}, {0x28, 0x01, 0x44, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x3a, 0x00}, {0x06, 0x01, 0x3a, 0x00},
+        {0x0a, 0x01, 0x3a, 0x00}, {0x0f, 0x01, 0x3a, 0x00},
+        {0x18, 0x01, 0x3a, 0x00}, {0x1f, 0x01, 0x3a, 0x00},
+        {0x29, 0x01, 0x3a, 0x00}, {0x38, 0x01, 0x3a, 0x01},
+        {0x03, 0x01, 0x42, 0x00}, {0x06, 0x01, 0x42, 0x00},
+        {0x0a, 0x01, 0x42, 0x00}, {0x0f, 0x01, 0x42, 0x00},
+        {0x18, 0x01, 0x42, 0x00}, {0x1f, 0x01, 0x42, 0x00},
+        {0x29, 0x01, 0x42, 0x00}, {0x38, 0x01, 0x42, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x43, 0x00}, {0x06, 0x01, 0x43, 0x00},
+        {0x0a, 0x01, 0x43, 0x00}, {0x0f, 0x01, 0x43, 0x00},
+        {0x18, 0x01, 0x43, 0x00}, {0x1f, 0x01, 0x43, 0x00},
+        {0x29, 0x01, 0x43, 0x00}, {0x38, 0x01, 0x43, 0x01},
+        {0x03, 0x01, 0x44, 0x00}, {0x06, 0x01, 0x44, 0x00},
+        {0x0a, 0x01, 0x44, 0x00}, {0x0f, 0x01, 0x44, 0x00},
+        {0x18, 0x01, 0x44, 0x00}, {0x1f, 0x01, 0x44, 0x00},
+        {0x29, 0x01, 0x44, 0x00}, {0x38, 0x01, 0x44, 0x01}
+    },
+    /* 40 */
+    {
+        {0x2c, 0x00, 0x00, 0x00}, {0x2d, 0x00, 0x00, 0x00},
+        {0x2f, 0x00, 0x00, 0x00}, {0x30, 0x00, 0x00, 0x00},
+        {0x33, 0x00, 0x00, 0x00}, {0x34, 0x00, 0x00, 0x00},
+        {0x36, 0x00, 0x00, 0x00}, {0x37, 0x00, 0x00, 0x00},
+        {0x3b, 0x00, 0x00, 0x00}, {0x3c, 0x00, 0x00, 0x00},
+        {0x3e, 0x00, 0x00, 0x00}, {0x3f, 0x00, 0x00, 0x00},
+        {0x42, 0x00, 0x00, 0x00}, {0x43, 0x00, 0x00, 0x00},
+        {0x45, 0x00, 0x00, 0x00}, {0x48, 0x00, 0x00, 0x01}
+    },
+    {
+        {0x00, 0x01, 0x45, 0x01}, {0x00, 0x01, 0x46, 0x01},
+        {0x00, 0x01, 0x47, 0x01}, {0x00, 0x01, 0x48, 0x01},
+        {0x00, 0x01, 0x49, 0x01}, {0x00, 0x01, 0x4a, 0x01},
+        {0x00, 0x01, 0x4b, 0x01}, {0x00, 0x01, 0x4c, 0x01},
+        {0x00, 0x01, 0x4d, 0x01}, {0x00, 0x01, 0x4e, 0x01},
+        {0x00, 0x01, 0x4f, 0x01}, {0x00, 0x01, 0x50, 0x01},
+        {0x00, 0x01, 0x51, 0x01}, {0x00, 0x01, 0x52, 0x01},
+        {0x00, 0x01, 0x53, 0x01}, {0x00, 0x01, 0x54, 0x01}
+    },
+    {
+        {0x01, 0x01, 0x45, 0x00}, {0x16, 0x01, 0x45, 0x01},
+        {0x01, 0x01, 0x46, 0x00}, {0x16, 0x01, 0x46, 0x01},
+        {0x01, 0x01, 0x47, 0x00}, {0x16, 0x01, 0x47, 0x01},
+        {0x01, 0x01, 0x48, 0x00}, {0x16, 0x01, 0x48, 0x01},
+        {0x01, 0x01, 0x49, 0x00}, {0x16, 0x01, 0x49, 0x01},
+        {0x01, 0x01, 0x4a, 0x00}, {0x16, 0x01, 0x4a, 0x01},
+        {0x01, 0x01, 0x4b, 0x00}, {0x16, 0x01, 0x4b, 0x01},
+        {0x01, 0x01, 0x4c, 0x00}, {0x16, 0x01, 0x4c, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x45, 0x00}, {0x09, 0x01, 0x45, 0x00},
+        {0x17, 0x01, 0x45, 0x00}, {0x28, 0x01, 0x45, 0x01},
+        {0x02, 0x01, 0x46, 0x00}, {0x09, 0x01, 0x46, 0x00},
+        {0x17, 0x01, 0x46, 0x00}, {0x28, 0x01, 0x46, 0x01},
+        {0x02, 0x01, 0x47, 0x00}, {0x09, 0x01, 0x47, 0x00},
+        {0x17, 0x01, 0x47, 0x00}, {0x28, 0x01, 0x47, 0x01},
+        {0x02, 0x01, 0x48, 0x00}, {0x09, 0x01, 0x48, 0x00},
+        {0x17, 0x01, 0x48, 0x00}, {0x28, 0x01, 0x48, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x45, 0x00}, {0x06, 0x01, 0x45, 0x00},
+        {0x0a, 0x01, 0x45, 0x00}, {0x0f, 0x01, 0x45, 0x00},
+        {0x18, 0x01, 0x45, 0x00}, {0x1f, 0x01, 0x45, 0x00},
+        {0x29, 0x01, 0x45, 0x00}, {0x38, 0x01, 0x45, 0x01},
+        {0x03, 0x01, 0x46, 0x00}, {0x06, 0x01, 0x46, 0x00},
+        {0x0a, 0x01, 0x46, 0x00}, {0x0f, 0x01, 0x46, 0x00},
+        {0x18, 0x01, 0x46, 0x00}, {0x1f, 0x01, 0x46, 0x00},
+        {0x29, 0x01, 0x46, 0x00}, {0x38, 0x01, 0x46, 0x01}
+    },
+    /* 45 */
+    {
+        {0x03, 0x01, 0x47, 0x00}, {0x06, 0x01, 0x47, 0x00},
+        {0x0a, 0x01, 0x47, 0x00}, {0x0f, 0x01, 0x47, 0x00},
+        {0x18, 0x01, 0x47, 0x00}, {0x1f, 0x01, 0x47, 0x00},
+        {0x29, 0x01, 0x47, 0x00}, {0x38, 0x01, 0x47, 0x01},
+        {0x03, 0x01, 0x48, 0x00}, {0x06, 0x01, 0x48, 0x00},
+        {0x0a, 0x01, 0x48, 0x00}, {0x0f, 0x01, 0x48, 0x00},
+        {0x18, 0x01, 0x48, 0x00}, {0x1f, 0x01, 0x48, 0x00},
+        {0x29, 0x01, 0x48, 0x00}, {0x38, 0x01, 0x48, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x49, 0x00}, {0x09, 0x01, 0x49, 0x00},
+        {0x17, 0x01, 0x49, 0x00}, {0x28, 0x01, 0x49, 0x01},
+        {0x02, 0x01, 0x4a, 0x00}, {0x09, 0x01, 0x4a, 0x00},
+        {0x17, 0x01, 0x4a, 0x00}, {0x28, 0x01, 0x4a, 0x01},
+        {0x02, 0x01, 0x4b, 0x00}, {0x09, 0x01, 0x4b, 0x00},
+        {0x17, 0x01, 0x4b, 0x00}, {0x28, 0x01, 0x4b, 0x01},
+        {0x02, 0x01, 0x4c, 0x00}, {0x09, 0x01, 0x4c, 0x00},
+        {0x17, 0x01, 0x4c, 0x00}, {0x28, 0x01, 0x4c, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x49, 0x00}, {0x06, 0x01, 0x49, 0x00},
+        {0x0a, 0x01, 0x49, 0x00}, {0x0f, 0x01, 0x49, 0x00},
+        {0x18, 0x01, 0x49, 0x00}, {0x1f, 0x01, 0x49, 0x00},
+        {0x29, 0x01, 0x49, 0x00}, {0x38, 0x01, 0x49, 0x01},
+        {0x03, 0x01, 0x4a, 0x00}, {0x06, 0x01, 0x4a, 0x00},
+        {0x0a, 0x01, 0x4a, 0x00}, {0x0f, 0x01, 0x4a, 0x00},
+        {0x18, 0x01, 0x4a, 0x00}, {0x1f, 0x01, 0x4a, 0x00},
+        {0x29, 0x01, 0x4a, 0x00}, {0x38, 0x01, 0x4a, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x4b, 0x00}, {0x06, 0x01, 0x4b, 0x00},
+        {0x0a, 0x01, 0x4b, 0x00}, {0x0f, 0x01, 0x4b, 0x00},
+        {0x18, 0x01, 0x4b, 0x00}, {0x1f, 0x01, 0x4b, 0x00},
+        {0x29, 0x01, 0x4b, 0x00}, {0x38, 0x01, 0x4b, 0x01},
+        {0x03, 0x01, 0x4c, 0x00}, {0x06, 0x01, 0x4c, 0x00},
+        {0x0a, 0x01, 0x4c, 0x00}, {0x0f, 0x01, 0x4c, 0x00},
+        {0x18, 0x01, 0x4c, 0x00}, {0x1f, 0x01, 0x4c, 0x00},
+        {0x29, 0x01, 0x4c, 0x00}, {0x38, 0x01, 0x4c, 0x01}
+    },
+    {
+        {0x01, 0x01, 0x4d, 0x00}, {0x16, 0x01, 0x4d, 0x01},
+        {0x01, 0x01, 0x4e, 0x00}, {0x16, 0x01, 0x4e, 0x01},
+        {0x01, 0x01, 0x4f, 0x00}, {0x16, 0x01, 0x4f, 0x01},
+        {0x01, 0x01, 0x50, 0x00}, {0x16, 0x01, 0x50, 0x01},
+        {0x01, 0x01, 0x51, 0x00}, {0x16, 0x01, 0x51, 0x01},
+        {0x01, 0x01, 0x52, 0x00}, {0x16, 0x01, 0x52, 0x01},
+        {0x01, 0x01, 0x53, 0x00}, {0x16, 0x01, 0x53, 0x01},
+        {0x01, 0x01, 0x54, 0x00}, {0x16, 0x01, 0x54, 0x01}
+    },
+    /* 50 */
+    {
+        {0x02, 0x01, 0x4d, 0x00}, {0x09, 0x01, 0x4d, 0x00},
+        {0x17, 0x01, 0x4d, 0x00}, {0x28, 0x01, 0x4d, 0x01},
+        {0x02, 0x01, 0x4e, 0x00}, {0x09, 0x01, 0x4e, 0x00},
+        {0x17, 0x01, 0x4e, 0x00}, {0x28, 0x01, 0x4e, 0x01},
+        {0x02, 0x01, 0x4f, 0x00}, {0x09, 0x01, 0x4f, 0x00},
+        {0x17, 0x01, 0x4f, 0x00}, {0x28, 0x01, 0x4f, 0x01},
+        {0x02, 0x01, 0x50, 0x00}, {0x09, 0x01, 0x50, 0x00},
+        {0x17, 0x01, 0x50, 0x00}, {0x28, 0x01, 0x50, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x4d, 0x00}, {0x06, 0x01, 0x4d, 0x00},
+        {0x0a, 0x01, 0x4d, 0x00}, {0x0f, 0x01, 0x4d, 0x00},
+        {0x18, 0x01, 0x4d, 0x00}, {0x1f, 0x01, 0x4d, 0x00},
+        {0x29, 0x01, 0x4d, 0x00}, {0x38, 0x01, 0x4d, 0x01},
+        {0x03, 0x01, 0x4e, 0x00}, {0x06, 0x01, 0x4e, 0x00},
+        {0x0a, 0x01, 0x4e, 0x00}, {0x0f, 0x01, 0x4e, 0x00},
+        {0x18, 0x01, 0x4e, 0x00}, {0x1f, 0x01, 0x4e, 0x00},
+        {0x29, 0x01, 0x4e, 0x00}, {0x38, 0x01, 0x4e, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x4f, 0x00}, {0x06, 0x01, 0x4f, 0x00},
+        {0x0a, 0x01, 0x4f, 0x00}, {0x0f, 0x01, 0x4f, 0x00},
+        {0x18, 0x01, 0x4f, 0x00}, {0x1f, 0x01, 0x4f, 0x00},
+        {0x29, 0x01, 0x4f, 0x00}, {0x38, 0x01, 0x4f, 0x01},
+        {0x03, 0x01, 0x50, 0x00}, {0x06, 0x01, 0x50, 0x00},
+        {0x0a, 0x01, 0x50, 0x00}, {0x0f, 0x01, 0x50, 0x00},
+        {0x18, 0x01, 0x50, 0x00}, {0x1f, 0x01, 0x50, 0x00},
+        {0x29, 0x01, 0x50, 0x00}, {0x38, 0x01, 0x50, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x51, 0x00}, {0x09, 0x01, 0x51, 0x00},
+        {0x17, 0x01, 0x51, 0x00}, {0x28, 0x01, 0x51, 0x01},
+        {0x02, 0x01, 0x52, 0x00}, {0x09, 0x01, 0x52, 0x00},
+        {0x17, 0x01, 0x52, 0x00}, {0x28, 0x01, 0x52, 0x01},
+        {0x02, 0x01, 0x53, 0x00}, {0x09, 0x01, 0x53, 0x00},
+        {0x17, 0x01, 0x53, 0x00}, {0x28, 0x01, 0x53, 0x01},
+        {0x02, 0x01, 0x54, 0x00}, {0x09, 0x01, 0x54, 0x00},
+        {0x17, 0x01, 0x54, 0x00}, {0x28, 0x01, 0x54, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x51, 0x00}, {0x06, 0x01, 0x51, 0x00},
+        {0x0a, 0x01, 0x51, 0x00}, {0x0f, 0x01, 0x51, 0x00},
+        {0x18, 0x01, 0x51, 0x00}, {0x1f, 0x01, 0x51, 0x00},
+        {0x29, 0x01, 0x51, 0x00}, {0x38, 0x01, 0x51, 0x01},
+        {0x03, 0x01, 0x52, 0x00}, {0x06, 0x01, 0x52, 0x00},
+        {0x0a, 0x01, 0x52, 0x00}, {0x0f, 0x01, 0x52, 0x00},
+        {0x18, 0x01, 0x52, 0x00}, {0x1f, 0x01, 0x52, 0x00},
+        {0x29, 0x01, 0x52, 0x00}, {0x38, 0x01, 0x52, 0x01}
+    },
+    /* 55 */
+    {
+        {0x03, 0x01, 0x53, 0x00}, {0x06, 0x01, 0x53, 0x00},
+        {0x0a, 0x01, 0x53, 0x00}, {0x0f, 0x01, 0x53, 0x00},
+        {0x18, 0x01, 0x53, 0x00}, {0x1f, 0x01, 0x53, 0x00},
+        {0x29, 0x01, 0x53, 0x00}, {0x38, 0x01, 0x53, 0x01},
+        {0x03, 0x01, 0x54, 0x00}, {0x06, 0x01, 0x54, 0x00},
+        {0x0a, 0x01, 0x54, 0x00}, {0x0f, 0x01, 0x54, 0x00},
+        {0x18, 0x01, 0x54, 0x00}, {0x1f, 0x01, 0x54, 0x00},
+        {0x29, 0x01, 0x54, 0x00}, {0x38, 0x01, 0x54, 0x01}
+    },
+    {
+        {0x00, 0x01, 0x55, 0x01}, {0x00, 0x01, 0x56, 0x01},
+        {0x00, 0x01, 0x57, 0x01}, {0x00, 0x01, 0x59, 0x01},
+        {0x00, 0x01, 0x6a, 0x01}, {0x00, 0x01, 0x6b, 0x01},
+        {0x00, 0x01, 0x71, 0x01}, {0x00, 0x01, 0x76, 0x01},
+        {0x00, 0x01, 0x77, 0x01}, {0x00, 0x01, 0x78, 0x01},
+        {0x00, 0x01, 0x79, 0x01}, {0x00, 0x01, 0x7a, 0x01},
+        {0x46, 0x00, 0x00, 0x00}, {0x47, 0x00, 0x00, 0x00},
+        {0x49, 0x00, 0x00, 0x00}, {0x4a, 0x00, 0x00, 0x01}
+    },
+    {
+        {0x01, 0x01, 0x55, 0x00}, {0x16, 0x01, 0x55, 0x01},
+        {0x01, 0x01, 0x56, 0x00}, {0x16, 0x01, 0x56, 0x01},
+        {0x01, 0x01, 0x57, 0x00}, {0x16, 0x01, 0x57, 0x01},
+        {0x01, 0x01, 0x59, 0x00}, {0x16, 0x01, 0x59, 0x01},
+        {0x01, 0x01, 0x6a, 0x00}, {0x16, 0x01, 0x6a, 0x01},
+        {0x01, 0x01, 0x6b, 0x00}, {0x16, 0x01, 0x6b, 0x01},
+        {0x01, 0x01, 0x71, 0x00}, {0x16, 0x01, 0x71, 0x01},
+        {0x01, 0x01, 0x76, 0x00}, {0x16, 0x01, 0x76, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x55, 0x00}, {0x09, 0x01, 0x55, 0x00},
+        {0x17, 0x01, 0x55, 0x00}, {0x28, 0x01, 0x55, 0x01},
+        {0x02, 0x01, 0x56, 0x00}, {0x09, 0x01, 0x56, 0x00},
+        {0x17, 0x01, 0x56, 0x00}, {0x28, 0x01, 0x56, 0x01},
+        {0x02, 0x01, 0x57, 0x00}, {0x09, 0x01, 0x57, 0x00},
+        {0x17, 0x01, 0x57, 0x00}, {0x28, 0x01, 0x57, 0x01},
+        {0x02, 0x01, 0x59, 0x00}, {0x09, 0x01, 0x59, 0x00},
+        {0x17, 0x01, 0x59, 0x00}, {0x28, 0x01, 0x59, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x55, 0x00}, {0x06, 0x01, 0x55, 0x00},
+        {0x0a, 0x01, 0x55, 0x00}, {0x0f, 0x01, 0x55, 0x00},
+        {0x18, 0x01, 0x55, 0x00}, {0x1f, 0x01, 0x55, 0x00},
+        {0x29, 0x01, 0x55, 0x00}, {0x38, 0x01, 0x55, 0x01},
+        {0x03, 0x01, 0x56, 0x00}, {0x06, 0x01, 0x56, 0x00},
+        {0x0a, 0x01, 0x56, 0x00}, {0x0f, 0x01, 0x56, 0x00},
+        {0x18, 0x01, 0x56, 0x00}, {0x1f, 0x01, 0x56, 0x00},
+        {0x29, 0x01, 0x56, 0x00}, {0x38, 0x01, 0x56, 0x01}
+    },
+    /* 60 */
+    {
+        {0x03, 0x01, 0x57, 0x00}, {0x06, 0x01, 0x57, 0x00},
+        {0x0a, 0x01, 0x57, 0x00}, {0x0f, 0x01, 0x57, 0x00},
+        {0x18, 0x01, 0x57, 0x00}, {0x1f, 0x01, 0x57, 0x00},
+        {0x29, 0x01, 0x57, 0x00}, {0x38, 0x01, 0x57, 0x01},
+        {0x03, 0x01, 0x59, 0x00}, {0x06, 0x01, 0x59, 0x00},
+        {0x0a, 0x01, 0x59, 0x00}, {0x0f, 0x01, 0x59, 0x00},
+        {0x18, 0x01, 0x59, 0x00}, {0x1f, 0x01, 0x59, 0x00},
+        {0x29, 0x01, 0x59, 0x00}, {0x38, 0x01, 0x59, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x6a, 0x00}, {0x09, 0x01, 0x6a, 0x00},
+        {0x17, 0x01, 0x6a, 0x00}, {0x28, 0x01, 0x6a, 0x01},
+        {0x02, 0x01, 0x6b, 0x00}, {0x09, 0x01, 0x6b, 0x00},
+        {0x17, 0x01, 0x6b, 0x00}, {0x28, 0x01, 0x6b, 0x01},
+        {0x02, 0x01, 0x71, 0x00}, {0x09, 0x01, 0x71, 0x00},
+        {0x17, 0x01, 0x71, 0x00}, {0x28, 0x01, 0x71, 0x01},
+        {0x02, 0x01, 0x76, 0x00}, {0x09, 0x01, 0x76, 0x00},
+        {0x17, 0x01, 0x76, 0x00}, {0x28, 0x01, 0x76, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x6a, 0x00}, {0x06, 0x01, 0x6a, 0x00},
+        {0x0a, 0x01, 0x6a, 0x00}, {0x0f, 0x01, 0x6a, 0x00},
+        {0x18, 0x01, 0x6a, 0x00}, {0x1f, 0x01, 0x6a, 0x00},
+        {0x29, 0x01, 0x6a, 0x00}, {0x38, 0x01, 0x6a, 0x01},
+        {0x03, 0x01, 0x6b, 0x00}, {0x06, 0x01, 0x6b, 0x00},
+        {0x0a, 0x01, 0x6b, 0x00}, {0x0f, 0x01, 0x6b, 0x00},
+        {0x18, 0x01, 0x6b, 0x00}, {0x1f, 0x01, 0x6b, 0x00},
+        {0x29, 0x01, 0x6b, 0x00}, {0x38, 0x01, 0x6b, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x71, 0x00}, {0x06, 0x01, 0x71, 0x00},
+        {0x0a, 0x01, 0x71, 0x00}, {0x0f, 0x01, 0x71, 0x00},
+        {0x18, 0x01, 0x71, 0x00}, {0x1f, 0x01, 0x71, 0x00},
+        {0x29, 0x01, 0x71, 0x00}, {0x38, 0x01, 0x71, 0x01},
+        {0x03, 0x01, 0x76, 0x00}, {0x06, 0x01, 0x76, 0x00},
+        {0x0a, 0x01, 0x76, 0x00}, {0x0f, 0x01, 0x76, 0x00},
+        {0x18, 0x01, 0x76, 0x00}, {0x1f, 0x01, 0x76, 0x00},
+        {0x29, 0x01, 0x76, 0x00}, {0x38, 0x01, 0x76, 0x01}
+    },
+    {
+        {0x01, 0x01, 0x77, 0x00}, {0x16, 0x01, 0x77, 0x01},
+        {0x01, 0x01, 0x78, 0x00}, {0x16, 0x01, 0x78, 0x01},
+        {0x01, 0x01, 0x79, 0x00}, {0x16, 0x01, 0x79, 0x01},
+        {0x01, 0x01, 0x7a, 0x00}, {0x16, 0x01, 0x7a, 0x01},
+        {0x00, 0x01, 0x26, 0x01}, {0x00, 0x01, 0x2a, 0x01},
+        {0x00, 0x01, 0x2c, 0x01}, {0x00, 0x01, 0x3b, 0x01},
+        {0x00, 0x01, 0x58, 0x01}, {0x00, 0x01, 0x5a, 0x01},
+        {0x4b, 0x00, 0x00, 0x00}, {0x4e, 0x00, 0x00, 0x01}
+    },
+    /* 65 */
+    {
+        {0x02, 0x01, 0x77, 0x00}, {0x09, 0x01, 0x77, 0x00},
+        {0x17, 0x01, 0x77, 0x00}, {0x28, 0x01, 0x77, 0x01},
+        {0x02, 0x01, 0x78, 0x00}, {0x09, 0x01, 0x78, 0x00},
+        {0x17, 0x01, 0x78, 0x00}, {0x28, 0x01, 0x78, 0x01},
+        {0x02, 0x01, 0x79, 0x00}, {0x09, 0x01, 0x79, 0x00},
+        {0x17, 0x01, 0x79, 0x00}, {0x28, 0x01, 0x79, 0x01},
+        {0x02, 0x01, 0x7a, 0x00}, {0x09, 0x01, 0x7a, 0x00},
+        {0x17, 0x01, 0x7a, 0x00}, {0x28, 0x01, 0x7a, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x77, 0x00}, {0x06, 0x01, 0x77, 0x00},
+        {0x0a, 0x01, 0x77, 0x00}, {0x0f, 0x01, 0x77, 0x00},
+        {0x18, 0x01, 0x77, 0x00}, {0x1f, 0x01, 0x77, 0x00},
+        {0x29, 0x01, 0x77, 0x00}, {0x38, 0x01, 0x77, 0x01},
+        {0x03, 0x01, 0x78, 0x00}, {0x06, 0x01, 0x78, 0x00},
+        {0x0a, 0x01, 0x78, 0x00}, {0x0f, 0x01, 0x78, 0x00},
+        {0x18, 0x01, 0x78, 0x00}, {0x1f, 0x01, 0x78, 0x00},
+        {0x29, 0x01, 0x78, 0x00}, {0x38, 0x01, 0x78, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x79, 0x00}, {0x06, 0x01, 0x79, 0x00},
+        {0x0a, 0x01, 0x79, 0x00}, {0x0f, 0x01, 0x79, 0x00},
+        {0x18, 0x01, 0x79, 0x00}, {0x1f, 0x01, 0x79, 0x00},
+        {0x29, 0x01, 0x79, 0x00}, {0x38, 0x01, 0x79, 0x01},
+        {0x03, 0x01, 0x7a, 0x00}, {0x06, 0x01, 0x7a, 0x00},
+        {0x0a, 0x01, 0x7a, 0x00}, {0x0f, 0x01, 0x7a, 0x00},
+        {0x18, 0x01, 0x7a, 0x00}, {0x1f, 0x01, 0x7a, 0x00},
+        {0x29, 0x01, 0x7a, 0x00}, {0x38, 0x01, 0x7a, 0x01}
+    },
+    {
+        {0x01, 0x01, 0x26, 0x00}, {0x16, 0x01, 0x26, 0x01},
+        {0x01, 0x01, 0x2a, 0x00}, {0x16, 0x01, 0x2a, 0x01},
+        {0x01, 0x01, 0x2c, 0x00}, {0x16, 0x01, 0x2c, 0x01},
+        {0x01, 0x01, 0x3b, 0x00}, {0x16, 0x01, 0x3b, 0x01},
+        {0x01, 0x01, 0x58, 0x00}, {0x16, 0x01, 0x58, 0x01},
+        {0x01, 0x01, 0x5a, 0x00}, {0x16, 0x01, 0x5a, 0x01},
+        {0x4c, 0x00, 0x00, 0x00}, {0x4d, 0x00, 0x00, 0x00},
+        {0x4f, 0x00, 0x00, 0x00}, {0x51, 0x00, 0x00, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x26, 0x00}, {0x09, 0x01, 0x26, 0x00},
+        {0x17, 0x01, 0x26, 0x00}, {0x28, 0x01, 0x26, 0x01},
+        {0x02, 0x01, 0x2a, 0x00}, {0x09, 0x01, 0x2a, 0x00},
+        {0x17, 0x01, 0x2a, 0x00}, {0x28, 0x01, 0x2a, 0x01},
+        {0x02, 0x01, 0x2c, 0x00}, {0x09, 0x01, 0x2c, 0x00},
+        {0x17, 0x01, 0x2c, 0x00}, {0x28, 0x01, 0x2c, 0x01},
+        {0x02, 0x01, 0x3b, 0x00}, {0x09, 0x01, 0x3b, 0x00},
+        {0x17, 0x01, 0x3b, 0x00}, {0x28, 0x01, 0x3b, 0x01}
+    },
+    /* 70 */
+    {
+        {0x03, 0x01, 0x26, 0x00}, {0x06, 0x01, 0x26, 0x00},
+        {0x0a, 0x01, 0x26, 0x00}, {0x0f, 0x01, 0x26, 0x00},
+        {0x18, 0x01, 0x26, 0x00}, {0x1f, 0x01, 0x26, 0x00},
+        {0x29, 0x01, 0x26, 0x00}, {0x38, 0x01, 0x26, 0x01},
+        {0x03, 0x01, 0x2a, 0x00}, {0x06, 0x01, 0x2a, 0x00},
+        {0x0a, 0x01, 0x2a, 0x00}, {0x0f, 0x01, 0x2a, 0x00},
+        {0x18, 0x01, 0x2a, 0x00}, {0x1f, 0x01, 0x2a, 0x00},
+        {0x29, 0x01, 0x2a, 0x00}, {0x38, 0x01, 0x2a, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x2c, 0x00}, {0x06, 0x01, 0x2c, 0x00},
+        {0x0a, 0x01, 0x2c, 0x00}, {0x0f, 0x01, 0x2c, 0x00},
+        {0x18, 0x01, 0x2c, 0x00}, {0x1f, 0x01, 0x2c, 0x00},
+        {0x29, 0x01, 0x2c, 0x00}, {0x38, 0x01, 0x2c, 0x01},
+        {0x03, 0x01, 0x3b, 0x00}, {0x06, 0x01, 0x3b, 0x00},
+        {0x0a, 0x01, 0x3b, 0x00}, {0x0f, 0x01, 0x3b, 0x00},
+        {0x18, 0x01, 0x3b, 0x00}, {0x1f, 0x01, 0x3b, 0x00},
+        {0x29, 0x01, 0x3b, 0x00}, {0x38, 0x01, 0x3b, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x58, 0x00}, {0x09, 0x01, 0x58, 0x00},
+        {0x17, 0x01, 0x58, 0x00}, {0x28, 0x01, 0x58, 0x01},
+        {0x02, 0x01, 0x5a, 0x00}, {0x09, 0x01, 0x5a, 0x00},
+        {0x17, 0x01, 0x5a, 0x00}, {0x28, 0x01, 0x5a, 0x01},
+        {0x00, 0x01, 0x21, 0x01}, {0x00, 0x01, 0x22, 0x01},
+        {0x00, 0x01, 0x28, 0x01}, {0x00, 0x01, 0x29, 0x01},
+        {0x00, 0x01, 0x3f, 0x01}, {0x50, 0x00, 0x00, 0x00},
+        {0x52, 0x00, 0x00, 0x00}, {0x54, 0x00, 0x00, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x58, 0x00}, {0x06, 0x01, 0x58, 0x00},
+        {0x0a, 0x01, 0x58, 0x00}, {0x0f, 0x01, 0x58, 0x00},
+        {0x18, 0x01, 0x58, 0x00}, {0x1f, 0x01, 0x58, 0x00},
+        {0x29, 0x01, 0x58, 0x00}, {0x38, 0x01, 0x58, 0x01},
+        {0x03, 0x01, 0x5a, 0x00}, {0x06, 0x01, 0x5a, 0x00},
+        {0x0a, 0x01, 0x5a, 0x00}, {0x0f, 0x01, 0x5a, 0x00},
+        {0x18, 0x01, 0x5a, 0x00}, {0x1f, 0x01, 0x5a, 0x00},
+        {0x29, 0x01, 0x5a, 0x00}, {0x38, 0x01, 0x5a, 0x01}
+    },
+    {
+        {0x01, 0x01, 0x21, 0x00}, {0x16, 0x01, 0x21, 0x01},
+        {0x01, 0x01, 0x22, 0x00}, {0x16, 0x01, 0x22, 0x01},
+        {0x01, 0x01, 0x28, 0x00}, {0x16, 0x01, 0x28, 0x01},
+        {0x01, 0x01, 0x29, 0x00}, {0x16, 0x01, 0x29, 0x01},
+        {0x01, 0x01, 0x3f, 0x00}, {0x16, 0x01, 0x3f, 0x01},
+        {0x00, 0x01, 0x27, 0x01}, {0x00, 0x01, 0x2b, 0x01},
+        {0x00, 0x01, 0x7c, 0x01}, {0x53, 0x00, 0x00, 0x00},
+        {0x55, 0x00, 0x00, 0x00}, {0x58, 0x00, 0x00, 0x01}
+    },
+    /* 75 */
+    {
+        {0x02, 0x01, 0x21, 0x00}, {0x09, 0x01, 0x21, 0x00},
+        {0x17, 0x01, 0x21, 0x00}, {0x28, 0x01, 0x21, 0x01},
+        {0x02, 0x01, 0x22, 0x00}, {0x09, 0x01, 0x22, 0x00},
+        {0x17, 0x01, 0x22, 0x00}, {0x28, 0x01, 0x22, 0x01},
+        {0x02, 0x01, 0x28, 0x00}, {0x09, 0x01, 0x28, 0x00},
+        {0x17, 0x01, 0x28, 0x00}, {0x28, 0x01, 0x28, 0x01},
+        {0x02, 0x01, 0x29, 0x00}, {0x09, 0x01, 0x29, 0x00},
+        {0x17, 0x01, 0x29, 0x00}, {0x28, 0x01, 0x29, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x21, 0x00}, {0x06, 0x01, 0x21, 0x00},
+        {0x0a, 0x01, 0x21, 0x00}, {0x0f, 0x01, 0x21, 0x00},
+        {0x18, 0x01, 0x21, 0x00}, {0x1f, 0x01, 0x21, 0x00},
+        {0x29, 0x01, 0x21, 0x00}, {0x38, 0x01, 0x21, 0x01},
+        {0x03, 0x01, 0x22, 0x00}, {0x06, 0x01, 0x22, 0x00},
+        {0x0a, 0x01, 0x22, 0x00}, {0x0f, 0x01, 0x22, 0x00},
+        {0x18, 0x01, 0x22, 0x00}, {0x1f, 0x01, 0x22, 0x00},
+        {0x29, 0x01, 0x22, 0x00}, {0x38, 0x01, 0x22, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x28, 0x00}, {0x06, 0x01, 0x28, 0x00},
+        {0x0a, 0x01, 0x28, 0x00}, {0x0f, 0x01, 0x28, 0x00},
+        {0x18, 0x01, 0x28, 0x00}, {0x1f, 0x01, 0x28, 0x00},
+        {0x29, 0x01, 0x28, 0x00}, {0x38, 0x01, 0x28, 0x01},
+        {0x03, 0x01, 0x29, 0x00}, {0x06, 0x01, 0x29, 0x00},
+        {0x0a, 0x01, 0x29, 0x00}, {0x0f, 0x01, 0x29, 0x00},
+        {0x18, 0x01, 0x29, 0x00}, {0x1f, 0x01, 0x29, 0x00},
+        {0x29, 0x01, 0x29, 0x00}, {0x38, 0x01, 0x29, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x3f, 0x00}, {0x09, 0x01, 0x3f, 0x00},
+        {0x17, 0x01, 0x3f, 0x00}, {0x28, 0x01, 0x3f, 0x01},
+        {0x01, 0x01, 0x27, 0x00}, {0x16, 0x01, 0x27, 0x01},
+        {0x01, 0x01, 0x2b, 0x00}, {0x16, 0x01, 0x2b, 0x01},
+        {0x01, 0x01, 0x7c, 0x00}, {0x16, 0x01, 0x7c, 0x01},
+        {0x00, 0x01, 0x23, 0x01}, {0x00, 0x01, 0x3e, 0x01},
+        {0x56, 0x00, 0x00, 0x00}, {0x57, 0x00, 0x00, 0x00},
+        {0x59, 0x00, 0x00, 0x00}, {0x5a, 0x00, 0x00, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x3f, 0x00}, {0x06, 0x01, 0x3f, 0x00},
+        {0x0a, 0x01, 0x3f, 0x00}, {0x0f, 0x01, 0x3f, 0x00},
+        {0x18, 0x01, 0x3f, 0x00}, {0x1f, 0x01, 0x3f, 0x00},
+        {0x29, 0x01, 0x3f, 0x00}, {0x38, 0x01, 0x3f, 0x01},
+        {0x02, 0x01, 0x27, 0x00}, {0x09, 0x01, 0x27, 0x00},
+        {0x17, 0x01, 0x27, 0x00}, {0x28, 0x01, 0x27, 0x01},
+        {0x02, 0x01, 0x2b, 0x00}, {0x09, 0x01, 0x2b, 0x00},
+        {0x17, 0x01, 0x2b, 0x00}, {0x28, 0x01, 0x2b, 0x01}
+    },
+    /* 80 */
+    {
+        {0x03, 0x01, 0x27, 0x00}, {0x06, 0x01, 0x27, 0x00},
+        {0x0a, 0x01, 0x27, 0x00}, {0x0f, 0x01, 0x27, 0x00},
+        {0x18, 0x01, 0x27, 0x00}, {0x1f, 0x01, 0x27, 0x00},
+        {0x29, 0x01, 0x27, 0x00}, {0x38, 0x01, 0x27, 0x01},
+        {0x03, 0x01, 0x2b, 0x00}, {0x06, 0x01, 0x2b, 0x00},
+        {0x0a, 0x01, 0x2b, 0x00}, {0x0f, 0x01, 0x2b, 0x00},
+        {0x18, 0x01, 0x2b, 0x00}, {0x1f, 0x01, 0x2b, 0x00},
+        {0x29, 0x01, 0x2b, 0x00}, {0x38, 0x01, 0x2b, 0x01}
+    },
+    {
+        {0x02, 0x01, 0x7c, 0x00}, {0x09, 0x01, 0x7c, 0x00},
+        {0x17, 0x01, 0x7c, 0x00}, {0x28, 0x01, 0x7c, 0x01},
+        {0x01, 0x01, 0x23, 0x00}, {0x16, 0x01, 0x23, 0x01},
+        {0x01, 0x01, 0x3e, 0x00}, {0x16, 0x01, 0x3e, 0x01},
+        {0x00, 0x01, 0x00, 0x01}, {0x00, 0x01, 0x24, 0x01},
+        {0x00, 0x01, 0x40, 0x01}, {0x00, 0x01, 0x5b, 0x01},
+        {0x00, 0x01, 0x5d, 0x01}, {0x00, 0x01, 0x7e, 0x01},
+        {0x5b, 0x00, 0x00, 0x00}, {0x5c, 0x00, 0x00, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x7c, 0x00}, {0x06, 0x01, 0x7c, 0x00},
+        {0x0a, 0x01, 0x7c, 0x00}, {0x0f, 0x01, 0x7c, 0x00},
+        {0x18, 0x01, 0x7c, 0x00}, {0x1f, 0x01, 0x7c, 0x00},
+        {0x29, 0x01, 0x7c, 0x00}, {0x38, 0x01, 0x7c, 0x01},
+        {0x02, 0x01, 0x23, 0x00}, {0x09, 0x01, 0x23, 0x00},
+        {0x17, 0x01, 0x23, 0x00}, {0x28, 0x01, 0x23, 0x01},
+        {0x02, 0x01, 0x3e, 0x00}, {0x09, 0x01, 0x3e, 0x00},
+        {0x17, 0x01, 0x3e, 0x00}, {0x28, 0x01, 0x3e, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x23, 0x00}, {0x06, 0x01, 0x23, 0x00},
+        {0x0a, 0x01, 0x23, 0x00}, {0x0f, 0x01, 0x23, 0x00},
+        {0x18, 0x01, 0x23, 0x00}, {0x1f, 0x01, 0x23, 0x00},
+        {0x29, 0x01, 0x23, 0x00}, {0x38, 0x01, 0x23, 0x01},
+        {0x03, 0x01, 0x3e, 0x00}, {0x06, 0x01, 0x3e, 0x00},
+        {0x0a, 0x01, 0x3e, 0x00}, {0x0f, 0x01, 0x3e, 0x00},
+        {0x18, 0x01, 0x3e, 0x00}, {0x1f, 0x01, 0x3e, 0x00},
+        {0x29, 0x01, 0x3e, 0x00}, {0x38, 0x01, 0x3e, 0x01}
+    },
+    {
+        {0x01, 0x01, 0x00, 0x00}, {0x16, 0x01, 0x00, 0x01},
+        {0x01, 0x01, 0x24, 0x00}, {0x16, 0x01, 0x24, 0x01},
+        {0x01, 0x01, 0x40, 0x00}, {0x16, 0x01, 0x40, 0x01},
+        {0x01, 0x01, 0x5b, 0x00}, {0x16, 0x01, 0x5b, 0x01},
+        {0x01, 0x01, 0x5d, 0x00}, {0x16, 0x01, 0x5d, 0x01},
+        {0x01, 0x01, 0x7e, 0x00}, {0x16, 0x01, 0x7e, 0x01},
+        {0x00, 0x01, 0x5e, 0x01}, {0x00, 0x01, 0x7d, 0x01},
+        {0x5d, 0x00, 0x00, 0x00}, {0x5e, 0x00, 0x00, 0x01}
+    },
+    /* 85 */
+    {
+        {0x02, 0x01, 0x00, 0x00}, {0x09, 0x01, 0x00, 0x00},
+        {0x17, 0x01, 0x00, 0x00}, {0x28, 0x01, 0x00, 0x01},
+        {0x02, 0x01, 0x24, 0x00}, {0x09, 0x01, 0x24, 0x00},
+        {0x17, 0x01, 0x24, 0x00}, {0x28, 0x01, 0x24, 0x01},
+        {0x02, 0x01, 0x40, 0x00}, {0x09, 0x01, 0x40, 0x00},
+        {0x17, 0x01, 0x40, 0x00}, {0x28, 0x01, 0x40, 0x01},
+        {0x02, 0x01, 0x5b, 0x00}, {0x09, 0x01, 0x5b, 0x00},
+        {0x17, 0x01, 0x5b, 0x00}, {0x28, 0x01, 0x5b, 0x01}
+    },
+    {
+        {0x03, 0x01, 0x00, 0x00}, {0x06, 0x01, 0x00, 0x00},
+        {0x0a, 0x01, 0x00, 0x00}, {0x0f, 0x01, 0x00, 0x00},
+        {0x18, 0x01, 0x00, 0x00}, {0x1f, 0x01, 0x00, 0x00},
+        {0x29, 0x01, 0x00, 0x00}, {0x38, 0x01, 0x00, 0x01},
+        {0x03, 0x01, 0x24, 0x00}, {0x06, 0x01, 0x24, 0x00},
+        {0x0a, 0x01, 0x24, 0x00}, {0x0f, 0x01, 0x24, 0x00},

From xeioex at nginx.com  Fri Dec 24 16:58:35 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 24 Dec 2021 16:58:35 +0000
Subject: [njs] Fixed 1-byte memory over-read introduced in previous commit.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/53510b595bb5
branches:  
changeset: 1780:53510b595bb5
user:      Dmitry Volyntsev 
date:      Thu Dec 23 14:28:12 2021 +0000
description:
Fixed 1-byte memory over-read introduced in previous commit.

sizeof("\0") returns 2 because of the implicit zero byte added at
the end of string literals. Instead njs_length() was intended to be
used.

diffstat:

 external/njs_fs.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r 9e1fd062a1d8 -r 53510b595bb5 external/njs_fs.c
--- a/external/njs_fs.c	Thu Dec 23 13:30:44 2021 +0000
+++ b/external/njs_fs.c	Thu Dec 23 14:28:12 2021 +0000
@@ -2153,7 +2153,7 @@ njs_ftw(char *path, njs_file_tree_walk_c
             }
 
             path[base] = '/';
-            memcpy(&path[base + 1], d_name, length + sizeof("\0"));
+            memcpy(&path[base + 1], d_name, length + njs_length("\0"));
 
             if (fd_limit != 0) {
                 ret = njs_ftw(path, cb, fd_limit - 1, flags, &trace);

From xeioex at nginx.com  Fri Dec 24 16:58:37 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 24 Dec 2021 16:58:37 +0000
Subject: [njs] Tests: added help info for test262.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/1d95f84bc0ce
branches:  
changeset: 1781:1d95f84bc0ce
user:      Dmitry Volyntsev 
date:      Fri Dec 24 15:48:08 2021 +0000
description:
Tests: added help info for test262.

diffstat:

 test/help |  22 ++++++++++++++++++++++
 1 files changed, 22 insertions(+), 0 deletions(-)

diffs (26 lines):

diff -r 53510b595bb5 -r 1d95f84bc0ce test/help
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/test/help	Fri Dec 24 15:48:08 2021 +0000
@@ -0,0 +1,22 @@
+#!/bin/sh
+
+# Copyright (C) Dmitry Volyntsev
+# Copyright (C) NGINX, Inc.
+
+cat << END
+
+test/test262 [options] 
+
+options:
+
+  --binary=PATH             sets path to tested binary, \
+default: "$NJS_TEST_BINARY"
+  --log=PATH                sets path to output log, \
+default: "$NJS_TEST_LOG"
+  --test-dir=PATH           sets directory for test files, \
+default: "$NJS_TEST_DIR"
+  --verbose=YES             enables verbose output, \
+default: "$NJS_TEST_VERBOSE"
+  --leave=YES               disables removing of "$NJS_TEST_DIR", \
+default: "$NJS_TEST_LEAVE"
+END

From xeioex at nginx.com  Fri Dec 24 16:58:39 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 24 Dec 2021 16:58:39 +0000
Subject: [njs] Introducing --no-openssl to disable OpenSSL discover.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/4ff1c16a652a
branches:  
changeset: 1782:4ff1c16a652a
user:      Dmitry Volyntsev 
date:      Fri Dec 24 15:48:09 2021 +0000
description:
Introducing --no-openssl to disable OpenSSL discover.

diffstat:

 auto/modules      |   2 +-
 auto/openssl      |  57 +++++++++++++++++++++++++++---------------------------
 auto/options      |   4 +-
 nginx/config.make |   2 +-
 4 files changed, 33 insertions(+), 32 deletions(-)

diffs (116 lines):

diff -r 1d95f84bc0ce -r 4ff1c16a652a auto/modules
--- a/auto/modules	Fri Dec 24 15:48:08 2021 +0000
+++ b/auto/modules	Fri Dec 24 15:48:09 2021 +0000
@@ -13,7 +13,7 @@ njs_module_srcs=external/njs_crypto.c
 
 . auto/module
 
-if [ $NJS_WEBCRYPTO = YES -a $NJS_HAVE_OPENSSL = YES ]; then
+if [ $NJS_OPENSSL = YES -a $NJS_HAVE_OPENSSL = YES ]; then
 	njs_module_name=njs_webcrypto_module
 	njs_module_incs=
 	njs_module_srcs=external/njs_webcrypto.c
diff -r 1d95f84bc0ce -r 4ff1c16a652a auto/openssl
--- a/auto/openssl	Fri Dec 24 15:48:08 2021 +0000
+++ b/auto/openssl	Fri Dec 24 15:48:09 2021 +0000
@@ -6,38 +6,39 @@
 NJS_OPENSSL_LIB=
 NJS_HAVE_OPENSSL=NO
 
+if [ $NJS_OPENSSL = YES ]; then
+    njs_found=no
 
-njs_found=no
+    njs_feature="OpenSSL library"
+    njs_feature_name=NJS_HAVE_OPENSSL
+    njs_feature_run=yes
+    njs_feature_incs=
+    njs_feature_libs="-lcrypto"
+    njs_feature_test="#include 
+
+                      int main() {
+                          EVP_CIPHER_CTX  *ctx;
+                          ctx = EVP_CIPHER_CTX_new();
+                          EVP_CIPHER_CTX_free(ctx);
+                          return 0;
+                     }"
+    . auto/feature
 
 
-njs_feature="OpenSSL library"
-njs_feature_name=NJS_HAVE_OPENSSL
-njs_feature_run=yes
-njs_feature_incs=
-njs_feature_libs="-lcrypto"
-njs_feature_test="#include 
-
-                  int main() {
-                      EVP_CIPHER_CTX  *ctx;
-                      ctx = EVP_CIPHER_CTX_new();
-                      EVP_CIPHER_CTX_free(ctx);
-                      return 0;
-                 }"
-. auto/feature
-
+    if [ $njs_found = yes ]; then
+        njs_feature="OpenSSL version"
+        njs_feature_name=NJS_OPENSSL_VERSION
+        njs_feature_run=value
+        njs_feature_test="#include 
 
-if [ $njs_found = yes ]; then
-    njs_feature="OpenSSL version"
-    njs_feature_name=NJS_OPENSSL_VERSION
-    njs_feature_run=value
-    njs_feature_test="#include 
+                          int main() {
+                              printf(\"\\\"%s\\\"\", OPENSSL_VERSION_TEXT);
+                              return 0;
+                          }"
+        . auto/feature
 
-                      int main() {
-                          printf(\"\\\"%s\\\"\", OPENSSL_VERSION_TEXT);
-                          return 0;
-                      }"
-    . auto/feature
+        NJS_HAVE_OPENSSL=YES
+        NJS_OPENSSL_LIB="$njs_feature_libs"
+    fi
 
-    NJS_HAVE_OPENSSL=YES
-    NJS_OPENSSL_LIB="$njs_feature_libs"
 fi
diff -r 1d95f84bc0ce -r 4ff1c16a652a auto/options
--- a/auto/options	Fri Dec 24 15:48:08 2021 +0000
+++ b/auto/options	Fri Dec 24 15:48:09 2021 +0000
@@ -11,7 +11,7 @@ NJS_DEBUG_MEMORY=NO
 NJS_ADDRESS_SANITIZER=NO
 NJS_TEST262=YES
 
-NJS_WEBCRYPTO=YES
+NJS_OPENSSL=YES
 NJS_TRY_PCRE2=YES
 
 NJS_CONFIGURE_OPTIONS=
@@ -34,7 +34,7 @@ do
         --debug-memory=*)                NJS_DEBUG_MEMORY="$value"           ;;
         --test262=*)                     NJS_TEST262="$value"                ;;
 
-        --no-webcrypto)                  NJS_WEBCRYPTO=NO                    ;;
+        --no-openssl)                    NJS_OPENSSL=NO                      ;;
         --no-pcre2)                      NJS_TRY_PCRE2=NO                    ;;
 
         --help)
diff -r 1d95f84bc0ce -r 4ff1c16a652a nginx/config.make
--- a/nginx/config.make	Fri Dec 24 15:48:08 2021 +0000
+++ b/nginx/config.make	Fri Dec 24 15:48:09 2021 +0000
@@ -3,7 +3,7 @@ cat << END                              
 $ngx_addon_dir/../build/libnjs.a: $NGX_MAKEFILE
 	cd $ngx_addon_dir/.. \\
 	&& if [ -f build/Makefile ]; then \$(MAKE) clean; fi \\
-	&& CFLAGS="\$(CFLAGS)" CC="\$(CC)" ./configure --no-webcrypto --no-pcre2 \\
+	&& CFLAGS="\$(CFLAGS)" CC="\$(CC)" ./configure --no-openssl --no-pcre2 \\
 	&& \$(MAKE)
 
 END

From xeioex at nginx.com  Fri Dec 24 16:58:41 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 24 Dec 2021 16:58:41 +0000
Subject: [njs] Adding module suffix for njs module files.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/c72703d60d43
branches:  
changeset: 1783:c72703d60d43
user:      Dmitry Volyntsev 
date:      Fri Dec 24 15:48:09 2021 +0000
description:
Adding module suffix for njs module files.

diffstat:

 auto/make                          |     2 +-
 auto/modules                       |     8 +-
 external/njs_crypto.c              |   650 -------
 external/njs_crypto_module.c       |   650 +++++++
 external/njs_fs.c                  |  3094 ------------------------------------
 external/njs_fs_module.c           |  3094 ++++++++++++++++++++++++++++++++++++
 external/njs_query_string.c        |   971 -----------
 external/njs_query_string_module.c |   971 +++++++++++
 external/njs_webcrypto.c           |  2702 -------------------------------
 external/njs_webcrypto_module.c    |  2702 +++++++++++++++++++++++++++++++
 nginx/config                       |     2 +-
 11 files changed, 7423 insertions(+), 7423 deletions(-)

diffs (truncated from 14924 to 1000 lines):

diff -r 4ff1c16a652a -r c72703d60d43 auto/make
--- a/auto/make	Fri Dec 24 15:48:09 2021 +0000
+++ b/auto/make	Fri Dec 24 15:48:09 2021 +0000
@@ -174,7 +174,7 @@ njs_dep_post=`njs_gen_dep_post $njs_dep 
 cat << END >> $NJS_MAKEFILE
 
 $NJS_BUILD_DIR/$njs_externals_obj: \\
-    $njs_src external/njs_webcrypto.c
+    $njs_src
 	\$(NJS_CC) -c \$(NJS_CFLAGS) $NJS_LIB_AUX_CFLAGS \\
 		\$(NJS_LIB_INCS) -Injs \\
 		-o $NJS_BUILD_DIR/$njs_externals_obj \\
diff -r 4ff1c16a652a -r c72703d60d43 auto/modules
--- a/auto/modules	Fri Dec 24 15:48:09 2021 +0000
+++ b/auto/modules	Fri Dec 24 15:48:09 2021 +0000
@@ -9,26 +9,26 @@ njs_module_srcs=src/njs_buffer.c
 
 njs_module_name=njs_crypto_module
 njs_module_incs=
-njs_module_srcs=external/njs_crypto.c
+njs_module_srcs=external/njs_crypto_module.c
 
 . auto/module
 
 if [ $NJS_OPENSSL = YES -a $NJS_HAVE_OPENSSL = YES ]; then
 	njs_module_name=njs_webcrypto_module
 	njs_module_incs=
-	njs_module_srcs=external/njs_webcrypto.c
+	njs_module_srcs=external/njs_webcrypto_module.c
 
 	. auto/module
 fi
 
 njs_module_name=njs_fs_module
 njs_module_incs=
-njs_module_srcs=external/njs_fs.c
+njs_module_srcs=external/njs_fs_module.c
 
 . auto/module
 
 njs_module_name=njs_query_string_module
 njs_module_incs=
-njs_module_srcs=external/njs_query_string.c
+njs_module_srcs=external/njs_query_string_module.c
 
 . auto/module
diff -r 4ff1c16a652a -r c72703d60d43 external/njs_crypto.c
--- a/external/njs_crypto.c	Fri Dec 24 15:48:09 2021 +0000
+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000
@@ -1,650 +0,0 @@
-
-/*
- * Copyright (C) Dmitry Volyntsev
- * Copyright (C) NGINX, Inc.
- */
-
-
-#include 
-
-
-typedef void (*njs_hash_init)(void *ctx);
-typedef void (*njs_hash_update)(void *ctx, const void *data, size_t size);
-typedef void (*njs_hash_final)(u_char *result, void *ctx);
-
-typedef njs_int_t (*njs_digest_encode)(njs_vm_t *vm, njs_value_t *value,
-    const njs_str_t *src);
-
-
-typedef struct {
-    njs_str_t           name;
-
-    size_t              size;
-    njs_hash_init       init;
-    njs_hash_update     update;
-    njs_hash_final      final;
-} njs_hash_alg_t;
-
-typedef struct {
-    union {
-        njs_md5_t       md5;
-        njs_sha1_t      sha1;
-        njs_sha2_t      sha2;
-    } u;
-
-    njs_hash_alg_t      *alg;
-} njs_digest_t;
-
-typedef struct {
-    u_char              opad[64];
-
-    union {
-        njs_md5_t       md5;
-        njs_sha1_t      sha1;
-        njs_sha2_t      sha2;
-    } u;
-
-    njs_hash_alg_t      *alg;
-} njs_hmac_t;
-
-
-typedef struct {
-    njs_str_t             name;
-
-    njs_digest_encode     encode;
-} njs_crypto_enc_t;
-
-
-static njs_hash_alg_t *njs_crypto_algorithm(njs_vm_t *vm,
-    const njs_value_t *value);
-static njs_crypto_enc_t *njs_crypto_encoding(njs_vm_t *vm,
-    const njs_value_t *value);
-static njs_int_t njs_buffer_digest(njs_vm_t *vm, njs_value_t *value,
-    const njs_str_t *src);
-static njs_int_t njs_crypto_create_hash(njs_vm_t *vm, njs_value_t *args,
-    njs_uint_t nargs, njs_index_t unused);
-static njs_int_t njs_hash_prototype_update(njs_vm_t *vm, njs_value_t *args,
-    njs_uint_t nargs, njs_index_t hmac);
-static njs_int_t njs_hash_prototype_digest(njs_vm_t *vm, njs_value_t *args,
-    njs_uint_t nargs, njs_index_t hmac);
-static njs_int_t njs_crypto_create_hmac(njs_vm_t *vm, njs_value_t *args,
-    njs_uint_t nargs, njs_index_t unused);
-
-static njs_int_t njs_crypto_init(njs_vm_t *vm);
-
-
-static njs_hash_alg_t njs_hash_algorithms[] = {
-
-   {
-     njs_str("md5"),
-     16,
-     (njs_hash_init) njs_md5_init,
-     (njs_hash_update) njs_md5_update,
-     (njs_hash_final) njs_md5_final
-   },
-
-   {
-     njs_str("sha1"),
-     20,
-     (njs_hash_init) njs_sha1_init,
-     (njs_hash_update) njs_sha1_update,
-     (njs_hash_final) njs_sha1_final
-   },
-
-   {
-     njs_str("sha256"),
-     32,
-     (njs_hash_init) njs_sha2_init,
-     (njs_hash_update) njs_sha2_update,
-     (njs_hash_final) njs_sha2_final
-   },
-
-   {
-    njs_null_str,
-    0,
-    NULL,
-    NULL,
-    NULL
-   }
-
-};
-
-
-static njs_crypto_enc_t njs_encodings[] = {
-
-   {
-     njs_str("buffer"),
-     njs_buffer_digest
-   },
-
-   {
-     njs_str("hex"),
-     njs_string_hex
-   },
-
-   {
-     njs_str("base64"),
-     njs_string_base64
-   },
-
-   {
-     njs_str("base64url"),
-     njs_string_base64url
-   },
-
-   {
-    njs_null_str,
-    NULL
-   }
-};
-
-
-static njs_external_t  njs_ext_crypto_hash[] = {
-
-    {
-        .flags = NJS_EXTERN_PROPERTY | NJS_EXTERN_SYMBOL,
-        .name.symbol = NJS_SYMBOL_TO_STRING_TAG,
-        .u.property = {
-            .value = "Hash",
-        }
-    },
-
-    {
-        .flags = NJS_EXTERN_METHOD,
-        .name.string = njs_str("update"),
-        .writable = 1,
-        .configurable = 1,
-        .u.method = {
-            .native = njs_hash_prototype_update,
-            .magic8 = 0,
-        }
-    },
-
-    {
-        .flags = NJS_EXTERN_METHOD,
-        .name.string = njs_str("digest"),
-        .writable = 1,
-        .configurable = 1,
-        .u.method = {
-            .native = njs_hash_prototype_digest,
-            .magic8 = 0,
-        }
-    },
-
-    {
-        .flags = NJS_EXTERN_METHOD,
-        .name.string = njs_str("constructor"),
-        .writable = 1,
-        .configurable = 1,
-        .u.method = {
-            .native = njs_crypto_create_hash,
-        }
-    },
-};
-
-
-static njs_external_t  njs_ext_crypto_hmac[] = {
-
-    {
-        .flags = NJS_EXTERN_PROPERTY | NJS_EXTERN_SYMBOL,
-        .name.symbol = NJS_SYMBOL_TO_STRING_TAG,
-        .u.property = {
-            .value = "Hmac",
-        }
-    },
-
-    {
-        .flags = NJS_EXTERN_METHOD,
-        .name.string = njs_str("update"),
-        .writable = 1,
-        .configurable = 1,
-        .u.method = {
-            .native = njs_hash_prototype_update,
-            .magic8 = 1,
-        }
-    },
-
-    {
-        .flags = NJS_EXTERN_METHOD,
-        .name.string = njs_str("digest"),
-        .writable = 1,
-        .configurable = 1,
-        .u.method = {
-            .native = njs_hash_prototype_digest,
-            .magic8 = 1,
-        }
-    },
-
-    {
-        .flags = NJS_EXTERN_METHOD,
-        .name.string = njs_str("constructor"),
-        .writable = 1,
-        .configurable = 1,
-        .u.method = {
-            .native = njs_crypto_create_hmac,
-            .magic8 = 0,
-        }
-    },
-};
-
-
-static njs_external_t  njs_ext_crypto_crypto[] = {
-
-    {
-        .flags = NJS_EXTERN_METHOD,
-        .name.string = njs_str("createHash"),
-        .writable = 1,
-        .configurable = 1,
-        .enumerable = 1,
-        .u.method = {
-            .native = njs_crypto_create_hash,
-            .magic8 = 0,
-        }
-    },
-
-    {
-        .flags = NJS_EXTERN_METHOD,
-        .name.string = njs_str("createHmac"),
-        .writable = 1,
-        .configurable = 1,
-        .enumerable = 1,
-        .u.method = {
-            .native = njs_crypto_create_hmac,
-            .magic8 = 0,
-        }
-    },
-};
-
-
-static njs_int_t    njs_crypto_hash_proto_id;
-static njs_int_t    njs_crypto_hmac_proto_id;
-
-
-njs_module_t  njs_crypto_module = {
-    .name = njs_str("crypto"),
-    .init = njs_crypto_init,
-};
-
-
-static njs_int_t
-njs_crypto_create_hash(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
-    njs_index_t unused)
-{
-    njs_digest_t    *dgst;
-    njs_hash_alg_t  *alg;
-
-    alg = njs_crypto_algorithm(vm, njs_arg(args, nargs, 1));
-    if (njs_slow_path(alg == NULL)) {
-        return NJS_ERROR;
-    }
-
-    dgst = njs_mp_alloc(vm->mem_pool, sizeof(njs_digest_t));
-    if (njs_slow_path(dgst == NULL)) {
-        njs_memory_error(vm);
-        return NJS_ERROR;
-    }
-
-    dgst->alg = alg;
-
-    alg->init(&dgst->u);
-
-    return njs_vm_external_create(vm, &vm->retval, njs_crypto_hash_proto_id,
-                                  dgst, 0);
-}
-
-
-static njs_int_t
-njs_hash_prototype_update(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
-    njs_index_t hmac)
-{
-    njs_str_t                    data;
-    njs_int_t                    ret;
-    njs_hmac_t                   *ctx;
-    njs_value_t                  *this, dst;
-    njs_digest_t                 *dgst;
-    njs_typed_array_t            *array;
-    const njs_value_t            *value;
-    njs_array_buffer_t           *buffer;
-    const njs_buffer_encoding_t  *encoding;
-
-    this = njs_argument(args, 0);
-
-    if (!hmac) {
-        dgst = njs_vm_external(vm, njs_crypto_hash_proto_id, this);
-        if (njs_slow_path(dgst == NULL)) {
-            njs_type_error(vm, "\"this\" is not a hash object");
-            return NJS_ERROR;
-        }
-
-        if (njs_slow_path(dgst->alg == NULL)) {
-            njs_error(vm, "Digest already called");
-            return NJS_ERROR;
-        }
-
-        ctx = NULL;
-
-    } else {
-        ctx = njs_vm_external(vm, njs_crypto_hmac_proto_id, this);
-        if (njs_slow_path(ctx == NULL)) {
-            njs_type_error(vm, "\"this\" is not a hmac object");
-            return NJS_ERROR;
-        }
-
-        if (njs_slow_path(ctx->alg == NULL)) {
-            njs_error(vm, "Digest already called");
-            return NJS_ERROR;
-        }
-
-        dgst = NULL;
-    }
-
-    value = njs_arg(args, nargs, 1);
-
-    switch (value->type) {
-    case NJS_STRING:
-        encoding = njs_buffer_encoding(vm, njs_arg(args, nargs, 2));
-        if (njs_slow_path(encoding == NULL)) {
-            return NJS_ERROR;
-        }
-
-        ret = njs_buffer_decode_string(vm, value, &dst, encoding);
-        if (njs_slow_path(ret != NJS_OK)) {
-            return NJS_ERROR;
-        }
-
-        njs_string_get(&dst, &data);
-        break;
-
-    case NJS_TYPED_ARRAY:
-    case NJS_DATA_VIEW:
-        array = njs_typed_array(value);
-        buffer = array->buffer;
-        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
-            njs_type_error(vm, "detached buffer");
-            return NJS_ERROR;
-        }
-
-        data.start = &buffer->u.u8[array->offset];
-        data.length = array->byte_length;
-        break;
-
-    default:
-        njs_type_error(vm, "data argument \"%s\" is not a string "
-                       "or Buffer-like object", njs_type_string(value->type));
-
-        return NJS_ERROR;
-    }
-
-    if (!hmac) {
-        dgst->alg->update(&dgst->u, data.start, data.length);
-
-    } else {
-        ctx->alg->update(&ctx->u, data.start, data.length);
-    }
-
-    vm->retval = *this;
-
-    return NJS_OK;
-}
-
-
-static njs_int_t
-njs_hash_prototype_digest(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
-    njs_index_t hmac)
-{
-    njs_str_t         str;
-    njs_hmac_t        *ctx;
-    njs_value_t       *this;
-    njs_digest_t      *dgst;
-    njs_hash_alg_t    *alg;
-    njs_crypto_enc_t  *enc;
-    u_char            hash1[32], digest[32];
-
-    this = njs_argument(args, 0);
-
-    if (!hmac) {
-        dgst = njs_vm_external(vm, njs_crypto_hash_proto_id, this);
-        if (njs_slow_path(dgst == NULL)) {
-            njs_type_error(vm, "\"this\" is not a hash object");
-            return NJS_ERROR;
-        }
-
-        if (njs_slow_path(dgst->alg == NULL)) {
-            goto exception;
-        }
-
-        ctx = NULL;
-
-    } else {
-        ctx = njs_vm_external(vm, njs_crypto_hmac_proto_id, this);
-        if (njs_slow_path(ctx == NULL)) {
-            njs_type_error(vm, "\"this\" is not a hmac object");
-            return NJS_ERROR;
-        }
-
-        if (njs_slow_path(ctx->alg == NULL)) {
-            goto exception;
-        }
-
-        dgst = NULL;
-    }
-
-    enc = njs_crypto_encoding(vm, njs_arg(args, nargs, 1));
-    if (njs_slow_path(enc == NULL)) {
-        return NJS_ERROR;
-    }
-
-    if (!hmac) {
-        alg = dgst->alg;
-        alg->final(digest, &dgst->u);
-        dgst->alg = NULL;
-
-    } else {
-        alg = ctx->alg;
-        alg->final(hash1, &ctx->u);
-
-        alg->init(&ctx->u);
-        alg->update(&ctx->u, ctx->opad, 64);
-        alg->update(&ctx->u, hash1, alg->size);
-        alg->final(digest, &ctx->u);
-        ctx->alg = NULL;
-    }
-
-    str.start = digest;
-    str.length = alg->size;
-
-    return enc->encode(vm, &vm->retval, &str);
-
-exception:
-
-    njs_error(vm, "Digest already called");
-    return NJS_ERROR;
-}
-
-
-static njs_int_t
-njs_crypto_create_hmac(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
-    njs_index_t unused)
-{
-    njs_str_t           key;
-    njs_uint_t          i;
-    njs_hmac_t          *ctx;
-    njs_hash_alg_t      *alg;
-    njs_typed_array_t   *array;
-    const njs_value_t   *value;
-    njs_array_buffer_t  *buffer;
-    u_char              digest[32], key_buf[64];
-
-    alg = njs_crypto_algorithm(vm, njs_arg(args, nargs, 1));
-    if (njs_slow_path(alg == NULL)) {
-        return NJS_ERROR;
-    }
-
-    value = njs_arg(args, nargs, 2);
-
-    switch (value->type) {
-    case NJS_STRING:
-        njs_string_get(value, &key);
-        break;
-
-    case NJS_TYPED_ARRAY:
-    case NJS_DATA_VIEW:
-        array = njs_typed_array(value);
-        buffer = array->buffer;
-        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
-            njs_type_error(vm, "detached buffer");
-            return NJS_ERROR;
-        }
-
-        key.start = &buffer->u.u8[array->offset];
-        key.length = array->byte_length;
-        break;
-
-    default:
-        njs_type_error(vm, "key argument \"%s\" is not a string "
-                       "or Buffer-like object", njs_type_string(value->type));
-
-        return NJS_ERROR;
-    }
-
-    ctx = njs_mp_alloc(vm->mem_pool, sizeof(njs_hmac_t));
-    if (njs_slow_path(ctx == NULL)) {
-        njs_memory_error(vm);
-        return NJS_ERROR;
-    }
-
-    ctx->alg = alg;
-
-    if (key.length > sizeof(key_buf)) {
-        alg->init(&ctx->u);
-        alg->update(&ctx->u, key.start, key.length);
-        alg->final(digest, &ctx->u);
-
-        memcpy(key_buf, digest, alg->size);
-        njs_explicit_memzero(key_buf + alg->size, sizeof(key_buf) - alg->size);
-
-    } else {
-        memcpy(key_buf, key.start, key.length);
-        njs_explicit_memzero(key_buf + key.length,
-                             sizeof(key_buf) - key.length);
-    }
-
-    for (i = 0; i < 64; i++) {
-        ctx->opad[i] = key_buf[i] ^ 0x5c;
-    }
-
-    for (i = 0; i < 64; i++) {
-         key_buf[i] ^= 0x36;
-    }
-
-    alg->init(&ctx->u);
-    alg->update(&ctx->u, key_buf, 64);
-
-    return njs_vm_external_create(vm, &vm->retval, njs_crypto_hmac_proto_id,
-                                  ctx, 0);
-}
-
-
-static njs_hash_alg_t *
-njs_crypto_algorithm(njs_vm_t *vm, const njs_value_t *value)
-{
-    njs_str_t       name;
-    njs_hash_alg_t  *e;
-
-    if (njs_slow_path(!njs_is_string(value))) {
-        njs_type_error(vm, "algorithm must be a string");
-        return NULL;
-    }
-
-    njs_string_get(value, &name);
-
-    for (e = &njs_hash_algorithms[0]; e->name.length != 0; e++) {
-        if (njs_strstr_eq(&name, &e->name)) {
-            return e;
-        }
-    }
-
-    njs_type_error(vm, "not supported algorithm: \"%V\"", &name);
-
-    return NULL;
-}
-
-
-static njs_crypto_enc_t *
-njs_crypto_encoding(njs_vm_t *vm, const njs_value_t *value)
-{
-    njs_str_t         name;
-    njs_crypto_enc_t  *e;
-
-    if (njs_slow_path(!njs_is_string(value))) {
-        if (njs_is_defined(value)) {
-            njs_type_error(vm, "encoding must be a string");
-            return NULL;
-        }
-
-        return &njs_encodings[0];
-    }
-
-    njs_string_get(value, &name);
-
-    for (e = &njs_encodings[1]; e->name.length != 0; e++) {
-        if (njs_strstr_eq(&name, &e->name)) {
-            return e;
-        }
-    }
-
-    njs_type_error(vm, "Unknown digest encoding: \"%V\"", &name);
-
-    return NULL;
-}
-
-
-static njs_int_t
-njs_buffer_digest(njs_vm_t *vm, njs_value_t *value, const njs_str_t *src)
-{
-    return njs_buffer_new(vm, value, src->start, src->length);
-}
-
-
-static njs_int_t
-njs_crypto_init(njs_vm_t *vm)
-{
-    njs_int_t           ret, proto_id;
-    njs_mod_t           *module;
-    njs_opaque_value_t  value;
-
-    njs_crypto_hash_proto_id =
-                     njs_vm_external_prototype(vm, njs_ext_crypto_hash,
-                                               njs_nitems(njs_ext_crypto_hash));
-    if (njs_slow_path(njs_crypto_hash_proto_id < 0)) {
-        return NJS_ERROR;
-    }
-
-    njs_crypto_hmac_proto_id =
-                     njs_vm_external_prototype(vm, njs_ext_crypto_hmac,
-                                               njs_nitems(njs_ext_crypto_hmac));
-    if (njs_slow_path(njs_crypto_hmac_proto_id < 0)) {
-        return NJS_ERROR;
-    }
-
-    proto_id = njs_vm_external_prototype(vm, njs_ext_crypto_crypto,
-                                         njs_nitems(njs_ext_crypto_crypto));
-    if (njs_slow_path(proto_id < 0)) {
-        return NJS_ERROR;
-    }
-
-    ret = njs_vm_external_create(vm, njs_value_arg(&value), proto_id, NULL, 1);
-    if (njs_slow_path(ret != NJS_OK)) {
-        return NJS_ERROR;
-    }
-
-    module = njs_module_add(vm, &njs_str_value("crypto"), 1);
-    if (njs_slow_path(module == NULL)) {
-        return NJS_ERROR;
-    }
-
-    njs_value_assign(&module->value, &value);
-    module->function.native = 1;
-
-    return NJS_OK;
-}
diff -r 4ff1c16a652a -r c72703d60d43 external/njs_crypto_module.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/external/njs_crypto_module.c	Fri Dec 24 15:48:09 2021 +0000
@@ -0,0 +1,650 @@
+
+/*
+ * Copyright (C) Dmitry Volyntsev
+ * Copyright (C) NGINX, Inc.
+ */
+
+
+#include 
+
+
+typedef void (*njs_hash_init)(void *ctx);
+typedef void (*njs_hash_update)(void *ctx, const void *data, size_t size);
+typedef void (*njs_hash_final)(u_char *result, void *ctx);
+
+typedef njs_int_t (*njs_digest_encode)(njs_vm_t *vm, njs_value_t *value,
+    const njs_str_t *src);
+
+
+typedef struct {
+    njs_str_t           name;
+
+    size_t              size;
+    njs_hash_init       init;
+    njs_hash_update     update;
+    njs_hash_final      final;
+} njs_hash_alg_t;
+
+typedef struct {
+    union {
+        njs_md5_t       md5;
+        njs_sha1_t      sha1;
+        njs_sha2_t      sha2;
+    } u;
+
+    njs_hash_alg_t      *alg;
+} njs_digest_t;
+
+typedef struct {
+    u_char              opad[64];
+
+    union {
+        njs_md5_t       md5;
+        njs_sha1_t      sha1;
+        njs_sha2_t      sha2;
+    } u;
+
+    njs_hash_alg_t      *alg;
+} njs_hmac_t;
+
+
+typedef struct {
+    njs_str_t             name;
+
+    njs_digest_encode     encode;
+} njs_crypto_enc_t;
+
+
+static njs_hash_alg_t *njs_crypto_algorithm(njs_vm_t *vm,
+    const njs_value_t *value);
+static njs_crypto_enc_t *njs_crypto_encoding(njs_vm_t *vm,
+    const njs_value_t *value);
+static njs_int_t njs_buffer_digest(njs_vm_t *vm, njs_value_t *value,
+    const njs_str_t *src);
+static njs_int_t njs_crypto_create_hash(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t unused);
+static njs_int_t njs_hash_prototype_update(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t hmac);
+static njs_int_t njs_hash_prototype_digest(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t hmac);
+static njs_int_t njs_crypto_create_hmac(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t unused);
+
+static njs_int_t njs_crypto_init(njs_vm_t *vm);
+
+
+static njs_hash_alg_t njs_hash_algorithms[] = {
+
+   {
+     njs_str("md5"),
+     16,
+     (njs_hash_init) njs_md5_init,
+     (njs_hash_update) njs_md5_update,
+     (njs_hash_final) njs_md5_final
+   },
+
+   {
+     njs_str("sha1"),
+     20,
+     (njs_hash_init) njs_sha1_init,
+     (njs_hash_update) njs_sha1_update,
+     (njs_hash_final) njs_sha1_final
+   },
+
+   {
+     njs_str("sha256"),
+     32,
+     (njs_hash_init) njs_sha2_init,
+     (njs_hash_update) njs_sha2_update,
+     (njs_hash_final) njs_sha2_final
+   },
+
+   {
+    njs_null_str,
+    0,
+    NULL,
+    NULL,
+    NULL
+   }
+
+};
+
+
+static njs_crypto_enc_t njs_encodings[] = {
+
+   {
+     njs_str("buffer"),
+     njs_buffer_digest
+   },
+
+   {
+     njs_str("hex"),
+     njs_string_hex
+   },
+
+   {
+     njs_str("base64"),
+     njs_string_base64
+   },
+
+   {
+     njs_str("base64url"),
+     njs_string_base64url
+   },
+
+   {
+    njs_null_str,
+    NULL
+   }
+};
+
+
+static njs_external_t  njs_ext_crypto_hash[] = {
+
+    {
+        .flags = NJS_EXTERN_PROPERTY | NJS_EXTERN_SYMBOL,
+        .name.symbol = NJS_SYMBOL_TO_STRING_TAG,
+        .u.property = {
+            .value = "Hash",
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("update"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_hash_prototype_update,
+            .magic8 = 0,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("digest"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_hash_prototype_digest,
+            .magic8 = 0,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("constructor"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_crypto_create_hash,
+        }
+    },
+};
+
+
+static njs_external_t  njs_ext_crypto_hmac[] = {
+
+    {
+        .flags = NJS_EXTERN_PROPERTY | NJS_EXTERN_SYMBOL,
+        .name.symbol = NJS_SYMBOL_TO_STRING_TAG,
+        .u.property = {
+            .value = "Hmac",
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("update"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_hash_prototype_update,
+            .magic8 = 1,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("digest"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_hash_prototype_digest,
+            .magic8 = 1,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("constructor"),
+        .writable = 1,
+        .configurable = 1,
+        .u.method = {
+            .native = njs_crypto_create_hmac,
+            .magic8 = 0,
+        }
+    },
+};
+
+
+static njs_external_t  njs_ext_crypto_crypto[] = {
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("createHash"),
+        .writable = 1,
+        .configurable = 1,
+        .enumerable = 1,
+        .u.method = {
+            .native = njs_crypto_create_hash,
+            .magic8 = 0,
+        }
+    },
+
+    {
+        .flags = NJS_EXTERN_METHOD,
+        .name.string = njs_str("createHmac"),
+        .writable = 1,
+        .configurable = 1,
+        .enumerable = 1,
+        .u.method = {
+            .native = njs_crypto_create_hmac,
+            .magic8 = 0,
+        }
+    },
+};
+
+
+static njs_int_t    njs_crypto_hash_proto_id;
+static njs_int_t    njs_crypto_hmac_proto_id;
+
+
+njs_module_t  njs_crypto_module = {
+    .name = njs_str("crypto"),
+    .init = njs_crypto_init,
+};
+
+
+static njs_int_t
+njs_crypto_create_hash(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    njs_digest_t    *dgst;
+    njs_hash_alg_t  *alg;
+
+    alg = njs_crypto_algorithm(vm, njs_arg(args, nargs, 1));
+    if (njs_slow_path(alg == NULL)) {
+        return NJS_ERROR;
+    }
+
+    dgst = njs_mp_alloc(vm->mem_pool, sizeof(njs_digest_t));
+    if (njs_slow_path(dgst == NULL)) {
+        njs_memory_error(vm);
+        return NJS_ERROR;
+    }
+
+    dgst->alg = alg;
+
+    alg->init(&dgst->u);
+
+    return njs_vm_external_create(vm, &vm->retval, njs_crypto_hash_proto_id,
+                                  dgst, 0);
+}
+
+
+static njs_int_t

From xeioex at nginx.com  Fri Dec 24 16:58:43 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 24 Dec 2021 16:58:43 +0000
Subject: [njs] Building regexp backend as an external.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/8ef86212d24b
branches:  
changeset: 1784:8ef86212d24b
user:      Dmitry Volyntsev 
date:      Fri Dec 24 15:48:11 2021 +0000
description:
Building regexp backend as an external.

This allows not to build PCRE specific code as a part of libnjs.a thus
supporting nginx builds with flags like --with-pcre=PCRE_DIR.  When
--no-pcre configure option is provided external code have to implement
methods declared in njs_regex.h.

This also closes #18 issue on Github.

diffstat:

 auto/options         |    4 +
 auto/pcre            |  113 ++++----
 auto/sources         |   14 +-
 auto/summary         |    5 +-
 external/njs_regex.c |  628 +++++++++++++++++++++++++++++++++++++++++++++++++++
 nginx/config         |    1 +
 nginx/config.make    |    4 +-
 nginx/ngx_js_regex.c |   16 +
 src/njs_pcre.c       |  408 ---------------------------------
 src/njs_pcre2.c      |  240 -------------------
 10 files changed, 718 insertions(+), 715 deletions(-)

diffs (truncated from 1536 to 1000 lines):

diff -r c72703d60d43 -r 8ef86212d24b auto/options
--- a/auto/options	Fri Dec 24 15:48:09 2021 +0000
+++ b/auto/options	Fri Dec 24 15:48:11 2021 +0000
@@ -12,6 +12,8 @@ NJS_ADDRESS_SANITIZER=NO
 NJS_TEST262=YES
 
 NJS_OPENSSL=YES
+
+NJS_PCRE=YES
 NJS_TRY_PCRE2=YES
 
 NJS_CONFIGURE_OPTIONS=
@@ -35,6 +37,8 @@ do
         --test262=*)                     NJS_TEST262="$value"                ;;
 
         --no-openssl)                    NJS_OPENSSL=NO                      ;;
+
+        --no-pcre)                       NJS_PCRE=NO                         ;;
         --no-pcre2)                      NJS_TRY_PCRE2=NO                    ;;
 
         --help)
diff -r c72703d60d43 -r 8ef86212d24b auto/pcre
--- a/auto/pcre	Fri Dec 24 15:48:09 2021 +0000
+++ b/auto/pcre	Fri Dec 24 15:48:11 2021 +0000
@@ -2,72 +2,81 @@
 # Copyright (C) Igor Sysoev
 # Copyright (C) NGINX, Inc.
 
-njs_found=no
-NJS_HAVE_PCRE2=NO
+NJS_PCRE_CFLAGS=
+NJS_PCRE_LIB=
+
+NJS_HAVE_PCRE=NO
+
+if [ $NJS_PCRE = YES ]; then
 
-if [ $NJS_TRY_PCRE2 = YES ]; then
-    if /bin/sh -c "(pcre2-config --version)" >> $NJS_AUTOCONF_ERR 2>&1; then
+    njs_found=no
 
-        NJS_PCRE_CFLAGS=`pcre2-config --cflags`
-        NJS_PCRE_LIB=`pcre2-config --libs8`
+    if [ $NJS_TRY_PCRE2 = YES ]; then
+        if /bin/sh -c "(pcre2-config --version)" >> $NJS_AUTOCONF_ERR 2>&1; then
+
+            NJS_PCRE_CFLAGS=`pcre2-config --cflags`
+            NJS_PCRE_LIB=`pcre2-config --libs8`
 
-        njs_feature="PCRE2 library"
-        njs_feature_name=NJS_HAVE_PCRE2
-        njs_feature_run=no
-        njs_feature_incs="-DPCRE2_CODE_UNIT_WIDTH=8 $NJS_PCRE_CFLAGS"
-        njs_feature_libs=$NJS_PCRE_LIB
-        njs_feature_test="#include 
+            njs_feature="PCRE2 library"
+            njs_feature_name=NJS_HAVE_PCRE2
+            njs_feature_run=no
+            njs_feature_incs="-DPCRE2_CODE_UNIT_WIDTH=8 $NJS_PCRE_CFLAGS"
+            njs_feature_libs=$NJS_PCRE_LIB
+            njs_feature_test="#include 
 
-                          int main(void) {
-                              pcre2_code  *re;
+                              int main(void) {
+                                  pcre2_code  *re;
 
-                              re = pcre2_compile((PCRE2_SPTR)\"\",
-                                                 PCRE2_ZERO_TERMINATED, 0,
-                                                 NULL, NULL, NULL);
-                              return (re == NULL);
-                          }"
+                                  re = pcre2_compile((PCRE2_SPTR)\"\",
+                                                     PCRE2_ZERO_TERMINATED, 0,
+                                                     NULL, NULL, NULL);
+                                  return (re == NULL);
+                              }"
 
-        . auto/feature
+            . auto/feature
 
-        if [ $njs_found = yes ]; then
-            NJS_HAVE_PCRE2=YES
-            echo " + PCRE2 version: `pcre2-config --version`"
+            if [ $njs_found = yes ]; then
+                NJS_HAVE_PCRE=YES
+                echo " + PCRE2 version: `pcre2-config --version`"
+            fi
         fi
     fi
-fi
+
+    if [ $njs_found = no ]; then
+        if /bin/sh -c "(pcre-config --version)" >> $NJS_AUTOCONF_ERR 2>&1; then
+
+            NJS_PCRE_CFLAGS=`pcre-config --cflags`
+            NJS_PCRE_LIB=`pcre-config --libs`
 
-if [ $njs_found = no ]; then
-    if /bin/sh -c "(pcre-config --version)" >> $NJS_AUTOCONF_ERR 2>&1; then
-
-        NJS_PCRE_CFLAGS=`pcre-config --cflags`
-        NJS_PCRE_LIB=`pcre-config --libs`
+            njs_feature="PCRE library"
+            njs_feature_name=NJS_HAVE_PCRE
+            njs_feature_run=no
+            njs_feature_incs=$NJS_PCRE_CFLAGS
+            njs_feature_libs=$NJS_PCRE_LIB
+            njs_feature_test="#include 
 
-        njs_feature="PCRE library"
-        njs_feature_name=NJS_HAVE_PCRE
-        njs_feature_run=no
-        njs_feature_incs=$NJS_PCRE_CFLAGS
-        njs_feature_libs=$NJS_PCRE_LIB
-        njs_feature_test="#include 
-
-                         int main(void) {
-                             pcre  *re;
+                             int main(void) {
+                                 pcre  *re;
 
-                             re = pcre_compile(NULL, 0, NULL, 0, NULL);
-                             if (re == NULL)
-                                 return 1;
-                             return 0;
-                         }"
-        . auto/feature
+                                 re = pcre_compile(NULL, 0, NULL, 0, NULL);
+                                 if (re == NULL)
+                                     return 1;
+                                 return 0;
+                             }"
+            . auto/feature
 
-        if [ $njs_found = yes ]; then
-            echo " + PCRE version: `pcre-config --version`"
+            if [ $njs_found = yes ]; then
+                NJS_HAVE_PCRE=YES
+                echo " + PCRE version: `pcre-config --version`"
+            fi
         fi
     fi
-fi
 
-if [ $njs_found = no ]; then
-    echo
-    echo $0: error: no PCRE library found.
-    echo
-    exit 1;
+    if [ $njs_found = no ]; then
+        echo
+        echo $0: error: no PCRE library found.
+        echo
+        exit 1;
+    fi
+
 fi
diff -r c72703d60d43 -r 8ef86212d24b auto/sources
--- a/auto/sources	Fri Dec 24 15:48:09 2021 +0000
+++ b/auto/sources	Fri Dec 24 15:48:11 2021 +0000
@@ -59,14 +59,6 @@ NJS_LIB_SRCS=" \
    src/njs_async.c \
 "
 
-NJS_LIB_PCRE_SRCS=" \
-   src/njs_pcre.c \
-"
-
-NJS_LIB_PCRE2_SRCS=" \
-   src/njs_pcre2.c \
-"
-
 NJS_LIB_TEST_SRCS=" \
    src/test/lvlhsh_unit_test.c \
    src/test/random_unit_test.c \
@@ -79,10 +71,8 @@ NJS_TEST_SRCS=" \
    src/test/njs_benchmark.c \
 "
 
-if [ "$NJS_HAVE_PCRE2" = "YES" ]; then
-	NJS_LIB_SRCS="$NJS_LIB_SRCS $NJS_LIB_PCRE2_SRCS"
-else
-	NJS_LIB_SRCS="$NJS_LIB_SRCS $NJS_LIB_PCRE_SRCS"
+if [ "$NJS_PCRE" = "YES" ]; then
+	NJS_LIB_SRCS="$NJS_LIB_SRCS external/njs_regex.c"
 fi
 
 NJS_TS_SRCS=$(find ts/ -name "*.d.ts" -o -name "*.json")
diff -r c72703d60d43 -r 8ef86212d24b auto/summary
--- a/auto/summary	Fri Dec 24 15:48:09 2021 +0000
+++ b/auto/summary	Fri Dec 24 15:48:11 2021 +0000
@@ -9,7 +9,10 @@ echo
 echo " + using CC: \"$CC\""
 echo " + using CFLAGS: \"$NJS_CFLAGS $NJS_CC_OPT $CFLAGS\""
 echo
-echo " + using PCRE library: $NJS_PCRE_LIB"
+
+if [ $NJS_HAVE_PCRE = YES ]; then
+  echo " + using PCRE library: $NJS_PCRE_LIB"
+fi
 
 if [ $NJS_HAVE_READLINE = YES ]; then
   echo " + using readline library: $NJS_READLINE_LIB"
diff -r c72703d60d43 -r 8ef86212d24b external/njs_regex.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/external/njs_regex.c	Fri Dec 24 15:48:11 2021 +0000
@@ -0,0 +1,628 @@
+
+/*
+ * Copyright (C) Igor Sysoev
+ * Copyright (C) Dmitry Volyntsev
+ * Copyright (C) NGINX, Inc.
+ */
+
+
+#include 
+
+#ifdef NJS_HAVE_PCRE2
+
+#define PCRE2_CODE_UNIT_WIDTH 8
+#include 
+
+
+static const u_char* njs_regex_pcre2_error(int errcode, u_char buffer[128]);
+
+#else
+
+#include 
+
+
+static void *njs_pcre_malloc(size_t size);
+static void njs_pcre_free(void *p);
+
+
+static njs_regex_generic_ctx_t  *regex_context;
+
+#endif
+
+
+njs_regex_generic_ctx_t *
+njs_regex_generic_ctx_create(njs_pcre_malloc_t private_malloc,
+    njs_pcre_free_t private_free, void *memory_data)
+{
+#ifdef NJS_HAVE_PCRE2
+
+    return pcre2_general_context_create(private_malloc, private_free,
+                                        memory_data);
+#else
+
+    njs_regex_generic_ctx_t  *ctx;
+
+    ctx = private_malloc(sizeof(njs_regex_generic_ctx_t), memory_data);
+
+    if (njs_fast_path(ctx != NULL)) {
+        ctx->private_malloc = private_malloc;
+        ctx->private_free = private_free;
+        ctx->memory_data = memory_data;
+    }
+
+    return ctx;
+
+#endif
+}
+
+
+njs_regex_compile_ctx_t *
+njs_regex_compile_ctx_create(njs_regex_generic_ctx_t *ctx)
+{
+#ifdef NJS_HAVE_PCRE2
+
+    return pcre2_compile_context_create(ctx);
+
+#else
+
+    return ctx;
+
+#endif
+}
+
+
+
+njs_int_t
+njs_regex_escape(njs_mp_t *mp, njs_str_t *text)
+{
+#ifdef NJS_HAVE_PCRE2
+
+    return NJS_OK;
+
+#else
+
+    /*
+     * 1) PCRE with PCRE_JAVASCRIPT_COMPAT flag rejects regexps with
+     * lone closing square brackets as invalid.  Whereas according
+     * to ES6: 11.8.5 it is a valid regexp expression.
+     *
+     * 2) escaping zero byte characters as "\u0000".
+     *
+     * Escaping it here as a workaround.
+     */
+
+    size_t      brackets, zeros;
+    u_char      *p, *dst, *start, *end;
+    njs_bool_t  in;
+
+    start = text->start;
+    end = text->start + text->length;
+
+    in = 0;
+    zeros = 0;
+    brackets = 0;
+
+    for (p = start; p < end; p++) {
+
+        switch (*p) {
+        case '[':
+            in = 1;
+            break;
+
+        case ']':
+            if (!in) {
+                brackets++;
+            }
+
+            in = 0;
+            break;
+
+        case '\\':
+            p++;
+
+            if (p == end || *p != '\0') {
+                break;
+            }
+
+            /* Fall through. */
+
+        case '\0':
+            zeros++;
+            break;
+        }
+    }
+
+    if (!brackets && !zeros) {
+        return NJS_OK;
+    }
+
+    text->length = text->length + brackets + zeros * njs_length("\\u0000");
+
+    text->start = njs_mp_alloc(mp, text->length);
+    if (njs_slow_path(text->start == NULL)) {
+        return NJS_ERROR;
+    }
+
+    in = 0;
+    dst = text->start;
+
+    for (p = start; p < end; p++) {
+
+        switch (*p) {
+        case '[':
+            in = 1;
+            break;
+
+        case ']':
+            if (!in) {
+                *dst++ = '\\';
+            }
+
+            in = 0;
+            break;
+
+        case '\\':
+            *dst++ = *p++;
+
+            if (p == end) {
+                goto done;
+            }
+
+            if (*p != '\0') {
+                break;
+            }
+
+            /* Fall through. */
+
+        case '\0':
+            dst = njs_cpymem(dst, "\\u0000", 6);
+            continue;
+        }
+
+        *dst++ = *p;
+    }
+
+done:
+
+    text->length = dst - text->start;
+
+    return NJS_OK;
+
+#endif
+}
+
+
+njs_int_t
+njs_regex_compile(njs_regex_t *regex, u_char *source, size_t len,
+    njs_regex_flags_t flags, njs_regex_compile_ctx_t *cctx, njs_trace_t *trace)
+{
+#ifdef NJS_HAVE_PCRE2
+
+    int         ret;
+    u_char      *error;
+    size_t      erroff;
+    njs_uint_t  options;
+    u_char      errstr[128];
+
+    options = PCRE2_ALT_BSUX | PCRE2_MATCH_UNSET_BACKREF;
+
+    if ((flags & NJS_REGEX_IGNORE_CASE)) {
+         options |= PCRE2_CASELESS;
+    }
+
+    if ((flags & NJS_REGEX_MULTILINE)) {
+         options |= PCRE2_MULTILINE;
+    }
+
+    if ((flags & NJS_REGEX_STICKY)) {
+         options |= PCRE2_ANCHORED;
+    }
+
+    if ((flags & NJS_REGEX_UTF8)) {
+         options |= PCRE2_UTF;
+    }
+
+    regex->code = pcre2_compile(source, len, options, &ret, &erroff, cctx);
+
+    if (njs_slow_path(regex->code == NULL)) {
+        error = &source[erroff];
+
+        njs_alert(trace, NJS_LEVEL_ERROR,
+                  "pcre_compile2(\"%s\") failed: %s at \"%s\"",
+                  source, njs_regex_pcre2_error(ret, errstr), error);
+
+        return NJS_DECLINED;
+    }
+
+    ret = pcre2_pattern_info(regex->code, PCRE2_INFO_CAPTURECOUNT,
+                             ®ex->ncaptures);
+
+    if (njs_slow_path(ret < 0)) {
+        njs_alert(trace, NJS_LEVEL_ERROR,
+               "pcre2_pattern_info(\"%s\", PCRE2_INFO_CAPTURECOUNT) failed: %s",
+               source, njs_regex_pcre2_error(ret, errstr));
+
+        return NJS_ERROR;
+    }
+
+    ret = pcre2_pattern_info(regex->code, PCRE2_INFO_BACKREFMAX,
+                             ®ex->backrefmax);
+
+    if (njs_slow_path(ret < 0)) {
+        njs_alert(trace, NJS_LEVEL_ERROR,
+                 "pcre2_pattern_info(\"%s\", PCRE2_INFO_BACKREFMAX) failed: %s",
+                 source, njs_regex_pcre2_error(ret, errstr));
+
+        return NJS_ERROR;
+    }
+
+    /* Reserve additional elements for the first "$0" capture. */
+    regex->ncaptures++;
+
+    if (regex->ncaptures > 1) {
+        ret = pcre2_pattern_info(regex->code, PCRE2_INFO_NAMECOUNT,
+                                 ®ex->nentries);
+
+        if (njs_slow_path(ret < 0)) {
+            njs_alert(trace, NJS_LEVEL_ERROR,
+                  "pcre2_pattern_info(\"%s\", PCRE2_INFO_NAMECOUNT) failed: %s",
+                   source, njs_regex_pcre2_error(ret, errstr));
+
+            return NJS_ERROR;
+        }
+
+        if (regex->nentries != 0) {
+            ret = pcre2_pattern_info(regex->code, PCRE2_INFO_NAMEENTRYSIZE,
+                                     ®ex->entry_size);
+
+            if (njs_slow_path(ret < 0)) {
+                njs_alert(trace, NJS_LEVEL_ERROR,
+                          "pcre2_pattern_info(\"%s\", PCRE2_INFO_NAMEENTRYSIZE)"
+                          " failed: %s", source,
+                          njs_regex_pcre2_error(ret, errstr));
+
+                return NJS_ERROR;
+            }
+
+            ret = pcre2_pattern_info(regex->code, PCRE2_INFO_NAMETABLE,
+                                     ®ex->entries);
+
+            if (njs_slow_path(ret < 0)) {
+                njs_alert(trace, NJS_LEVEL_ERROR,
+                          "pcre2_pattern_info(\"%s\", PCRE2_INFO_NAMETABLE) "
+                          "failed: %s", source,
+                          njs_regex_pcre2_error(ret, errstr));
+
+                return NJS_ERROR;
+            }
+        }
+    }
+
+    return NJS_OK;
+
+#else
+
+    int                      ret, err, erroff;
+    char                     *pattern, *error;
+    void                     *(*saved_malloc)(size_t size);
+    void                     (*saved_free)(void *p);
+    njs_uint_t               options;
+    const char               *errstr;
+    njs_regex_generic_ctx_t  *ctx;
+
+    ctx = cctx;
+
+    ret = NJS_ERROR;
+
+    saved_malloc = pcre_malloc;
+    pcre_malloc = njs_pcre_malloc;
+    saved_free = pcre_free;
+    pcre_free = njs_pcre_free;
+    regex_context = ctx;
+
+#ifdef PCRE_JAVASCRIPT_COMPAT
+    /* JavaScript compatibility has been introduced in PCRE-7.7. */
+    options = PCRE_JAVASCRIPT_COMPAT;
+#else
+    options = 0;
+#endif
+
+    if ((flags & NJS_REGEX_IGNORE_CASE)) {
+         options |= PCRE_CASELESS;
+    }
+
+    if ((flags & NJS_REGEX_MULTILINE)) {
+         options |= PCRE_MULTILINE;
+    }
+
+    if ((flags & NJS_REGEX_STICKY)) {
+         options |= PCRE_ANCHORED;
+    }
+
+    if ((flags & NJS_REGEX_UTF8)) {
+         options |= PCRE_UTF8;
+    }
+
+    pattern = (char *) source;
+
+    regex->code = pcre_compile(pattern, options, &errstr, &erroff, NULL);
+
+    if (njs_slow_path(regex->code == NULL)) {
+        error = pattern + erroff;
+
+        if (*error != '\0') {
+            njs_alert(trace, NJS_LEVEL_ERROR,
+                      "pcre_compile(\"%s\") failed: %s at \"%s\"",
+                      pattern, errstr, error);
+
+        } else {
+            njs_alert(trace, NJS_LEVEL_ERROR,
+                      "pcre_compile(\"%s\") failed: %s", pattern, errstr);
+        }
+
+        ret = NJS_DECLINED;
+
+        goto done;
+    }
+
+    regex->extra = pcre_study(regex->code, 0, &errstr);
+
+    if (njs_slow_path(errstr != NULL)) {
+        njs_alert(trace, NJS_LEVEL_WARN,
+                  "pcre_study(\"%s\") failed: %s", pattern, errstr);
+    }
+
+    err = pcre_fullinfo(regex->code, NULL, PCRE_INFO_CAPTURECOUNT,
+                        ®ex->ncaptures);
+
+    if (njs_slow_path(err < 0)) {
+        njs_alert(trace, NJS_LEVEL_ERROR,
+                  "pcre_fullinfo(\"%s\", PCRE_INFO_CAPTURECOUNT) failed: %d",
+                  pattern, err);
+
+        goto done;
+    }
+
+    err = pcre_fullinfo(regex->code, NULL, PCRE_INFO_BACKREFMAX,
+                        ®ex->backrefmax);
+
+    if (njs_slow_path(err < 0)) {
+        njs_alert(trace, NJS_LEVEL_ERROR,
+                  "pcre_fullinfo(\"%s\", PCRE_INFO_BACKREFMAX) failed: %d",
+                  pattern, err);
+
+        goto done;
+    }
+
+    /* Reserve additional elements for the first "$0" capture. */
+    regex->ncaptures++;
+
+    if (regex->ncaptures > 1) {
+        err = pcre_fullinfo(regex->code, NULL, PCRE_INFO_NAMECOUNT,
+                            ®ex->nentries);
+
+        if (njs_slow_path(err < 0)) {
+            njs_alert(trace, NJS_LEVEL_ERROR,
+                      "pcre_fullinfo(\"%s\", PCRE_INFO_NAMECOUNT) failed: %d",
+                      pattern, err);
+
+            goto done;
+        }
+
+        if (regex->nentries != 0) {
+            err = pcre_fullinfo(regex->code, NULL, PCRE_INFO_NAMEENTRYSIZE,
+                                ®ex->entry_size);
+
+            if (njs_slow_path(err < 0)) {
+                njs_alert(trace, NJS_LEVEL_ERROR, "pcre_fullinfo(\"%s\", "
+                          "PCRE_INFO_NAMEENTRYSIZE) failed: %d", pattern, err);
+
+                goto done;
+            }
+
+            err = pcre_fullinfo(regex->code, NULL, PCRE_INFO_NAMETABLE,
+                                ®ex->entries);
+
+            if (njs_slow_path(err < 0)) {
+                njs_alert(trace, NJS_LEVEL_ERROR, "pcre_fullinfo(\"%s\", "
+                          "PCRE_INFO_NAMETABLE) failed: %d", pattern, err);
+
+                goto done;
+            }
+        }
+    }
+
+    ret = NJS_OK;
+
+done:
+
+    pcre_malloc = saved_malloc;
+    pcre_free = saved_free;
+    regex_context = NULL;
+
+    return ret;
+
+#endif
+}
+
+
+njs_bool_t
+njs_regex_is_valid(njs_regex_t *regex)
+{
+    return (regex->code != NULL);
+}
+
+
+njs_int_t
+njs_regex_named_captures(njs_regex_t *regex, njs_str_t *name, int n)
+{
+    char  *entry;
+
+    if (name == NULL) {
+        return regex->nentries;
+    }
+
+    if (n >= regex->nentries) {
+        return NJS_ERROR;
+    }
+
+    entry = regex->entries + regex->entry_size * n;
+
+    name->start = (u_char *) entry + 2;
+    name->length = njs_strlen(name->start);
+
+    return (entry[0] << 8) + entry[1];
+}
+
+
+njs_regex_match_data_t *
+njs_regex_match_data(njs_regex_t *regex, njs_regex_generic_ctx_t *ctx)
+{
+#ifdef NJS_HAVE_PCRE2
+
+    if (regex != NULL) {
+        return pcre2_match_data_create_from_pattern(regex->code, ctx);
+    }
+
+    return pcre2_match_data_create(0, ctx);
+
+#else
+
+    size_t                  size;
+    njs_uint_t              ncaptures;
+    njs_regex_match_data_t  *match_data;
+
+    if (regex != NULL) {
+        ncaptures = regex->ncaptures - 1;
+
+    } else {
+        ncaptures = 0;
+    }
+
+    /* Each capture is stored in 3 "int" vector elements. */
+    ncaptures *= 3;
+    size = sizeof(njs_regex_match_data_t) + ncaptures * sizeof(int);
+
+    match_data = ctx->private_malloc(size, ctx->memory_data);
+
+    if (njs_fast_path(match_data != NULL)) {
+        match_data->ncaptures = ncaptures + 3;
+    }
+
+    return match_data;
+
+#endif
+}
+
+
+void
+njs_regex_match_data_free(njs_regex_match_data_t *match_data,
+    njs_regex_generic_ctx_t *ctx)
+{
+#ifdef NJS_HAVE_PCRE2
+
+    pcre2_match_data_free(match_data);
+
+#else
+
+    ctx->private_free(match_data, ctx->memory_data);
+
+#endif
+}
+
+
+njs_int_t
+njs_regex_match(njs_regex_t *regex, const u_char *subject, size_t off,
+    size_t len, njs_regex_match_data_t *match_data, njs_trace_t *trace)
+{
+#ifdef NJS_HAVE_PCRE2
+
+    int     ret;
+    u_char  errstr[128];
+
+    ret = pcre2_match(regex->code, subject, len, off, 0, match_data, NULL);
+
+    if (ret < 0) {
+        if (ret == PCRE2_ERROR_NOMATCH) {
+            return NJS_DECLINED;
+        }
+
+        njs_alert(trace, NJS_LEVEL_ERROR, "pcre2_match() failed: %s",
+                  njs_regex_pcre2_error(ret, errstr));
+        return NJS_ERROR;
+    }
+
+    return ret;
+
+#else
+
+    int  ret;
+
+    ret = pcre_exec(regex->code, regex->extra, (const char *) subject, len,
+                    off, 0, match_data->captures, match_data->ncaptures);
+
+    if (ret <= PCRE_ERROR_NOMATCH) {
+        if (ret == PCRE_ERROR_NOMATCH) {
+            return NJS_DECLINED;
+        }
+
+        njs_alert(trace, NJS_LEVEL_ERROR, "pcre_exec() failed: %d", ret);
+        return NJS_ERROR;
+    }
+
+    return ret;
+
+#endif
+}
+
+
+size_t
+njs_regex_capture(njs_regex_match_data_t *match_data, njs_uint_t n)
+{
+#ifdef NJS_HAVE_PCRE2
+
+    size_t  c;
+
+    c = pcre2_get_ovector_pointer(match_data)[n];
+
+    if (c == PCRE2_UNSET) {
+        return NJS_REGEX_UNSET;
+    }
+
+    return c;
+
+#else
+
+    return match_data->captures[n];
+
+#endif
+}
+
+#ifdef NJS_HAVE_PCRE2
+
+static const u_char *
+njs_regex_pcre2_error(int errcode, u_char buffer[128])
+{
+    pcre2_get_error_message(errcode, buffer, 128);
+
+    return buffer;
+}
+
+#else
+
+static void *
+njs_pcre_malloc(size_t size)
+{
+    return regex_context->private_malloc(size, regex_context->memory_data);
+}
+
+
+static void
+njs_pcre_free(void *p)
+{
+    regex_context->private_free(p, regex_context->memory_data);
+}
+
+#endif
+
+
diff -r c72703d60d43 -r 8ef86212d24b nginx/config
--- a/nginx/config	Fri Dec 24 15:48:09 2021 +0000
+++ b/nginx/config	Fri Dec 24 15:48:11 2021 +0000
@@ -4,6 +4,7 @@ NJS_DEPS="$ngx_addon_dir/ngx_js.h \
     $ngx_addon_dir/ngx_js_fetch.h"
 NJS_SRCS="$ngx_addon_dir/ngx_js.c \
     $ngx_addon_dir/ngx_js_fetch.c \
+    $ngx_addon_dir/ngx_js_regex.c \
     $ngx_addon_dir/../external/njs_webcrypto_module.c"
 
 if [ $HTTP != NO ]; then
diff -r c72703d60d43 -r 8ef86212d24b nginx/config.make
--- a/nginx/config.make	Fri Dec 24 15:48:09 2021 +0000
+++ b/nginx/config.make	Fri Dec 24 15:48:11 2021 +0000
@@ -3,7 +3,7 @@ cat << END                              
 $ngx_addon_dir/../build/libnjs.a: $NGX_MAKEFILE
 	cd $ngx_addon_dir/.. \\
 	&& if [ -f build/Makefile ]; then \$(MAKE) clean; fi \\
-	&& CFLAGS="\$(CFLAGS)" CC="\$(CC)" ./configure --no-openssl --no-pcre2 \\
-	&& \$(MAKE)
+	&& CFLAGS="\$(CFLAGS)" CC="\$(CC)" ./configure --no-openssl --no-pcre \\
+	&& \$(MAKE) libnjs
 
 END
diff -r c72703d60d43 -r 8ef86212d24b nginx/ngx_js_regex.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/nginx/ngx_js_regex.c	Fri Dec 24 15:48:11 2021 +0000
@@ -0,0 +1,16 @@
+
+/*
+ * Copyright (C) Dmitry Volyntsev
+ * Copyright (C) NGINX, Inc.
+ */
+
+
+#include 
+
+#if (NGX_PCRE2)
+
+#define NJS_HAVE_PCRE2  1
+
+#endif
+
+#include "../external/njs_regex.c"
diff -r c72703d60d43 -r 8ef86212d24b src/njs_pcre.c
--- a/src/njs_pcre.c	Fri Dec 24 15:48:09 2021 +0000
+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000
@@ -1,408 +0,0 @@
-
-/*
- * Copyright (C) Igor Sysoev
- * Copyright (C) NGINX, Inc.
- */
-
-
-#include 
-
-#include 
-
-
-static void *njs_pcre_malloc(size_t size);
-static void njs_pcre_free(void *p);
-
-
-static njs_regex_generic_ctx_t  *regex_context;
-
-
-njs_regex_generic_ctx_t *
-njs_regex_generic_ctx_create(njs_pcre_malloc_t private_malloc,
-    njs_pcre_free_t private_free, void *memory_data)
-{
-    njs_regex_generic_ctx_t  *ctx;
-
-    ctx = private_malloc(sizeof(njs_regex_generic_ctx_t), memory_data);
-
-    if (njs_fast_path(ctx != NULL)) {
-        ctx->private_malloc = private_malloc;
-        ctx->private_free = private_free;
-        ctx->memory_data = memory_data;
-    }
-
-    return ctx;
-}
-
-
-njs_regex_compile_ctx_t *
-njs_regex_compile_ctx_create(njs_regex_generic_ctx_t *ctx)
-{
-    return ctx;
-}
-
-
-/*
- * 1) PCRE with PCRE_JAVASCRIPT_COMPAT flag rejects regexps with
- * lone closing square brackets as invalid.  Whereas according
- * to ES6: 11.8.5 it is a valid regexp expression.
- *
- * 2) escaping zero byte characters as "\u0000".
- *
- * Escaping it here as a workaround.
- */
-
-njs_int_t
-njs_regex_escape(njs_mp_t *mp, njs_str_t *text)
-{
-    size_t      brackets, zeros;
-    u_char      *p, *dst, *start, *end;
-    njs_bool_t  in;
-
-    start = text->start;
-    end = text->start + text->length;
-
-    in = 0;
-    zeros = 0;
-    brackets = 0;
-
-    for (p = start; p < end; p++) {
-
-        switch (*p) {
-        case '[':
-            in = 1;
-            break;
-
-        case ']':
-            if (!in) {
-                brackets++;
-            }
-
-            in = 0;
-            break;
-
-        case '\\':
-            p++;
-
-            if (p == end || *p != '\0') {
-                break;
-            }
-
-            /* Fall through. */
-
-        case '\0':
-            zeros++;
-            break;
-        }
-    }
-
-    if (!brackets && !zeros) {
-        return NJS_OK;
-    }
-
-    text->length = text->length + brackets + zeros * njs_length("\\u0000");
-
-    text->start = njs_mp_alloc(mp, text->length);
-    if (njs_slow_path(text->start == NULL)) {
-        return NJS_ERROR;
-    }
-
-    in = 0;
-    dst = text->start;
-
-    for (p = start; p < end; p++) {
-
-        switch (*p) {
-        case '[':

From xeioex at nginx.com  Fri Dec 24 16:58:45 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 24 Dec 2021 16:58:45 +0000
Subject: [njs] Passing --ld-opt flags while linking.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/d493f47f2734
branches:  
changeset: 1785:d493f47f2734
user:      Dmitry Volyntsev 
date:      Fri Dec 24 15:48:12 2021 +0000
description:
Passing --ld-opt flags while linking.

diffstat:

 auto/make |  6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diffs (30 lines):

diff -r 8ef86212d24b -r d493f47f2734 auto/make
--- a/auto/make	Fri Dec 24 15:48:11 2021 +0000
+++ b/auto/make	Fri Dec 24 15:48:12 2021 +0000
@@ -109,7 +109,7 @@ cat << END >> $NJS_MAKEFILE
 		$NJS_LIB_AUX_CFLAGS \$(NJS_LIB_INCS) -Injs \\
 		src/njs_shell.c \\
 		$NJS_BUILD_DIR/libnjs.a \\
-		-lm $NJS_LIBS $NJS_LIB_AUX_LIBS $NJS_READLINE_LIB
+		$NJS_LD_OPT -lm $NJS_LIBS $NJS_LIB_AUX_LIBS $NJS_READLINE_LIB
 
 END
 
@@ -153,7 +153,7 @@ do
 	\$(NJS_LINK) -o $NJS_BUILD_DIR/$njs_bin \$(NJS_CFLAGS) \\
 		\$(NJS_LIB_INCS) $njs_dep_flags \\
 		$njs_src $NJS_BUILD_DIR/libnjs.a \\
-		$njs_dep_post -lm
+		$njs_dep_post -lm $NJS_LD_OPT
 
 -include $NJS_BUILD_DIR/$njs_dep
 
@@ -203,7 +203,7 @@ do
 		$njs_dep_flags \\
 		$NJS_BUILD_DIR/$njs_externals_obj \\
 		-Injs $njs_src $NJS_BUILD_DIR/libnjs.a \\
-		-lm $NJS_LIBS $NJS_LIB_AUX_LIBS
+		$NJS_LD_OPT -lm $NJS_LIBS $NJS_LIB_AUX_LIBS
 		$njs_dep_post
 
 -include $NJS_BUILD_DIR/$njs_dep

From xeioex at nginx.com  Fri Dec 24 16:58:49 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 24 Dec 2021 16:58:49 +0000
Subject: [njs] Improved discovery of PCRE libraries.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/9e2e4d04dfc4
branches:  
changeset: 1786:9e2e4d04dfc4
user:      Dmitry Volyntsev 
date:      Fri Dec 24 16:54:12 2021 +0000
description:
Improved discovery of PCRE libraries.

Trying to link again the library using --cc-opt and --ld-opt before
attempting to use pcre-config or pcre2-config.

diffstat:

 auto/pcre |  132 +++++++++++++++++++++++++++++++++++++++++++------------------
 1 files changed, 93 insertions(+), 39 deletions(-)

diffs (160 lines):

diff -r d493f47f2734 -r 9e2e4d04dfc4 auto/pcre
--- a/auto/pcre	Fri Dec 24 15:48:12 2021 +0000
+++ b/auto/pcre	Fri Dec 24 16:54:12 2021 +0000
@@ -12,63 +12,117 @@ if [ $NJS_PCRE = YES ]; then
     njs_found=no
 
     if [ $NJS_TRY_PCRE2 = YES ]; then
-        if /bin/sh -c "(pcre2-config --version)" >> $NJS_AUTOCONF_ERR 2>&1; then
+
+        njs_feature="PCRE2 library"
+        njs_feature_name=NJS_HAVE_PCRE2
+        njs_feature_run=no
+        njs_feature_incs=
+        njs_feature_libs=
+        njs_feature_test="#define PCRE2_CODE_UNIT_WIDTH 8
+                          #include 
 
-            NJS_PCRE_CFLAGS=`pcre2-config --cflags`
-            NJS_PCRE_LIB=`pcre2-config --libs8`
+                          int main(void) {
+                              pcre2_code  *re;
+
+                              re = pcre2_compile((PCRE2_SPTR)\"\",
+                                                 PCRE2_ZERO_TERMINATED, 0,
+                                                 NULL, NULL, NULL);
+                              return (re == NULL);
+                          }"
+
+        . auto/feature
+
+         if [ $njs_found = no ]; then
 
-            njs_feature="PCRE2 library"
-            njs_feature_name=NJS_HAVE_PCRE2
-            njs_feature_run=no
-            njs_feature_incs="-DPCRE2_CODE_UNIT_WIDTH=8 $NJS_PCRE_CFLAGS"
-            njs_feature_libs=$NJS_PCRE_LIB
-            njs_feature_test="#include 
+            # pcre2-config
+
+            if /bin/sh -c "(pcre2-config --version)" >> $NJS_AUTOCONF_ERR 2>&1; then
+
+                NJS_PCRE_CFLAGS=`pcre2-config --cflags`
+                NJS_PCRE_LIB=`pcre2-config --libs8`
+
+                njs_feature="PCRE2 library in `pcre2-config --prefix 2>/dev/null`"
+                njs_feature_incs=$NJS_PCRE_CFLAGS
+                njs_feature_libs=$NJS_PCRE_LIB
+
+                . auto/feature
+            fi
+
+         fi
+
+        if [ $njs_found = yes ]; then
+            njs_feature="PCRE2 version"
+            njs_feature_name=NJS_PCRE2_VERSION
+            njs_feature_run=value
+            njs_feature_test="#define PCRE2_CODE_UNIT_WIDTH 8
+                              #include 
+                              #include 
 
                               int main(void) {
-                                  pcre2_code  *re;
-
-                                  re = pcre2_compile((PCRE2_SPTR)\"\",
-                                                     PCRE2_ZERO_TERMINATED, 0,
-                                                     NULL, NULL, NULL);
-                                  return (re == NULL);
+                                  printf(\"%d.%d\", PCRE2_MAJOR, PCRE2_MINOR);
+                                  return 0;
                               }"
 
             . auto/feature
 
-            if [ $njs_found = yes ]; then
-                NJS_HAVE_PCRE=YES
-                echo " + PCRE2 version: `pcre2-config --version`"
-            fi
+            NJS_HAVE_PCRE=YES
         fi
     fi
 
     if [ $njs_found = no ]; then
-        if /bin/sh -c "(pcre-config --version)" >> $NJS_AUTOCONF_ERR 2>&1; then
+
+        njs_feature="PCRE library"
+        njs_feature_name=NJS_HAVE_PCRE
+        njs_feature_run=no
+        njs_feature_incs=
+        njs_feature_libs=
+        njs_feature_test="#include 
+
+                          int main(void) {
+                              pcre  *re;
 
-            NJS_PCRE_CFLAGS=`pcre-config --cflags`
-            NJS_PCRE_LIB=`pcre-config --libs`
+                              re = pcre_compile(NULL, 0, NULL, 0, NULL);
+                              if (re == NULL)
+                                  return 1;
+                              return 0;
+                          }"
+
+        . auto/feature
+
+         if [ $njs_found = no ]; then
+
+            # pcre-config
+
+            njs_pcre_prefix=`pcre-config --prefix 2>/dev/null`
 
-            njs_feature="PCRE library"
-            njs_feature_name=NJS_HAVE_PCRE
-            njs_feature_run=no
-            njs_feature_incs=$NJS_PCRE_CFLAGS
-            njs_feature_libs=$NJS_PCRE_LIB
-            njs_feature_test="#include 
+            if [ -n "$njs_pcre_prefix" ]; then
+
+                NJS_PCRE_CFLAGS=`pcre-config --cflags`
+                NJS_PCRE_LIB=`pcre-config --libs`
+
+                njs_feature="PCRE library in $njs_pcre_prefix"
+                njs_feature_incs="$NJS_PCRE_CFLAGS"
+                njs_feature_libs=$NJS_PCRE_LIB
+
+                . auto/feature
+            fi
+        fi
 
-                             int main(void) {
-                                 pcre  *re;
+        if [ $njs_found = yes ]; then
+            njs_feature="PCRE version"
+            njs_feature_name=NJS_PCRE_VERSION
+            njs_feature_run=value
+            njs_feature_test="#include 
+                              #include 
 
-                                 re = pcre_compile(NULL, 0, NULL, 0, NULL);
-                                 if (re == NULL)
-                                     return 1;
-                                 return 0;
-                             }"
+                              int main(void) {
+                                  printf(\"%d.%d\", PCRE_MAJOR, PCRE_MINOR);
+                                  return 0;
+                              }"
+
             . auto/feature
 
-            if [ $njs_found = yes ]; then
-                NJS_HAVE_PCRE=YES
-                echo " + PCRE version: `pcre-config --version`"
-            fi
+            NJS_HAVE_PCRE=YES
         fi
     fi
 

From xeioex at nginx.com  Fri Dec 24 16:58:51 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 24 Dec 2021 16:58:51 +0000
Subject: [njs] Added missing element in typeof table for DataView() type.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/256421f5cca2
branches:  
changeset: 1787:256421f5cca2
user:      Artem S. Povalyukhin 
date:      Fri Dec 24 10:00:15 2021 +0300
description:
Added missing element in typeof table for DataView() type.

Previously, typeof operation for DataView object resulted
in out of bounds array accessing.

This fixes #450 issue on Github.

diffstat:

 src/njs_vmcode.c         |  1 +
 src/test/njs_unit_test.c |  3 +++
 2 files changed, 4 insertions(+), 0 deletions(-)

diffs (24 lines):

diff -r 9e2e4d04dfc4 -r 256421f5cca2 src/njs_vmcode.c
--- a/src/njs_vmcode.c	Fri Dec 24 16:54:12 2021 +0000
+++ b/src/njs_vmcode.c	Fri Dec 24 10:00:15 2021 +0300
@@ -1516,6 +1516,7 @@ njs_vmcode_typeof(njs_vm_t *vm, njs_valu
         &njs_string_object,
         &njs_string_object,
         &njs_string_object,
+        &njs_string_object,
     };
 
     vm->retval = *types[value->type];
diff -r 9e2e4d04dfc4 -r 256421f5cca2 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Fri Dec 24 16:54:12 2021 +0000
+++ b/src/test/njs_unit_test.c	Fri Dec 24 10:00:15 2021 +0300
@@ -6390,6 +6390,9 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("(new DataView(new ArrayBuffer(3)))"),
       njs_str("[object DataView]") },
 
+    { njs_str("var x = new ArrayBuffer(3); [typeof x, typeof new DataView(x)]"),
+      njs_str("object,object") },
+
     { njs_str("(new DataView(new ArrayBuffer(3))).buffer"),
       njs_str("[object ArrayBuffer]") },
 

From mdounin at mdounin.ru  Fri Dec 24 22:11:23 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 24 Dec 2021 22:11:23 +0000
Subject: [nginx] Core: fixed ngx_pcre_studies cleanup.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/2ca57257252d
branches:  
changeset: 7978:2ca57257252d
user:      Maxim Dounin 
date:      Sat Dec 25 01:07:10 2021 +0300
description:
Core: fixed ngx_pcre_studies cleanup.

If a configuration parsing fails for some reason, ngx_regex_module_init()
is not called, and ngx_pcre_studies remained set despite the fact that
the pool it was allocated from is already freed.  This might result in
a segmentation fault during runtime regular expression compilation, such
as in SSI, for example, in the single process mode, or if a worker process
died and was respawned from a master process in such an inconsistent state.

Fix is to clear ngx_pcre_studies from the pool cleanup handler (which is
anyway used to free JIT-compiled patterns).

diffstat:

 src/core/ngx_regex.c |  103 ++++++++++++++++++++++++++------------------------
 1 files changed, 53 insertions(+), 50 deletions(-)

diffs (168 lines):

diff -r 336084ff943b -r 2ca57257252d src/core/ngx_regex.c
--- a/src/core/ngx_regex.c	Tue Dec 21 07:54:16 2021 +0300
+++ b/src/core/ngx_regex.c	Sat Dec 25 01:07:10 2021 +0300
@@ -10,15 +10,14 @@
 
 
 typedef struct {
-    ngx_flag_t  pcre_jit;
+    ngx_flag_t   pcre_jit;
+    ngx_list_t  *studies;
 } ngx_regex_conf_t;
 
 
 static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
 static void ngx_libc_cdecl ngx_regex_free(void *p);
-#if (NGX_HAVE_PCRE_JIT)
-static void ngx_pcre_free_studies(void *data);
-#endif
+static void ngx_regex_cleanup(void *data);
 
 static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
 
@@ -248,18 +247,17 @@ ngx_regex_free(void *p)
 }
 
 
+static void
+ngx_regex_cleanup(void *data)
+{
 #if (NGX_HAVE_PCRE_JIT)
-
-static void
-ngx_pcre_free_studies(void *data)
-{
-    ngx_list_t *studies = data;
+    ngx_regex_conf_t *rcf = data;
 
     ngx_uint_t        i;
     ngx_list_part_t  *part;
     ngx_regex_elt_t  *elts;
 
-    part = &studies->part;
+    part = &rcf->studies->part;
     elts = part->elts;
 
     for (i = 0; /* void */ ; i++) {
@@ -274,56 +272,50 @@ ngx_pcre_free_studies(void *data)
             i = 0;
         }
 
-        if (elts[i].regex->extra != NULL) {
-            pcre_free_study(elts[i].regex->extra);
-        }
-    }
-}
-
-#endif
-
-
-static ngx_int_t
-ngx_regex_module_init(ngx_cycle_t *cycle)
-{
-    int               opt;
-    const char       *errstr;
-    ngx_uint_t        i;
-    ngx_list_part_t  *part;
-    ngx_regex_elt_t  *elts;
-
-    opt = 0;
-
-#if (NGX_HAVE_PCRE_JIT)
-    {
-    ngx_regex_conf_t    *rcf;
-    ngx_pool_cleanup_t  *cln;
-
-    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
-
-    if (rcf->pcre_jit) {
-        opt = PCRE_STUDY_JIT_COMPILE;
-
         /*
          * The PCRE JIT compiler uses mmap for its executable codes, so we
          * have to explicitly call the pcre_free_study() function to free
          * this memory.
          */
 
-        cln = ngx_pool_cleanup_add(cycle->pool, 0);
-        if (cln == NULL) {
-            return NGX_ERROR;
+        if (elts[i].regex->extra != NULL) {
+            pcre_free_study(elts[i].regex->extra);
         }
+    }
+#endif
 
-        cln->handler = ngx_pcre_free_studies;
-        cln->data = ngx_pcre_studies;
-    }
+    /*
+     * On configuration parsing errors ngx_regex_module_init() will not
+     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
+     */
+
+    ngx_pcre_studies = NULL;
+}
+
+
+static ngx_int_t
+ngx_regex_module_init(ngx_cycle_t *cycle)
+{
+    int                opt;
+    const char        *errstr;
+    ngx_uint_t         i;
+    ngx_list_part_t   *part;
+    ngx_regex_elt_t   *elts;
+    ngx_regex_conf_t  *rcf;
+
+    opt = 0;
+
+    rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
+
+#if (NGX_HAVE_PCRE_JIT)
+    if (rcf->pcre_jit) {
+        opt = PCRE_STUDY_JIT_COMPILE;
     }
 #endif
 
     ngx_regex_malloc_init(cycle->pool);
 
-    part = &ngx_pcre_studies->part;
+    part = &rcf->studies->part;
     elts = part->elts;
 
     for (i = 0; /* void */ ; i++) {
@@ -374,7 +366,8 @@ ngx_regex_module_init(ngx_cycle_t *cycle
 static void *
 ngx_regex_create_conf(ngx_cycle_t *cycle)
 {
-    ngx_regex_conf_t  *rcf;
+    ngx_regex_conf_t    *rcf;
+    ngx_pool_cleanup_t  *cln;
 
     rcf = ngx_pcalloc(cycle->pool, sizeof(ngx_regex_conf_t));
     if (rcf == NULL) {
@@ -383,11 +376,21 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
 
     rcf->pcre_jit = NGX_CONF_UNSET;
 
-    ngx_pcre_studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
-    if (ngx_pcre_studies == NULL) {
+    cln = ngx_pool_cleanup_add(cycle->pool, 0);
+    if (cln == NULL) {
         return NULL;
     }
 
+    cln->handler = ngx_regex_cleanup;
+    cln->data = rcf;
+
+    rcf->studies = ngx_list_create(cycle->pool, 8, sizeof(ngx_regex_elt_t));
+    if (rcf->studies == NULL) {
+        return NULL;
+    }
+
+    ngx_pcre_studies = rcf->studies;
+
     return rcf;
 }
 

From mdounin at mdounin.ru  Fri Dec 24 22:11:25 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 24 Dec 2021 22:11:25 +0000
Subject: [nginx] Core: ngx_regex.c style cleanup.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/060bf88d2473
branches:  
changeset: 7979:060bf88d2473
user:      Maxim Dounin 
date:      Sat Dec 25 01:07:12 2021 +0300
description:
Core: ngx_regex.c style cleanup.

Notably, ngx_pcre_pool and ngx_pcre_studies are renamed to ngx_regex_pool
and ngx_regex_studies, respectively.

diffstat:

 src/core/ngx_regex.c |  34 +++++++++++++++++-----------------
 1 files changed, 17 insertions(+), 17 deletions(-)

diffs (113 lines):

diff -r 2ca57257252d -r 060bf88d2473 src/core/ngx_regex.c
--- a/src/core/ngx_regex.c	Sat Dec 25 01:07:10 2021 +0300
+++ b/src/core/ngx_regex.c	Sat Dec 25 01:07:12 2021 +0300
@@ -15,6 +15,9 @@ typedef struct {
 } ngx_regex_conf_t;
 
 
+static ngx_inline void ngx_regex_malloc_init(ngx_pool_t *pool);
+static ngx_inline void ngx_regex_malloc_done(void);
+
 static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
 static void ngx_libc_cdecl ngx_regex_free(void *p);
 static void ngx_regex_cleanup(void *data);
@@ -64,8 +67,8 @@ ngx_module_t  ngx_regex_module = {
 };
 
 
-static ngx_pool_t  *ngx_pcre_pool;
-static ngx_list_t  *ngx_pcre_studies;
+static ngx_pool_t  *ngx_regex_pool;
+static ngx_list_t  *ngx_regex_studies;
 
 
 void
@@ -79,14 +82,14 @@ ngx_regex_init(void)
 static ngx_inline void
 ngx_regex_malloc_init(ngx_pool_t *pool)
 {
-    ngx_pcre_pool = pool;
+    ngx_regex_pool = pool;
 }
 
 
 static ngx_inline void
 ngx_regex_malloc_done(void)
 {
-    ngx_pcre_pool = NULL;
+    ngx_regex_pool = NULL;
 }
 
 
@@ -112,13 +115,13 @@ ngx_regex_compile(ngx_regex_compile_t *r
            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
                               "pcre_compile() failed: %s in \"%V\"",
                                errstr, &rc->pattern)
-                      - rc->err.data;
+                         - rc->err.data;
 
         } else {
            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
                               "pcre_compile() failed: %s in \"%V\" at \"%s\"",
                                errstr, &rc->pattern, rc->pattern.data + erroff)
-                      - rc->err.data;
+                         - rc->err.data;
         }
 
         return NGX_ERROR;
@@ -133,8 +136,8 @@ ngx_regex_compile(ngx_regex_compile_t *r
 
     /* do not study at runtime */
 
-    if (ngx_pcre_studies != NULL) {
-        elt = ngx_list_push(ngx_pcre_studies);
+    if (ngx_regex_studies != NULL) {
+        elt = ngx_list_push(ngx_regex_studies);
         if (elt == NULL) {
             goto nomem;
         }
@@ -229,11 +232,8 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
 static void * ngx_libc_cdecl
 ngx_regex_malloc(size_t size)
 {
-    ngx_pool_t      *pool;
-    pool = ngx_pcre_pool;
-
-    if (pool) {
-        return ngx_palloc(pool, size);
+    if (ngx_regex_pool) {
+        return ngx_palloc(ngx_regex_pool, size);
     }
 
     return NULL;
@@ -286,10 +286,10 @@ ngx_regex_cleanup(void *data)
 
     /*
      * On configuration parsing errors ngx_regex_module_init() will not
-     * be called.  Make sure ngx_pcre_studies is properly cleared anyway.
+     * be called.  Make sure ngx_regex_studies is properly cleared anyway.
      */
 
-    ngx_pcre_studies = NULL;
+    ngx_regex_studies = NULL;
 }
 
 
@@ -357,7 +357,7 @@ ngx_regex_module_init(ngx_cycle_t *cycle
 
     ngx_regex_malloc_done();
 
-    ngx_pcre_studies = NULL;
+    ngx_regex_studies = NULL;
 
     return NGX_OK;
 }
@@ -389,7 +389,7 @@ ngx_regex_create_conf(ngx_cycle_t *cycle
         return NULL;
     }
 
-    ngx_pcre_studies = rcf->studies;
+    ngx_regex_studies = rcf->studies;
 
     return rcf;
 }

From mdounin at mdounin.ru  Fri Dec 24 22:11:28 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 24 Dec 2021 22:11:28 +0000
Subject: [nginx] Configure: simplified PCRE compilation.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/8007ea138d6a
branches:  
changeset: 7980:8007ea138d6a
user:      Maxim Dounin 
date:      Sat Dec 25 01:07:14 2021 +0300
description:
Configure: simplified PCRE compilation.

Removed ICC-specific PCRE optimizations which tried to link with PCRE
object files instead of the library.  Made compiler-specific code
minimal.

diffstat:

 auto/lib/pcre/conf |  73 +++++------------------------------------------------
 1 files changed, 8 insertions(+), 65 deletions(-)

diffs (93 lines):

diff -r 060bf88d2473 -r 8007ea138d6a auto/lib/pcre/conf
--- a/auto/lib/pcre/conf	Sat Dec 25 01:07:12 2021 +0300
+++ b/auto/lib/pcre/conf	Sat Dec 25 01:07:14 2021 +0300
@@ -4,81 +4,24 @@
 
 
 if [ $PCRE != NONE ]; then
+
+    have=NGX_PCRE . auto/have
+
+    if [ "$NGX_PLATFORM" = win32 ]; then
+        have=PCRE_STATIC . auto/have
+    fi
+
     CORE_INCS="$CORE_INCS $PCRE"
+    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
 
     case "$NGX_CC_NAME" in
 
         msvc | owc | bcc)
-            have=NGX_PCRE . auto/have
-            have=PCRE_STATIC . auto/have
-            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
             LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
             CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
         ;;
 
-        icc)
-            have=NGX_PCRE . auto/have
-            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
-
-            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
-
-            echo $ngx_n "checking for PCRE library ...$ngx_c"
-
-            if [ -f $PCRE/pcre.h ]; then
-                ngx_pcre_ver=`grep PCRE_MAJOR $PCRE/pcre.h \
-                              | sed -e 's/^.*PCRE_MAJOR.* \(.*\)$/\1/'`
-
-            else if [ -f $PCRE/configure.in ]; then
-                ngx_pcre_ver=`grep PCRE_MAJOR= $PCRE/configure.in \
-                              | sed -e 's/^.*=\(.*\)$/\1/'`
-
-            else
-                ngx_pcre_ver=`grep pcre_major, $PCRE/configure.ac \
-                              | sed -e 's/^.*pcre_major,.*\[\(.*\)\].*$/\1/'`
-            fi
-            fi
-
-            echo " $ngx_pcre_ver major version found"
-
-            # to allow -ipo optimization we link with the *.o but not library
-
-            case "$ngx_pcre_ver" in
-                4|5)
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre.o"
-                ;;
-
-                6)
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
-                ;;
-
-                *)
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_chartables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_compile.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_exec.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_fullinfo.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_globals.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_tables.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_try_flipped.o"
-                    CORE_LIBS="$CORE_LIBS $PCRE/pcre_newline.o"
-                ;;
-
-            esac
-        ;;
-
         *)
-            have=NGX_PCRE . auto/have
-
-            if [ "$NGX_PLATFORM" = win32 ]; then
-                have=PCRE_STATIC . auto/have
-            fi
-
-            CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
             LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
             CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
         ;;

From mdounin at mdounin.ru  Fri Dec 24 22:11:31 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 24 Dec 2021 22:11:31 +0000
Subject: [nginx] PCRE2 library support.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/0b5f12d5c531
branches:  
changeset: 7981:0b5f12d5c531
user:      Maxim Dounin 
date:      Sat Dec 25 01:07:15 2021 +0300
description:
PCRE2 library support.

The PCRE2 library is now used by default if found, instead of the
original PCRE library.  If needed for some reason, this can be disabled
with the --without-pcre2 configure option.

To make it possible to specify paths to the library and include files
via --with-cc-opt / --with-ld-opt, the library is first tested without
any additional paths and options.  If this fails, the pcre2-config script
is used.

Similarly to the original PCRE library, it is now possible to build PCRE2
from sources with nginx configure, by using the --with-pcre= option.
It automatically detects if PCRE or PCRE2 sources are provided.

Note that compiling PCRE2 10.33 and later requires inttypes.h.  When
compiling on Windows with MSVC, inttypes.h is only available starting
with MSVC 2013.  In older versions some replacement needs to be provided
("echo '#include ' > pcre2-10.xx/src/inttypes.h" is good enough
for MSVC 2010).

The interface on nginx side remains unchanged.

diffstat:

 auto/lib/pcre/conf   |  107 +++++++++++++--
 auto/lib/pcre/make   |  152 +++++++++++++++++++---
 auto/options         |    3 +
 auto/summary         |    4 +-
 src/core/ngx_regex.c |  331 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 src/core/ngx_regex.h |   34 ++++-
 6 files changed, 576 insertions(+), 55 deletions(-)

diffs (888 lines):

diff -r 8007ea138d6a -r 0b5f12d5c531 auto/lib/pcre/conf
--- a/auto/lib/pcre/conf	Sat Dec 25 01:07:14 2021 +0300
+++ b/auto/lib/pcre/conf	Sat Dec 25 01:07:15 2021 +0300
@@ -5,29 +5,61 @@
 
 if [ $PCRE != NONE ]; then
 
-    have=NGX_PCRE . auto/have
+    if [ -f $PCRE/src/pcre2.h.generic ]; then
+
+        PCRE_LIBRARY=PCRE2
+
+        have=NGX_PCRE . auto/have
+        have=NGX_PCRE2 . auto/have
+
+        if [ "$NGX_PLATFORM" = win32 ]; then
+            have=PCRE2_STATIC . auto/have
+        fi
+
+        CORE_INCS="$CORE_INCS $PCRE/src/"
+        CORE_DEPS="$CORE_DEPS $PCRE/src/pcre2.h"
 
-    if [ "$NGX_PLATFORM" = win32 ]; then
-        have=PCRE_STATIC . auto/have
-    fi
+        case "$NGX_CC_NAME" in
+
+            msvc)
+                LINK_DEPS="$LINK_DEPS $PCRE/src/pcre2-8.lib"
+                CORE_LIBS="$CORE_LIBS $PCRE/src/pcre2-8.lib"
+            ;;
 
-    CORE_INCS="$CORE_INCS $PCRE"
-    CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
+            *)
+                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre2-8.a"
+                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre2-8.a"
+            ;;
 
-    case "$NGX_CC_NAME" in
+        esac
 
-        msvc | owc | bcc)
-            LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
-            CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
-        ;;
+    else
+
+        PCRE_LIBRARY=PCRE
+
+        have=NGX_PCRE . auto/have
+
+        if [ "$NGX_PLATFORM" = win32 ]; then
+            have=PCRE_STATIC . auto/have
+        fi
+
+        CORE_INCS="$CORE_INCS $PCRE"
+        CORE_DEPS="$CORE_DEPS $PCRE/pcre.h"
 
-        *)
-            LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
-            CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
-        ;;
+        case "$NGX_CC_NAME" in
+
+            msvc | owc | bcc)
+                LINK_DEPS="$LINK_DEPS $PCRE/pcre.lib"
+                CORE_LIBS="$CORE_LIBS $PCRE/pcre.lib"
+            ;;
 
-    esac
+            *)
+                LINK_DEPS="$LINK_DEPS $PCRE/.libs/libpcre.a"
+                CORE_LIBS="$CORE_LIBS $PCRE/.libs/libpcre.a"
+            ;;
 
+        esac
+    fi
 
     if [ $PCRE_JIT = YES ]; then
         have=NGX_HAVE_PCRE_JIT . auto/have
@@ -37,8 +69,48 @@ if [ $PCRE != NONE ]; then
 else
 
     if [ "$NGX_PLATFORM" != win32 ]; then
+        PCRE=NO
+    fi
 
-        PCRE=NO
+    if [ $PCRE = NO -a $PCRE2 != DISABLED ]; then
+
+        ngx_feature="PCRE2 library"
+        ngx_feature_name="NGX_PCRE2"
+        ngx_feature_run=no
+        ngx_feature_incs="#define PCRE2_CODE_UNIT_WIDTH 8
+                          #include "
+        ngx_feature_path=
+        ngx_feature_libs="-lpcre2-8"
+        ngx_feature_test="pcre2_code *re;
+                          re = pcre2_compile(NULL, 0, 0, NULL, NULL, NULL);
+                          if (re == NULL) return 1"
+        . auto/feature
+
+        if [ $ngx_found = no ]; then
+
+            # pcre2-config
+
+            ngx_pcre2_prefix=`pcre2-config --prefix 2>/dev/null`
+
+            if [ -n "$ngx_pcre2_prefix" ]; then
+                ngx_feature="PCRE2 library in $ngx_pcre2_prefix"
+                ngx_feature_path=`pcre2-config --cflags \
+                                  | sed -n -e 's/.*-I *\([^ ][^ ]*\).*/\1/p'`
+                ngx_feature_libs=`pcre2-config --libs8`
+                . auto/feature
+            fi
+        fi
+
+        if [ $ngx_found = yes ]; then
+            have=NGX_PCRE . auto/have
+            CORE_INCS="$CORE_INCS $ngx_feature_path"
+            CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
+            PCRE=YES
+            PCRE_LIBRARY=PCRE2
+        fi
+    fi
+
+    if [ $PCRE = NO ]; then
 
         ngx_feature="PCRE library"
         ngx_feature_name="NGX_PCRE"
@@ -114,6 +186,7 @@ else
             CORE_INCS="$CORE_INCS $ngx_feature_path"
             CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
             PCRE=YES
+            PCRE_LIBRARY=PCRE
         fi
 
         if [ $PCRE = YES ]; then
diff -r 8007ea138d6a -r 0b5f12d5c531 auto/lib/pcre/make
--- a/auto/lib/pcre/make	Sat Dec 25 01:07:14 2021 +0300
+++ b/auto/lib/pcre/make	Sat Dec 25 01:07:15 2021 +0300
@@ -3,36 +3,138 @@
 # Copyright (C) Nginx, Inc.
 
 
-case "$NGX_CC_NAME" in
+if [ $PCRE_LIBRARY = PCRE2 ]; then
+
+    # PCRE2
+
+    if [ $NGX_CC_NAME = msvc ]; then
+
+        # With PCRE2, it is not possible to compile all sources.
+        # Since list of source files changes between versions, we
+        # test files which might not be present.
 
-    msvc)
-        ngx_makefile=makefile.msvc
-        ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
-        ngx_pcre="PCRE=\"$PCRE\""
-    ;;
+        ngx_pcre_srcs="pcre2_auto_possess.c \
+                       pcre2_chartables.c \
+                       pcre2_compile.c \
+                       pcre2_config.c \
+                       pcre2_context.c \
+                       pcre2_dfa_match.c \
+                       pcre2_error.c \
+                       pcre2_jit_compile.c \
+                       pcre2_maketables.c \
+                       pcre2_match.c \
+                       pcre2_match_data.c \
+                       pcre2_newline.c \
+                       pcre2_ord2utf.c \
+                       pcre2_pattern_info.c \
+                       pcre2_string_utils.c \
+                       pcre2_study.c \
+                       pcre2_substitute.c \
+                       pcre2_substring.c \
+                       pcre2_tables.c \
+                       pcre2_ucd.c \
+                       pcre2_valid_utf.c \
+                       pcre2_xclass.c"
+
+        ngx_pcre_test="pcre2_convert.c \
+                       pcre2_extuni.c \
+                       pcre2_find_bracket.c \
+                       pcre2_script_run.c \
+                       pcre2_serialize.c"
+
+        for ngx_src in $ngx_pcre_test
+        do
+            if [ -f $PCRE/src/$ngx_src ]; then
+                ngx_pcre_srcs="$ngx_pcre_srcs $ngx_src"
+            fi
+        done
 
-    owc)
-        ngx_makefile=makefile.owc
-        ngx_opt="CPU_OPT=\"$CPU_OPT\""
-        ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
-    ;;
+        ngx_pcre_objs=`echo $ngx_pcre_srcs \
+            | sed -e "s#\([^ ]*\.\)c#\1$ngx_objext#g"`
+
+        ngx_pcre_srcs=`echo $ngx_pcre_srcs \
+            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
+        ngx_pcre_objs=`echo $ngx_pcre_objs \
+            | sed -e "s/  *\([^ ][^ ]*\)/$ngx_regex_cont\1/g"`
+
+        cat << END                                            >> $NGX_MAKEFILE
+
+PCRE_CFLAGS =	-O2 -Ob1 -Oi -Gs $LIBC $CPU_OPT
+PCRE_FLAGS =	-DHAVE_CONFIG_H -DPCRE2_STATIC -DPCRE2_CODE_UNIT_WIDTH=8 \\
+		-DHAVE_MEMMOVE
+
+PCRE_SRCS =	 $ngx_pcre_srcs
+PCRE_OBJS =	 $ngx_pcre_objs
+
+$PCRE/src/pcre2.h:
+	cd $PCRE/src \\
+	&& copy /y config.h.generic config.h \\
+	&& copy /y pcre2.h.generic pcre2.h \\
+	&& copy /y pcre2_chartables.c.dist pcre2_chartables.c
 
-    bcc)
-        ngx_makefile=makefile.bcc
-        ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
-        ngx_pcre=`echo \-DPCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
-    ;;
+$PCRE/src/pcre2-8.lib:	$PCRE/src/pcre2.h $NGX_MAKEFILE
+	cd $PCRE/src \\
+	&& cl -nologo -c \$(PCRE_CFLAGS) -I . \$(PCRE_FLAGS) \$(PCRE_SRCS) \\
+	&& link -lib -out:pcre2-8.lib -verbose:lib \$(PCRE_OBJS)
+
+END
+
+    else
+
+        cat << END                                            >> $NGX_MAKEFILE
+
+$PCRE/src/pcre2.h:	$PCRE/Makefile
 
-    *)
-        ngx_makefile=
-    ;;
+$PCRE/Makefile:	$NGX_MAKEFILE
+	cd $PCRE \\
+	&& if [ -f Makefile ]; then \$(MAKE) distclean; fi \\
+	&& CC="\$(CC)" CFLAGS="$PCRE_OPT" \\
+	./configure --disable-shared $PCRE_CONF_OPT
 
-esac
+$PCRE/.libs/libpcre2-8.a:	$PCRE/Makefile
+	cd $PCRE \\
+	&& \$(MAKE) libpcre2-8.la
+
+END
+
+    fi
 
 
-if [ -n "$ngx_makefile" ]; then
+else
+
+    # PCRE
+
+    case "$NGX_CC_NAME" in
+
+        msvc)
+            ngx_makefile=makefile.msvc
+            ngx_opt="CPU_OPT=\"$CPU_OPT\" LIBC=$LIBC"
+            ngx_pcre="PCRE=\"$PCRE\""
+        ;;
+
+        owc)
+            ngx_makefile=makefile.owc
+            ngx_opt="CPU_OPT=\"$CPU_OPT\""
+            ngx_pcre=`echo PCRE=\"$PCRE\" | sed -e "s/\//$ngx_regex_dirsep/g"`
+        ;;
 
-    cat << END                                                >> $NGX_MAKEFILE
+        bcc)
+            ngx_makefile=makefile.bcc
+            ngx_opt="-DCPU_OPT=\"$CPU_OPT\""
+            ngx_pcre=`echo \-DPCRE=\"$PCRE\" \
+                | sed -e "s/\//$ngx_regex_dirsep/g"`
+        ;;
+
+        *)
+            ngx_makefile=
+        ;;
+
+    esac
+
+
+    if [ -n "$ngx_makefile" ]; then
+
+        cat << END                                            >> $NGX_MAKEFILE
 
 `echo "$PCRE/pcre.lib:	$PCRE/pcre.h $NGX_MAKEFILE"			\
 	| sed -e "s/\//$ngx_regex_dirsep/g"`
@@ -43,9 +145,9 @@ if [ -n "$ngx_makefile" ]; then
 
 END
 
-else
+    else
 
-    cat << END                                                >> $NGX_MAKEFILE
+        cat << END                                            >> $NGX_MAKEFILE
 
 $PCRE/pcre.h:	$PCRE/Makefile
 
@@ -61,4 +163,6 @@ else
 
 END
 
+    fi
+
 fi
diff -r 8007ea138d6a -r 0b5f12d5c531 auto/options
--- a/auto/options	Sat Dec 25 01:07:14 2021 +0300
+++ b/auto/options	Sat Dec 25 01:07:15 2021 +0300
@@ -146,6 +146,7 @@ PCRE=NONE
 PCRE_OPT=
 PCRE_CONF_OPT=
 PCRE_JIT=NO
+PCRE2=YES
 
 USE_OPENSSL=NO
 OPENSSL=NONE
@@ -357,6 +358,7 @@ use the \"--with-mail_ssl_module\" optio
         --with-pcre=*)                   PCRE="$value"              ;;
         --with-pcre-opt=*)               PCRE_OPT="$value"          ;;
         --with-pcre-jit)                 PCRE_JIT=YES               ;;
+        --without-pcre2)                 PCRE2=DISABLED             ;;
 
         --with-openssl=*)                OPENSSL="$value"           ;;
         --with-openssl-opt=*)            OPENSSL_OPT="$value"       ;;
@@ -573,6 +575,7 @@ cat << END
   --with-pcre=DIR                    set path to PCRE library sources
   --with-pcre-opt=OPTIONS            set additional build options for PCRE
   --with-pcre-jit                    build PCRE with JIT compilation support
+  --without-pcre2                    do not use PCRE2 library
 
   --with-zlib=DIR                    set path to zlib library sources
   --with-zlib-opt=OPTIONS            set additional build options for zlib
diff -r 8007ea138d6a -r 0b5f12d5c531 auto/summary
--- a/auto/summary	Sat Dec 25 01:07:14 2021 +0300
+++ b/auto/summary	Sat Dec 25 01:07:15 2021 +0300
@@ -16,9 +16,9 @@ if [ $USE_PCRE = DISABLED ]; then
 
 else
     case $PCRE in
-        YES)   echo "  + using system PCRE library" ;;
+        YES)   echo "  + using system $PCRE_LIBRARY library" ;;
         NONE)  echo "  + PCRE library is not used" ;;
-        *)     echo "  + using PCRE library: $PCRE" ;;
+        *)     echo "  + using $PCRE_LIBRARY library: $PCRE" ;;
     esac
 fi
 
diff -r 8007ea138d6a -r 0b5f12d5c531 src/core/ngx_regex.c
--- a/src/core/ngx_regex.c	Sat Dec 25 01:07:14 2021 +0300
+++ b/src/core/ngx_regex.c	Sat Dec 25 01:07:15 2021 +0300
@@ -18,8 +18,13 @@ typedef struct {
 static ngx_inline void ngx_regex_malloc_init(ngx_pool_t *pool);
 static ngx_inline void ngx_regex_malloc_done(void);
 
+#if (NGX_PCRE2)
+static void * ngx_libc_cdecl ngx_regex_malloc(size_t size, void *data);
+static void ngx_libc_cdecl ngx_regex_free(void *p, void *data);
+#else
 static void * ngx_libc_cdecl ngx_regex_malloc(size_t size);
 static void ngx_libc_cdecl ngx_regex_free(void *p);
+#endif
 static void ngx_regex_cleanup(void *data);
 
 static ngx_int_t ngx_regex_module_init(ngx_cycle_t *cycle);
@@ -67,15 +72,24 @@ ngx_module_t  ngx_regex_module = {
 };
 
 
-static ngx_pool_t  *ngx_regex_pool;
-static ngx_list_t  *ngx_regex_studies;
+static ngx_pool_t             *ngx_regex_pool;
+static ngx_list_t             *ngx_regex_studies;
+static ngx_uint_t              ngx_regex_direct_alloc;
+
+#if (NGX_PCRE2)
+static pcre2_compile_context  *ngx_regex_compile_context;
+static pcre2_match_data       *ngx_regex_match_data;
+static ngx_uint_t              ngx_regex_match_data_size;
+#endif
 
 
 void
 ngx_regex_init(void)
 {
+#if !(NGX_PCRE2)
     pcre_malloc = ngx_regex_malloc;
     pcre_free = ngx_regex_free;
+#endif
 }
 
 
@@ -83,6 +97,7 @@ static ngx_inline void
 ngx_regex_malloc_init(ngx_pool_t *pool)
 {
     ngx_regex_pool = pool;
+    ngx_regex_direct_alloc = (pool == NULL) ? 1 : 0;
 }
 
 
@@ -90,9 +105,146 @@ static ngx_inline void
 ngx_regex_malloc_done(void)
 {
     ngx_regex_pool = NULL;
+    ngx_regex_direct_alloc = 0;
 }
 
 
+#if (NGX_PCRE2)
+
+ngx_int_t
+ngx_regex_compile(ngx_regex_compile_t *rc)
+{
+    int                     n, errcode;
+    char                   *p;
+    u_char                  errstr[128];
+    size_t                  erroff;
+    pcre2_code             *re;
+    ngx_regex_elt_t        *elt;
+    pcre2_general_context  *gctx;
+    pcre2_compile_context  *cctx;
+
+    if (ngx_regex_compile_context == NULL) {
+        /*
+         * Allocate a compile context if not yet allocated.  This uses
+         * direct allocations from heap, so the result can be cached
+         * even at runtime.
+         */
+
+        ngx_regex_malloc_init(NULL);
+
+        gctx = pcre2_general_context_create(ngx_regex_malloc, ngx_regex_free,
+                                            NULL);
+        if (gctx == NULL) {
+            ngx_regex_malloc_done();
+            goto nomem;
+        }
+
+        cctx = pcre2_compile_context_create(gctx);
+        if (cctx == NULL) {
+            pcre2_general_context_free(gctx);
+            ngx_regex_malloc_done();
+            goto nomem;
+        }
+
+        ngx_regex_compile_context = cctx;
+
+        pcre2_general_context_free(gctx);
+        ngx_regex_malloc_done();
+    }
+
+    ngx_regex_malloc_init(rc->pool);
+
+    re = pcre2_compile(rc->pattern.data, rc->pattern.len,
+                       (uint32_t) rc->options, &errcode, &erroff,
+                       ngx_regex_compile_context);
+
+    /* ensure that there is no current pool */
+    ngx_regex_malloc_done();
+
+    if (re == NULL) {
+        pcre2_get_error_message(errcode, errstr, 128);
+
+        if ((size_t) erroff == rc->pattern.len) {
+            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                              "pcre2_compile() failed: %s in \"%V\"",
+                               errstr, &rc->pattern)
+                          - rc->err.data;
+
+        } else {
+            rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                              "pcre2_compile() failed: %s in \"%V\" at \"%s\"",
+                               errstr, &rc->pattern, rc->pattern.data + erroff)
+                          - rc->err.data;
+        }
+
+        return NGX_ERROR;
+    }
+
+    rc->regex = re;
+
+    /* do not study at runtime */
+
+    if (ngx_regex_studies != NULL) {
+        elt = ngx_list_push(ngx_regex_studies);
+        if (elt == NULL) {
+            goto nomem;
+        }
+
+        elt->regex = rc->regex;
+        elt->name = rc->pattern.data;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_CAPTURECOUNT, &rc->captures);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_CAPTURECOUNT) failed: %d";
+        goto failed;
+    }
+
+    if (rc->captures == 0) {
+        return NGX_OK;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_NAMECOUNT, &rc->named_captures);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMECOUNT) failed: %d";
+        goto failed;
+    }
+
+    if (rc->named_captures == 0) {
+        return NGX_OK;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_NAMEENTRYSIZE, &rc->name_size);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMEENTRYSIZE) failed: %d";
+        goto failed;
+    }
+
+    n = pcre2_pattern_info(re, PCRE2_INFO_NAMETABLE, &rc->names);
+    if (n < 0) {
+        p = "pcre2_pattern_info(\"%V\", PCRE2_INFO_NAMETABLE) failed: %d";
+        goto failed;
+    }
+
+    return NGX_OK;
+
+failed:
+
+    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len, p, &rc->pattern, n)
+                  - rc->err.data;
+    return NGX_ERROR;
+
+nomem:
+
+    rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                               "regex \"%V\" compilation failed: no memory",
+                               &rc->pattern)
+                  - rc->err.data;
+    return NGX_ERROR;
+}
+
+#else
+
 ngx_int_t
 ngx_regex_compile(ngx_regex_compile_t *rc)
 {
@@ -195,6 +347,74 @@ nomem:
     return NGX_ERROR;
 }
 
+#endif
+
+
+#if (NGX_PCRE2)
+
+ngx_int_t
+ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures, ngx_uint_t size)
+{
+    size_t      *ov;
+    ngx_int_t    rc;
+    ngx_uint_t   n, i;
+
+    /*
+     * The pcre2_match() function might allocate memory for backtracking
+     * frames, typical allocations are from 40k and above.  So the allocator
+     * is configured to do direct allocations from heap during matching.
+     */
+
+    ngx_regex_malloc_init(NULL);
+
+    if (ngx_regex_match_data == NULL
+        || size > ngx_regex_match_data_size)
+    {
+        /*
+         * Allocate a match data if not yet allocated or smaller than
+         * needed.
+         */
+
+        if (ngx_regex_match_data) {
+            pcre2_match_data_free(ngx_regex_match_data);
+        }
+
+        ngx_regex_match_data_size = size;
+        ngx_regex_match_data = pcre2_match_data_create(size / 3, NULL);
+
+        if (ngx_regex_match_data == NULL) {
+            rc = PCRE2_ERROR_NOMEMORY;
+            goto failed;
+        }
+    }
+
+    rc = pcre2_match(re, s->data, s->len, 0, 0, ngx_regex_match_data, NULL);
+
+    if (rc < 0) {
+        goto failed;
+    }
+
+    n = pcre2_get_ovector_count(ngx_regex_match_data);
+    ov = pcre2_get_ovector_pointer(ngx_regex_match_data);
+
+    if (n > size / 3) {
+        n = size / 3;
+    }
+
+    for (i = 0; i < n; i++) {
+        captures[i * 2] = ov[i * 2];
+        captures[i * 2 + 1] = ov[i * 2 + 1];
+    }
+
+failed:
+
+    ngx_regex_malloc_done();
+
+    return rc;
+}
+
+#endif
+
 
 ngx_int_t
 ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log)
@@ -229,6 +449,35 @@ ngx_regex_exec_array(ngx_array_t *a, ngx
 }
 
 
+#if (NGX_PCRE2)
+
+static void * ngx_libc_cdecl
+ngx_regex_malloc(size_t size, void *data)
+{
+    if (ngx_regex_pool) {
+        return ngx_palloc(ngx_regex_pool, size);
+    }
+
+    if (ngx_regex_direct_alloc) {
+        return ngx_alloc(size, ngx_cycle->log);
+    }
+
+    return NULL;
+}
+
+
+static void ngx_libc_cdecl
+ngx_regex_free(void *p, void *data)
+{
+    if (ngx_regex_direct_alloc) {
+        ngx_free(p);
+    }
+
+    return;
+}
+
+#else
+
 static void * ngx_libc_cdecl
 ngx_regex_malloc(size_t size)
 {
@@ -246,11 +495,13 @@ ngx_regex_free(void *p)
     return;
 }
 
+#endif
+
 
 static void
 ngx_regex_cleanup(void *data)
 {
-#if (NGX_HAVE_PCRE_JIT)
+#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
     ngx_regex_conf_t *rcf = data;
 
     ngx_uint_t        i;
@@ -275,12 +526,17 @@ ngx_regex_cleanup(void *data)
         /*
          * The PCRE JIT compiler uses mmap for its executable codes, so we
          * have to explicitly call the pcre_free_study() function to free
-         * this memory.
+         * this memory.  In PCRE2, we call the pcre2_code_free() function
+         * for the same reason.
          */
 
+#if (NGX_PCRE2)
+        pcre2_code_free(elts[i].regex);
+#else
         if (elts[i].regex->extra != NULL) {
             pcre_free_study(elts[i].regex->extra);
         }
+#endif
     }
 #endif
 
@@ -290,6 +546,26 @@ ngx_regex_cleanup(void *data)
      */
 
     ngx_regex_studies = NULL;
+
+#if (NGX_PCRE2)
+
+    /*
+     * Free compile context and match data.  If needed at runtime by
+     * the new cycle, these will be re-allocated.
+     */
+
+    if (ngx_regex_compile_context) {
+        pcre2_compile_context_free(ngx_regex_compile_context);
+        ngx_regex_compile_context = NULL;
+    }
+
+    if (ngx_regex_match_data) {
+        pcre2_match_data_free(ngx_regex_match_data);
+        ngx_regex_match_data = NULL;
+        ngx_regex_match_data_size = 0;
+    }
+
+#endif
 }
 
 
@@ -297,7 +573,9 @@ static ngx_int_t
 ngx_regex_module_init(ngx_cycle_t *cycle)
 {
     int                opt;
+#if !(NGX_PCRE2)
     const char        *errstr;
+#endif
     ngx_uint_t         i;
     ngx_list_part_t   *part;
     ngx_regex_elt_t   *elts;
@@ -307,10 +585,16 @@ ngx_regex_module_init(ngx_cycle_t *cycle
 
     rcf = (ngx_regex_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_regex_module);
 
-#if (NGX_HAVE_PCRE_JIT)
+#if (NGX_PCRE2 || NGX_HAVE_PCRE_JIT)
+
     if (rcf->pcre_jit) {
+#if (NGX_PCRE2)
+        opt = 1;
+#else
         opt = PCRE_STUDY_JIT_COMPILE;
+#endif
     }
+
 #endif
 
     ngx_regex_malloc_init(cycle->pool);
@@ -330,6 +614,23 @@ ngx_regex_module_init(ngx_cycle_t *cycle
             i = 0;
         }
 
+#if (NGX_PCRE2)
+
+        if (opt) {
+            int  n;
+
+            n = pcre2_jit_compile(elts[i].regex, PCRE2_JIT_COMPLETE);
+
+            if (n != 0) {
+                ngx_log_error(NGX_LOG_INFO, cycle->log, 0,
+                              "pcre2_jit_compile() failed: %d in \"%s\", "
+                              "ignored",
+                              n, elts[i].name);
+            }
+        }
+
+#else
+
         elts[i].regex->extra = pcre_study(elts[i].regex->code, opt, &errstr);
 
         if (errstr != NULL) {
@@ -353,11 +654,15 @@ ngx_regex_module_init(ngx_cycle_t *cycle
             }
         }
 #endif
+#endif
     }
 
     ngx_regex_malloc_done();
 
     ngx_regex_studies = NULL;
+#if (NGX_PCRE2)
+    ngx_regex_compile_context = NULL;
+#endif
 
     return NGX_OK;
 }
@@ -415,7 +720,21 @@ ngx_regex_pcre_jit(ngx_conf_t *cf, void 
         return NGX_CONF_OK;
     }
 
-#if (NGX_HAVE_PCRE_JIT)
+#if (NGX_PCRE2)
+    {
+    int       r;
+    uint32_t  jit;
+
+    jit = 0;
+    r = pcre2_config(PCRE2_CONFIG_JIT, &jit);
+
+    if (r != 0 || jit != 1) {
+        ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
+                           "PCRE2 library does not support JIT");
+        *fp = 0;
+    }
+    }
+#elif (NGX_HAVE_PCRE_JIT)
     {
     int  jit, r;
 
diff -r 8007ea138d6a -r 0b5f12d5c531 src/core/ngx_regex.h
--- a/src/core/ngx_regex.h	Sat Dec 25 01:07:14 2021 +0300
+++ b/src/core/ngx_regex.h	Sat Dec 25 01:07:15 2021 +0300
@@ -12,19 +12,31 @@
 #include 
 #include 
 
+
+#if (NGX_PCRE2)
+
+#define PCRE2_CODE_UNIT_WIDTH  8
+#include 
+
+#define NGX_REGEX_NO_MATCHED   PCRE2_ERROR_NOMATCH   /* -1 */
+#define NGX_REGEX_CASELESS     PCRE2_CASELESS
+
+typedef pcre2_code  ngx_regex_t;
+
+#else
+
 #include 
 
-
-#define NGX_REGEX_NO_MATCHED  PCRE_ERROR_NOMATCH   /* -1 */
-
-#define NGX_REGEX_CASELESS    PCRE_CASELESS
-
+#define NGX_REGEX_NO_MATCHED   PCRE_ERROR_NOMATCH    /* -1 */
+#define NGX_REGEX_CASELESS     PCRE_CASELESS
 
 typedef struct {
     pcre        *code;
     pcre_extra  *extra;
 } ngx_regex_t;
 
+#endif
+
 
 typedef struct {
     ngx_str_t     pattern;
@@ -49,10 +61,20 @@ typedef struct {
 void ngx_regex_init(void);
 ngx_int_t ngx_regex_compile(ngx_regex_compile_t *rc);
 
+#if (NGX_PCRE2)
+
+ngx_int_t ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures,
+    ngx_uint_t size);
+#define ngx_regex_exec_n       "pcre2_match()"
+
+#else
+
 #define ngx_regex_exec(re, s, captures, size)                                \
     pcre_exec(re->code, re->extra, (const char *) (s)->data, (s)->len, 0, 0, \
               captures, size)
-#define ngx_regex_exec_n      "pcre_exec()"
+#define ngx_regex_exec_n       "pcre_exec()"
+
+#endif
 
 ngx_int_t ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log);
 

From mdounin at mdounin.ru  Fri Dec 24 22:11:34 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 24 Dec 2021 22:11:34 +0000
Subject: [nginx] PCRE2 and PCRE binary compatibility.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/fbbb5ce52995
branches:  
changeset: 7982:fbbb5ce52995
user:      Maxim Dounin 
date:      Sat Dec 25 01:07:16 2021 +0300
description:
PCRE2 and PCRE binary compatibility.

With this change, dynamic modules using nginx regex interface can be used
regardless of the variant of the PCRE library nginx was compiled with.

If a module is compiled with different PCRE library variant, in case of
ngx_regex_exec() errors it will report wrong function name in error
messages.  This is believed to be tolerable, given that fixing this will
require interface changes.

diffstat:

 src/core/ngx_regex.c |  46 ++++++++++++++++++++++++++++++++++++++++++----
 src/core/ngx_regex.h |  17 ++++++-----------
 2 files changed, 48 insertions(+), 15 deletions(-)

diffs (139 lines):

diff -r 0b5f12d5c531 -r fbbb5ce52995 src/core/ngx_regex.c
--- a/src/core/ngx_regex.c	Sat Dec 25 01:07:15 2021 +0300
+++ b/src/core/ngx_regex.c	Sat Dec 25 01:07:16 2021 +0300
@@ -118,6 +118,7 @@ ngx_regex_compile(ngx_regex_compile_t *r
     char                   *p;
     u_char                  errstr[128];
     size_t                  erroff;
+    uint32_t                options;
     pcre2_code             *re;
     ngx_regex_elt_t        *elt;
     pcre2_general_context  *gctx;
@@ -152,11 +153,24 @@ ngx_regex_compile(ngx_regex_compile_t *r
         ngx_regex_malloc_done();
     }
 
+    options = 0;
+
+    if (rc->options & NGX_REGEX_CASELESS) {
+        options |= PCRE2_CASELESS;
+    }
+
+    if (rc->options & ~NGX_REGEX_CASELESS) {
+        rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                            "regex \"%V\" compilation failed: invalid options",
+                            &rc->pattern)
+                      - rc->err.data;
+        return NGX_ERROR;
+    }
+
     ngx_regex_malloc_init(rc->pool);
 
-    re = pcre2_compile(rc->pattern.data, rc->pattern.len,
-                       (uint32_t) rc->options, &errcode, &erroff,
-                       ngx_regex_compile_context);
+    re = pcre2_compile(rc->pattern.data, rc->pattern.len, options,
+                       &errcode, &erroff, ngx_regex_compile_context);
 
     /* ensure that there is no current pool */
     ngx_regex_malloc_done();
@@ -252,11 +266,26 @@ ngx_regex_compile(ngx_regex_compile_t *r
     char             *p;
     pcre             *re;
     const char       *errstr;
+    ngx_uint_t        options;
     ngx_regex_elt_t  *elt;
 
+    options = 0;
+
+    if (rc->options & NGX_REGEX_CASELESS) {
+        options |= PCRE_CASELESS;
+    }
+
+    if (rc->options & ~NGX_REGEX_CASELESS) {
+        rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
+                            "regex \"%V\" compilation failed: invalid options",
+                            &rc->pattern)
+                      - rc->err.data;
+        return NGX_ERROR;
+    }
+
     ngx_regex_malloc_init(rc->pool);
 
-    re = pcre_compile((const char *) rc->pattern.data, (int) rc->options,
+    re = pcre_compile((const char *) rc->pattern.data, (int) options,
                       &errstr, &erroff, NULL);
 
     /* ensure that there is no current pool */
@@ -413,6 +442,15 @@ failed:
     return rc;
 }
 
+#else
+
+ngx_int_t
+ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures, ngx_uint_t size)
+{
+    return pcre_exec(re->code, re->extra, (const char *) s->data, s->len,
+                     0, 0, captures, size);
+}
+
 #endif
 
 
diff -r 0b5f12d5c531 -r fbbb5ce52995 src/core/ngx_regex.h
--- a/src/core/ngx_regex.h	Sat Dec 25 01:07:15 2021 +0300
+++ b/src/core/ngx_regex.h	Sat Dec 25 01:07:16 2021 +0300
@@ -19,7 +19,6 @@
 #include 
 
 #define NGX_REGEX_NO_MATCHED   PCRE2_ERROR_NOMATCH   /* -1 */
-#define NGX_REGEX_CASELESS     PCRE2_CASELESS
 
 typedef pcre2_code  ngx_regex_t;
 
@@ -28,7 +27,6 @@ typedef pcre2_code  ngx_regex_t;
 #include 
 
 #define NGX_REGEX_NO_MATCHED   PCRE_ERROR_NOMATCH    /* -1 */
-#define NGX_REGEX_CASELESS     PCRE_CASELESS
 
 typedef struct {
     pcre        *code;
@@ -38,10 +36,13 @@ typedef struct {
 #endif
 
 
+#define NGX_REGEX_CASELESS     0x00000001
+
+
 typedef struct {
     ngx_str_t     pattern;
     ngx_pool_t   *pool;
-    ngx_int_t     options;
+    ngx_uint_t    options;
 
     ngx_regex_t  *regex;
     int           captures;
@@ -61,19 +62,13 @@ typedef struct {
 void ngx_regex_init(void);
 ngx_int_t ngx_regex_compile(ngx_regex_compile_t *rc);
 
-#if (NGX_PCRE2)
-
 ngx_int_t ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures,
     ngx_uint_t size);
+
+#if (NGX_PCRE2)
 #define ngx_regex_exec_n       "pcre2_match()"
-
 #else
-
-#define ngx_regex_exec(re, s, captures, size)                                \
-    pcre_exec(re->code, re->extra, (const char *) (s)->data, (s)->len, 0, 0, \
-              captures, size)
 #define ngx_regex_exec_n       "pcre_exec()"
-
 #endif
 
 ngx_int_t ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log);

From mdounin at mdounin.ru  Fri Dec 24 22:11:37 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 24 Dec 2021 22:11:37 +0000
Subject: [nginx] Core: added NGX_REGEX_MULTILINE for 3rd party modules.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/d07456044b61
branches:  
changeset: 7983:d07456044b61
user:      Maxim Dounin 
date:      Sat Dec 25 01:07:18 2021 +0300
description:
Core: added NGX_REGEX_MULTILINE for 3rd party modules.

Notably, NAXSI is known to misuse ngx_regex_compile() with rc.options set
to PCRE_CASELESS | PCRE_MULTILINE.  With PCRE2 support, and notably binary
compatibility changes, it is no longer possible to set PCRE[2]_MULTILINE
option without using proper interface.  To facilitate correct usage,
this change adds the NGX_REGEX_MULTILINE option.

diffstat:

 src/core/ngx_regex.c |  12 ++++++++++--
 src/core/ngx_regex.h |   1 +
 2 files changed, 11 insertions(+), 2 deletions(-)

diffs (40 lines):

diff -r fbbb5ce52995 -r d07456044b61 src/core/ngx_regex.c
--- a/src/core/ngx_regex.c	Sat Dec 25 01:07:16 2021 +0300
+++ b/src/core/ngx_regex.c	Sat Dec 25 01:07:18 2021 +0300
@@ -159,7 +159,11 @@ ngx_regex_compile(ngx_regex_compile_t *r
         options |= PCRE2_CASELESS;
     }
 
-    if (rc->options & ~NGX_REGEX_CASELESS) {
+    if (rc->options & NGX_REGEX_MULTILINE) {
+        options |= PCRE2_MULTILINE;
+    }
+
+    if (rc->options & ~(NGX_REGEX_CASELESS|NGX_REGEX_MULTILINE)) {
         rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
                             "regex \"%V\" compilation failed: invalid options",
                             &rc->pattern)
@@ -275,7 +279,11 @@ ngx_regex_compile(ngx_regex_compile_t *r
         options |= PCRE_CASELESS;
     }
 
-    if (rc->options & ~NGX_REGEX_CASELESS) {
+    if (rc->options & NGX_REGEX_MULTILINE) {
+        options |= PCRE_MULTILINE;
+    }
+
+    if (rc->options & ~(NGX_REGEX_CASELESS|NGX_REGEX_MULTILINE)) {
         rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
                             "regex \"%V\" compilation failed: invalid options",
                             &rc->pattern)
diff -r fbbb5ce52995 -r d07456044b61 src/core/ngx_regex.h
--- a/src/core/ngx_regex.h	Sat Dec 25 01:07:16 2021 +0300
+++ b/src/core/ngx_regex.h	Sat Dec 25 01:07:18 2021 +0300
@@ -37,6 +37,7 @@ typedef struct {
 
 
 #define NGX_REGEX_CASELESS     0x00000001
+#define NGX_REGEX_MULTILINE    0x00000002
 
 
 typedef struct {

From mdounin at mdounin.ru  Fri Dec 24 22:13:15 2021
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sat, 25 Dec 2021 01:13:15 +0300
Subject: [PATCH] Events: fixed balancing between workers with EPOLLEXCLUSIVE
Message-ID: <4fa260f60bac03037017.1640383995@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin 
# Date 1640383910 -10800
#      Sat Dec 25 01:11:50 2021 +0300
# Node ID 4fa260f60bac03037017a33672da1ac75ac31408
# Parent  d07456044b611848abe244d65b585159037f8980
Events: fixed balancing between workers with EPOLLEXCLUSIVE.

Linux with EPOLLEXCLUSIVE usually notifies only the process which was first
to add the listening socket to the epoll instance.  As a result most of the
connections are handled by the first worker process (ticket #2285).  To fix
this, we re-add the socket periodically, so other workers will get a chance
to accept connections.

diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
--- a/src/event/ngx_event.c
+++ b/src/event/ngx_event.c
@@ -55,6 +55,7 @@ ngx_uint_t            ngx_accept_events;
 ngx_uint_t            ngx_accept_mutex_held;
 ngx_msec_t            ngx_accept_mutex_delay;
 ngx_int_t             ngx_accept_disabled;
+ngx_uint_t            ngx_use_exclusive_accept;
 
 
 #if (NGX_STAT_STUB)
@@ -644,6 +645,8 @@ ngx_event_process_init(ngx_cycle_t *cycl
 
 #endif
 
+    ngx_use_exclusive_accept = 0;
+
     ngx_queue_init(&ngx_posted_accept_events);
     ngx_queue_init(&ngx_posted_next_events);
     ngx_queue_init(&ngx_posted_events);
@@ -889,6 +892,8 @@ ngx_event_process_init(ngx_cycle_t *cycl
         if ((ngx_event_flags & NGX_USE_EPOLL_EVENT)
             && ccf->worker_processes > 1)
         {
+            ngx_use_exclusive_accept = 1;
+
             if (ngx_add_event(rev, NGX_READ_EVENT, NGX_EXCLUSIVE_EVENT)
                 == NGX_ERROR)
             {
diff --git a/src/event/ngx_event.h b/src/event/ngx_event.h
--- a/src/event/ngx_event.h
+++ b/src/event/ngx_event.h
@@ -466,6 +466,7 @@ extern ngx_uint_t             ngx_accept
 extern ngx_uint_t             ngx_accept_mutex_held;
 extern ngx_msec_t             ngx_accept_mutex_delay;
 extern ngx_int_t              ngx_accept_disabled;
+extern ngx_uint_t             ngx_use_exclusive_accept;
 
 
 #if (NGX_STAT_STUB)
diff --git a/src/event/ngx_event_accept.c b/src/event/ngx_event_accept.c
--- a/src/event/ngx_event_accept.c
+++ b/src/event/ngx_event_accept.c
@@ -11,6 +11,9 @@
 
 
 static ngx_int_t ngx_disable_accept_events(ngx_cycle_t *cycle, ngx_uint_t all);
+#if (NGX_HAVE_EPOLLEXCLUSIVE)
+static void ngx_reorder_accept_events(ngx_listening_t *ls);
+#endif
 static void ngx_close_accepted_connection(ngx_connection_t *c);
 
 
@@ -314,6 +317,10 @@ ngx_event_accept(ngx_event_t *ev)
         }
 
     } while (ev->available);
+
+#if (NGX_HAVE_EPOLLEXCLUSIVE)
+    ngx_reorder_accept_events(ls);
+#endif
 }
 
 
@@ -420,6 +427,55 @@ ngx_disable_accept_events(ngx_cycle_t *c
 }
 
 
+#if (NGX_HAVE_EPOLLEXCLUSIVE)
+
+static void
+ngx_reorder_accept_events(ngx_listening_t *ls)
+{
+    ngx_connection_t  *c;
+
+    /*
+     * Linux with EPOLLEXCLUSIVE usually notifies only the process which
+     * was first to add the listening socket to the epoll instance.  As
+     * a result most of the connections are handled by the first worker
+     * process.  To fix this, we re-add the socket periodically, so other
+     * workers will get a chance to accept connections.
+     */
+
+    if (!ngx_use_exclusive_accept) {
+        return;
+    }
+
+#if (NGX_HAVE_REUSEPORT)
+
+    if (ls->reuseport) {
+        return;
+    }
+
+#endif
+
+    c = ls->connection;
+
+    if (c->requests++ % 16 != 0) {
+        return;
+    }
+
+    if (ngx_del_event(c->read, NGX_READ_EVENT, NGX_DISABLE_EVENT)
+        == NGX_ERROR)
+    {
+        return;
+    }
+
+    if (ngx_add_event(c->read, NGX_READ_EVENT, NGX_EXCLUSIVE_EVENT)
+        == NGX_ERROR)
+    {
+        return;
+    }
+}
+
+#endif
+
+
 static void
 ngx_close_accepted_connection(ngx_connection_t *c)
 {


From osa at freebsd.org.ru  Fri Dec 24 23:07:21 2021
From: osa at freebsd.org.ru (Sergey A. Osokin)
Date: Sat, 25 Dec 2021 02:07:21 +0300
Subject: PCRE1/PCRE2 support for FreeBSD's www/nginx-devel
Message-ID: 

Hi,

upcoming nginx release 1.21.5 will support both version of PCRE
library.
The attached patch enables such support for www/nginx-devel in
FreeBSD ports tree.

I appreciate your testing and feedback.

Thank you.

-- 
Sergey A. Osokin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx-devel.diff
Type: text/x-diff
Size: 4935 bytes
Desc: not available
URL: 

From mat999 at gmail.com  Sat Dec 25 11:17:38 2021
From: mat999 at gmail.com (Mathew Heard)
Date: Sat, 25 Dec 2021 22:17:38 +1100
Subject: [nginx] PCRE2 and PCRE binary compatibility.
In-Reply-To: 
References: 
Message-ID: 

By the way have you seen sregex ? Given its built with many of the
same principles as nginx and PREG(1/2) compatible maybe it might be of
interest?

And Merry Christmas for those who celebrate.

On Sat, 25 Dec 2021 at 09:11, Maxim Dounin  wrote:
>
> details:   https://hg.nginx.org/nginx/rev/fbbb5ce52995
> branches:
> changeset: 7982:fbbb5ce52995
> user:      Maxim Dounin 
> date:      Sat Dec 25 01:07:16 2021 +0300
> description:
> PCRE2 and PCRE binary compatibility.
>
> With this change, dynamic modules using nginx regex interface can be used
> regardless of the variant of the PCRE library nginx was compiled with.
>
> If a module is compiled with different PCRE library variant, in case of
> ngx_regex_exec() errors it will report wrong function name in error
> messages.  This is believed to be tolerable, given that fixing this will
> require interface changes.
>
> diffstat:
>
>  src/core/ngx_regex.c |  46 ++++++++++++++++++++++++++++++++++++++++++----
>  src/core/ngx_regex.h |  17 ++++++-----------
>  2 files changed, 48 insertions(+), 15 deletions(-)
>
> diffs (139 lines):
>
> diff -r 0b5f12d5c531 -r fbbb5ce52995 src/core/ngx_regex.c
> --- a/src/core/ngx_regex.c      Sat Dec 25 01:07:15 2021 +0300
> +++ b/src/core/ngx_regex.c      Sat Dec 25 01:07:16 2021 +0300
> @@ -118,6 +118,7 @@ ngx_regex_compile(ngx_regex_compile_t *r
>      char                   *p;
>      u_char                  errstr[128];
>      size_t                  erroff;
> +    uint32_t                options;
>      pcre2_code             *re;
>      ngx_regex_elt_t        *elt;
>      pcre2_general_context  *gctx;
> @@ -152,11 +153,24 @@ ngx_regex_compile(ngx_regex_compile_t *r
>          ngx_regex_malloc_done();
>      }
>
> +    options = 0;
> +
> +    if (rc->options & NGX_REGEX_CASELESS) {
> +        options |= PCRE2_CASELESS;
> +    }
> +
> +    if (rc->options & ~NGX_REGEX_CASELESS) {
> +        rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
> +                            "regex \"%V\" compilation failed: invalid options",
> +                            &rc->pattern)
> +                      - rc->err.data;
> +        return NGX_ERROR;
> +    }
> +
>      ngx_regex_malloc_init(rc->pool);
>
> -    re = pcre2_compile(rc->pattern.data, rc->pattern.len,
> -                       (uint32_t) rc->options, &errcode, &erroff,
> -                       ngx_regex_compile_context);
> +    re = pcre2_compile(rc->pattern.data, rc->pattern.len, options,
> +                       &errcode, &erroff, ngx_regex_compile_context);
>
>      /* ensure that there is no current pool */
>      ngx_regex_malloc_done();
> @@ -252,11 +266,26 @@ ngx_regex_compile(ngx_regex_compile_t *r
>      char             *p;
>      pcre             *re;
>      const char       *errstr;
> +    ngx_uint_t        options;
>      ngx_regex_elt_t  *elt;
>
> +    options = 0;
> +
> +    if (rc->options & NGX_REGEX_CASELESS) {
> +        options |= PCRE_CASELESS;
> +    }
> +
> +    if (rc->options & ~NGX_REGEX_CASELESS) {
> +        rc->err.len = ngx_snprintf(rc->err.data, rc->err.len,
> +                            "regex \"%V\" compilation failed: invalid options",
> +                            &rc->pattern)
> +                      - rc->err.data;
> +        return NGX_ERROR;
> +    }
> +
>      ngx_regex_malloc_init(rc->pool);
>
> -    re = pcre_compile((const char *) rc->pattern.data, (int) rc->options,
> +    re = pcre_compile((const char *) rc->pattern.data, (int) options,
>                        &errstr, &erroff, NULL);
>
>      /* ensure that there is no current pool */
> @@ -413,6 +442,15 @@ failed:
>      return rc;
>  }
>
> +#else
> +
> +ngx_int_t
> +ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures, ngx_uint_t size)
> +{
> +    return pcre_exec(re->code, re->extra, (const char *) s->data, s->len,
> +                     0, 0, captures, size);
> +}
> +
>  #endif
>
>
> diff -r 0b5f12d5c531 -r fbbb5ce52995 src/core/ngx_regex.h
> --- a/src/core/ngx_regex.h      Sat Dec 25 01:07:15 2021 +0300
> +++ b/src/core/ngx_regex.h      Sat Dec 25 01:07:16 2021 +0300
> @@ -19,7 +19,6 @@
>  #include 
>
>  #define NGX_REGEX_NO_MATCHED   PCRE2_ERROR_NOMATCH   /* -1 */
> -#define NGX_REGEX_CASELESS     PCRE2_CASELESS
>
>  typedef pcre2_code  ngx_regex_t;
>
> @@ -28,7 +27,6 @@ typedef pcre2_code  ngx_regex_t;
>  #include 
>
>  #define NGX_REGEX_NO_MATCHED   PCRE_ERROR_NOMATCH    /* -1 */
> -#define NGX_REGEX_CASELESS     PCRE_CASELESS
>
>  typedef struct {
>      pcre        *code;
> @@ -38,10 +36,13 @@ typedef struct {
>  #endif
>
>
> +#define NGX_REGEX_CASELESS     0x00000001
> +
> +
>  typedef struct {
>      ngx_str_t     pattern;
>      ngx_pool_t   *pool;
> -    ngx_int_t     options;
> +    ngx_uint_t    options;
>
>      ngx_regex_t  *regex;
>      int           captures;
> @@ -61,19 +62,13 @@ typedef struct {
>  void ngx_regex_init(void);
>  ngx_int_t ngx_regex_compile(ngx_regex_compile_t *rc);
>
> -#if (NGX_PCRE2)
> -
>  ngx_int_t ngx_regex_exec(ngx_regex_t *re, ngx_str_t *s, int *captures,
>      ngx_uint_t size);
> +
> +#if (NGX_PCRE2)
>  #define ngx_regex_exec_n       "pcre2_match()"
> -
>  #else
> -
> -#define ngx_regex_exec(re, s, captures, size)                                \
> -    pcre_exec(re->code, re->extra, (const char *) (s)->data, (s)->len, 0, 0, \
> -              captures, size)
>  #define ngx_regex_exec_n       "pcre_exec()"
> -
>  #endif
>
>  ngx_int_t ngx_regex_exec_array(ngx_array_t *a, ngx_str_t *s, ngx_log_t *log);
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

From mdounin at mdounin.ru  Mon Dec 27 14:08:49 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 27 Dec 2021 17:08:49 +0300
Subject: [PATCH] HTTP: keepalive_graceful_close support
In-Reply-To: 
References: 
Message-ID: 

Hello!

On Fri, Dec 17, 2021 at 01:59:40PM +0800, Youlin Feng wrote:

> # HG changeset patch
> # User Youlin Feng 
> # Date 1639718067 -28800
> #      Fri Dec 17 13:14:27 2021 +0800
> # Node ID 54db7272bb0c9040eaf657f92bb02c9147d927e6
> # Parent  a7a77549265ef46f1f0fdb3897f4beabf9e09c40
> HTTP: keepalive_graceful_close support.
> 
> Previously, keepalived connections will be suddenly closed during worker
> process exiting, without notification to the client. The client may be
> sending a request when the connection is closed, resulting in an error.
> 
> With "keepalive_graceful_close on;", the client will receive an
> "Connection: close" response header in the last request, then the
> connection can be closed gracefully.

Thanks for the patch.

First of all, you may want to clarify which problem you are trying 
to solve.  Note that clients are expected to be prepared for a 
connection close by the server, and retry requests as appropriate.  
If the client cannot handle this, probably there is a room for 
improvement in the client.  Further, there is an unavoidable race, 
and a client which cannot handle connection close is likely to see 
errors regardless of server behaviour.  See ticket #1022 
(https://trac.nginx.org/nginx/ticket/1022) for detailed 
explanation.

What probably nginx should do here is to disable keepalive and 
avoid sending "Connection: keep-alive" if it already knows that 
keepalive cannot be used when sending response headers.  This 
should reduce unneeded request failures and retries at no 
additional cost.

Patch:

# HG changeset patch
# User Maxim Dounin 
# Date 1640613750 -10800
#      Mon Dec 27 17:02:30 2021 +0300
# Node ID 0430ee0554a19281abcc074981702ac201f2922e
# Parent  4fa260f60bac03037017a33672da1ac75ac31408
Avoid sending "Connection: keep-alive" when shutting down.

When a worker process is shutting down, keepalive is not used: this is checked
before the ngx_http_set_keepalive() call in ngx_http_finalize_connection().
Yet the "Connection: keep-alive" header was still sent, even if we know that
the worker process is shutting down, potentially resulting in additional
requests being sent to the connection which is going to be closed anyway.
While clients are expected to be able to handle asynchronous close events
(see ticket #1022), it is certainly possible to send the "Connection: close"
header instead, informing the client that the connection is going to be closed
and potentially saving some unneeded work.

With this change, we additionally check for worker process shutdown just
before sending response headers, and disable keepalive accordingly.

diff --git a/src/http/ngx_http_header_filter_module.c b/src/http/ngx_http_header_filter_module.c
--- a/src/http/ngx_http_header_filter_module.c
+++ b/src/http/ngx_http_header_filter_module.c
@@ -197,6 +197,10 @@ ngx_http_header_filter(ngx_http_request_
         }
     }
 
+    if (r->keepalive && (ngx_terminate || ngx_exiting)) {
+        r->keepalive = 0;
+    }
+
     len = sizeof("HTTP/1.x ") - 1 + sizeof(CRLF) - 1
           /* the end of the header */
           + sizeof(CRLF) - 1;

-- 
Maxim Dounin
http://mdounin.ru/

From pluknet at nginx.com  Mon Dec 27 14:12:39 2021
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 27 Dec 2021 17:12:39 +0300
Subject: [PATCH 2 of 4] Simplified sendfile(SF_NODISKIO) usage
In-Reply-To: 
References: 
 <4a954e89b1ae8539bbe0.1636604470@vm-bsd.mdounin.ru>
 <20211130120503.552wawqq52udi3ag@MacBook-Air-Sergey.local>
  
Message-ID: <1FA7B6FA-A7A3-49F9-89B8-6CF9C2610508@nginx.com>


> On 14 Dec 2021, at 18:28, Maxim Dounin  wrote:
> 
> Hello!
> 
> On Tue, Dec 14, 2021 at 05:15:47PM +0300, Maxim Dounin wrote:
> 
>> On Tue, Nov 30, 2021 at 03:05:03PM +0300, Sergey Kandaurov wrote:
>> 
>>> On Thu, Nov 11, 2021 at 07:21:10AM +0300, Maxim Dounin wrote:
>>>> # HG changeset patch
>>>> # User Maxim Dounin 
>>>> # Date 1636603886 -10800
>>>> #      Thu Nov 11 07:11:26 2021 +0300
>>>> # Node ID 4a954e89b1ae8539bbe08c5afc1d5c9828d82d6f
>>>> # Parent  0fb75ef9dbca698e5e855145cf6a12180a36d400
>>>> Simplified sendfile(SF_NODISKIO) usage.
>>>> 
>>>> Starting with FreeBSD 11, there is no need to use AIO operations to preload
>>>> data into cache for sendfile(SF_NODISKIO) to work.  Instead, sendfile()
>>>> handles non-blocking loading data from disk by itself.  It still can, however,
>>>> return EBUSY if a page is already being loaded (for example, by a different
>>>> process).  If this happens, we now post an event for the next event loop
>>>> iteration, so sendfile() is retried "after a short period", as manpage
>>>> recommends.
>>>> 
>>>> The limit of the number of EBUSY tolerated without any progress is preserved,
>>>> but now it does not result in an alert, since on an idle system event loop
>>>> iteration might be very short and EBUSY can happen many times in a row.
>>>> Instead, SF_NODISKIO is simply disabled for one call once the limit is
>>>> reached.
>>>> 
>>>> With this change, sendfile(SF_NODISKIO) is now used automatically as long as
>>>> sendfile() is enabled, and no longer requires "aio on;".
>>>> 
>>>> diff --git a/auto/os/freebsd b/auto/os/freebsd
>>>> --- a/auto/os/freebsd
>>>> +++ b/auto/os/freebsd
>>>> @@ -44,12 +44,10 @@ if [ $osreldate -gt 300007 ]; then
>>>>     CORE_SRCS="$CORE_SRCS $FREEBSD_SENDFILE_SRCS"
>>>> fi
>>>> 
>>>> -if [ $NGX_FILE_AIO = YES ]; then
>>>> -    if [ $osreldate -gt 502103 ]; then
>>>> -        echo " + sendfile()'s SF_NODISKIO found"
>>>> +if [ $osreldate -gt 1100000 ]; then
>>>> +    echo " + sendfile()'s SF_NODISKIO found"
>>>> 
>>>> -        have=NGX_HAVE_AIO_SENDFILE . auto/have
>>>> -    fi
>>>> +    have=NGX_HAVE_SENDFILE_NODISKIO . auto/have
>>>> fi
>>>> 
>>>> # POSIX semaphores
>>> 
>>> We could check the exact __FreeBSD_version number 1100093 that was
>>> at the time the new sendfile() appeared, which is more accurate.
>>> 
>>> https://cgit.freebsd.org/src/commit/?id=2bab0c553588
>>> https://cgit.freebsd.org/src/tree/sys/sys/param.h?id=2bab0c553588#n48
>>> 
>>> Unfortunately, it was not bumped (same as with SF_NODISKIO in 5.2.1).
>> 
>> Yes, probably.  But the next version bump would be more 
>> appropriate, changed to 1100093.  Similarly to 502103, which is 
>> the next version after SF_NODISKIO introduction:
>> 
>> https://cgit.freebsd.org/src/commit/?id=b49d824e8bc1
>> https://cgit.freebsd.org/src/tree/sys/sys/param.h?id=b49d824e8bc1
>> 

I'm fine with it.

>> [...]
>> 
>>>> -static ngx_int_t
>>>> -ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx, ngx_file_t *file)
>>>> -{
>>>> -    ngx_event_aio_t  *aio;
>>>> -
>>>> -    if (file->aio == NULL && ngx_file_aio_init(file, ctx->pool) != NGX_OK) {
>>>> -        return NGX_ERROR;
>>>> -    }
>>>> -
>>>> -    aio = file->aio;
>>>> -
>>>> -    aio->data = ctx->filter_ctx;
>>>> -    aio->preload_handler = ctx->aio_preload;
>>>> -
>>>> -    return NGX_OK;
>>>> -}
>>>> -
>>>> -#endif
>>>> -
>>>> -
>>>> static ngx_int_t
>>>> ngx_output_chain_add_copy(ngx_pool_t *pool, ngx_chain_t **chain,
>>>>     ngx_chain_t *in)
>>> 
>>> After this change, ngx_file_aio_init() doesn't need to be external.
>>> It is only used in ngx_file_aio_read() in corresponding ngx_*_aio_read.c.
>> 
>> In practice, yes.  In theory, it might be needed if we'll ever 
>> implement other AIO operations, such as aio_write(), and/or will 
>> reconsider thread-based interface to use the aio structure.  So I 
>> would rather  preserve it as is, at least for now.
>> 

Ok, I won't insist.

>> [...]
>> 
>>>> ngx_chain_t *
>>>> ngx_freebsd_sendfile_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
>>>> {
>>>> -    int               rc, flags;
>>>> -    off_t             send, prev_send, sent;
>>>> -    size_t            file_size;
>>>> -    ssize_t           n;
>>>> -    ngx_uint_t        eintr, eagain;
>>>> -    ngx_err_t         err;
>>>> -    ngx_buf_t        *file;
>>>> -    ngx_event_t      *wev;
>>>> -    ngx_chain_t      *cl;
>>>> -    ngx_iovec_t       header, trailer;
>>>> -    struct sf_hdtr    hdtr;
>>>> -    struct iovec      headers[NGX_IOVS_PREALLOCATE];
>>>> -    struct iovec      trailers[NGX_IOVS_PREALLOCATE];
>>>> -#if (NGX_HAVE_AIO_SENDFILE)
>>>> -    ngx_uint_t        ebusy;
>>>> -    ngx_event_aio_t  *aio;
>>>> +    int              rc, flags;
>>>> +    off_t            send, prev_send, sent;
>>>> +    size_t           file_size;
>>>> +    ssize_t          n;
>>>> +    ngx_uint_t       eintr, eagain;
>>>> +    ngx_err_t        err;
>>>> +    ngx_buf_t       *file;
>>>> +    ngx_event_t     *wev;
>>>> +    ngx_chain_t     *cl;
>>>> +    ngx_iovec_t      header, trailer;
>>>> +    struct sf_hdtr   hdtr;
>>>> +    struct iovec     headers[NGX_IOVS_PREALLOCATE];
>>>> +    struct iovec     trailers[NGX_IOVS_PREALLOCATE];
>>>> +#if (NGX_HAVE_SENDFILE_NODISKIO)
>>>> +    ngx_uint_t       ebusy;
>>>> #endif
>>> 
>>> After ngx_event_aio_t *aio variable removal,
>>> this block could be placed under "eintr, eagain" line
>>> (which by itself looks unsorted).
>>> 
>>> The remaining part looks good to me.
>> 
>> Yes, thanks.  Changed to:
>> 
>> diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
>> --- a/src/os/unix/ngx_freebsd_sendfile_chain.c
>> +++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
>> @@ -36,18 +36,18 @@ ngx_freebsd_sendfile_chain(ngx_connectio
>>     off_t            send, prev_send, sent;
>>     size_t           file_size;
>>     ssize_t          n;
>> -    ngx_uint_t       eintr, eagain;
>>     ngx_err_t        err;
>>     ngx_buf_t       *file;
>> +    ngx_uint_t       eintr, eagain;
>> +#if (NGX_HAVE_SENDFILE_NODISKIO)
>> +    ngx_uint_t       ebusy;
>> +#endif
>>     ngx_event_t     *wev;
>>     ngx_chain_t     *cl;
>>     ngx_iovec_t      header, trailer;
>>     struct sf_hdtr   hdtr;
>>     struct iovec     headers[NGX_IOVS_PREALLOCATE];
>>     struct iovec     trailers[NGX_IOVS_PREALLOCATE];
>> -#if (NGX_HAVE_SENDFILE_NODISKIO)
>> -    ngx_uint_t       ebusy;
>> -#endif
>> 
>>     wev = c->write;
>> 
>> 

Looks good to me.

>> Full patch, updated:
> 
> [...]
> 
>> diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
>> --- a/src/os/unix/ngx_freebsd_sendfile_chain.c
>> +++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
> 
> [...]
> 
>> @@ -199,7 +192,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
>>                     eintr = 1;
>>                     break;
>> 
>> -#if (NGX_HAVE_AIO_SENDFILE)
>> +#if (NGX_HAVE_SENDFILE_NODISKIO)
>>                 case NGX_EBUSY:
>>                     ebusy = 1;
>>                     break;
>> @@ -258,35 +251,24 @@ ngx_freebsd_sendfile_chain(ngx_connectio
>>             if (sent == 0) {
>>                 c->busy_count++;
>> 
> 
> Err, mismerge here.  Should be:
> 
> diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
> --- a/src/os/unix/ngx_freebsd_sendfile_chain.c
> +++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
> @@ -245,7 +245,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
> 
>         in = ngx_chain_update_sent(in, sent);
> 
> -#if (NGX_HAVE_AIO_SENDFILE)
> +#if (NGX_HAVE_SENDFILE_NODISKIO)
> 
>         if (ebusy) {
>             if (sent == 0) {
> 

Looks like this hunk appears in the updated patch 3,
should belong to patch 2 (SF_NODISKIO refactoring).

> 
> Full series updated:
> 

[..]

-- 
Sergey Kandaurov


From pluknet at nginx.com  Mon Dec 27 14:33:12 2021
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 27 Dec 2021 17:33:12 +0300
Subject: [PATCH 4 of 4] Support for sendfile(SF_NOCACHE)
In-Reply-To: 
References: 
 <10f96e74ae73e1c53a3f.1636604472@vm-bsd.mdounin.ru>
 <20211130121550.2eh5sm5aedkph2oz@MacBook-Air-Sergey.local>
 
Message-ID: <39DE8171-8661-425E-A270-CC7AF094AA66@nginx.com>


> On 14 Dec 2021, at 17:16, Maxim Dounin  wrote:
> 
> Hello!
> 
> On Tue, Nov 30, 2021 at 03:15:50PM +0300, Sergey Kandaurov wrote:
> 
>> On Thu, Nov 11, 2021 at 07:21:12AM +0300, Maxim Dounin wrote:
>>> # HG changeset patch
>>> # User Maxim Dounin 
>>> # Date 1636603897 -10800
>>> #      Thu Nov 11 07:11:37 2021 +0300
>>> # Node ID 10f96e74ae73e1c53a3fd08e7e1c26754c8969ed
>>> # Parent  98d3beb63f32cbb68d1cdcec385614d32129cad0
>>> Support for sendfile(SF_NOCACHE).
>>> 
>>> The SF_NOCACHE flag, introduced in FreeBSD 11 along with the new non-blocking
>>> sendfile() implementation by glebius@, makes it possible to use sendfile()
>>> along with the "directio" directive.
>>> 
>>> diff --git a/src/core/ngx_output_chain.c b/src/core/ngx_output_chain.c
>>> --- a/src/core/ngx_output_chain.c
>>> +++ b/src/core/ngx_output_chain.c
>>> @@ -256,9 +256,11 @@ ngx_output_chain_as_is(ngx_output_chain_
>>>     }
>>> #endif
>>> 
>>> +#if !(NGX_HAVE_SENDFILE_NODISKIO)
>>>     if (buf->in_file && buf->file->directio) {
>>>         return 0;
>>>     }
>>> +#endif
>> 
>> This probably deserves a comment, why it depends on such a macro test.
>> Though, it should be pretty clear from the commit log.
> 
> I also tend to think that the original code is wrong (or, rather, 
> too aggressive), and it should only disable sendfile(), much like 
> the ctx->sendfile and NGX_SENDFILE_LIMIT checks below.
> 
> Changed to the following:
> 
> diff --git a/src/core/ngx_output_chain.c b/src/core/ngx_output_chain.c
> --- a/src/core/ngx_output_chain.c
> +++ b/src/core/ngx_output_chain.c
> @@ -256,12 +256,6 @@ ngx_output_chain_as_is(ngx_output_chain_
>     }
> #endif
> 
> -#if !(NGX_HAVE_SENDFILE_NODISKIO)
> -    if (buf->in_file && buf->file->directio) {
> -        return 0;
> -    }
> -#endif
> -
>     sendfile = ctx->sendfile;
> 
> #if (NGX_SENDFILE_LIMIT)
> @@ -272,6 +266,19 @@ ngx_output_chain_as_is(ngx_output_chain_
> 
> #endif
> 
> +#if !(NGX_HAVE_SENDFILE_NODISKIO)
> +
> +    /*
> +     * With DIRECTIO, disable sendfile() unless sendfile(SF_NOCACHE)
> +     * is available.
> +     */
> +
> +    if (buf->in_file && buf->file->directio) {
> +        sendfile = 0;
> +    }
> +
> +#endif
> +
>     if (!sendfile) {
> 
>         if (!ngx_buf_in_memory(buf)) {
> 
> 
> [...]
> 

Given the sendfile condition below, I couldn't find such cases with
buffers that have both in_file and ngx_buf_in_memory() tests passed
(at least), such that the patch would have a real behaviour change
resulting in just the in_file bit cleared.
Theoretically, it looks good and more aligned with documentation.

-- 
Sergey Kandaurov


From xeioex at nginx.com  Mon Dec 27 16:06:58 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 27 Dec 2021 16:06:58 +0000
Subject: [njs] Improved Buffer.from() with Buffer-like objects.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/2e544ef59092
branches:  
changeset: 1788:2e544ef59092
user:      Artem S. Povalyukhin 
date:      Sat Dec 25 22:45:30 2021 +0300
description:
Improved Buffer.from() with Buffer-like objects.

diffstat:

 src/njs_buffer.c         |  6 +++---
 src/test/njs_unit_test.c |  6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

diffs (43 lines):

diff -r 256421f5cca2 -r 2e544ef59092 src/njs_buffer.c
--- a/src/njs_buffer.c	Fri Dec 24 10:00:15 2021 +0300
+++ b/src/njs_buffer.c	Sat Dec 25 22:45:30 2021 +0300
@@ -358,12 +358,12 @@ next:
         ret = njs_value_property(vm, value, njs_value_arg(&njs_string_type),
                                  &retval);
         if (njs_slow_path(ret != NJS_OK)) {
-            return NJS_DECLINED;
+            return ret;
         }
 
         ret = njs_value_to_string(vm, &retval, &retval);
         if (njs_slow_path(ret != NJS_OK)) {
-            return NJS_DECLINED;
+            return ret;
         }
 
         njs_string_get(&retval, &str);
@@ -375,7 +375,7 @@ next:
         ret = njs_value_property(vm, value, njs_value_arg(&njs_string_data),
                                  &retval);
         if (njs_slow_path(ret != NJS_OK)) {
-            return NJS_DECLINED;
+            return ret;
         }
 
         if (njs_is_object(&retval)) {
diff -r 256421f5cca2 -r 2e544ef59092 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Fri Dec 24 10:00:15 2021 +0300
+++ b/src/test/njs_unit_test.c	Sat Dec 25 22:45:30 2021 +0300
@@ -19920,6 +19920,12 @@ static njs_unit_test_t  njs_buffer_modul
     { njs_str("njs.dump(Buffer.from(new String('test')))"),
       njs_str("Buffer [116,101,115,116]") },
 
+    { njs_str("Buffer.from({ get type() { throw new Error('test'); } })"),
+      njs_str("Error: test") },
+
+    { njs_str("Buffer.from({ type: 'Buffer', get data() { throw new Error('test'); } })"),
+      njs_str("Error: test") },
+
     { njs_str("["
              " ['6576696c', 'hex'],"
              " ['ZXZpbA==', 'base64'],"

From xeioex at nginx.com  Mon Dec 27 16:07:00 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 27 Dec 2021 16:07:00 +0000
Subject: [njs] Fixed information leak in Buffer.from().
Message-ID: 

details:   https://hg.nginx.org/njs/rev/752d3d8ab217
branches:  
changeset: 1789:752d3d8ab217
user:      Artem S. Povalyukhin 
date:      Sat Dec 25 22:45:30 2021 +0300
description:
Fixed information leak in Buffer.from().

This closes #446 on Github.

diffstat:

 src/njs_buffer.c         |  23 +++--------------------
 src/test/njs_unit_test.c |   8 ++++++++
 2 files changed, 11 insertions(+), 20 deletions(-)

diffs (65 lines):

diff -r 2e544ef59092 -r 752d3d8ab217 src/njs_buffer.c
--- a/src/njs_buffer.c	Sat Dec 25 22:45:30 2021 +0300
+++ b/src/njs_buffer.c	Sat Dec 25 22:45:30 2021 +0300
@@ -339,8 +339,7 @@ njs_buffer_from_object(njs_vm_t *vm, njs
     uint32_t           i;
     njs_str_t          str;
     njs_int_t          ret;
-    njs_array_t        *array;
-    njs_value_t        retval, length;
+    njs_value_t        data, retval, length;
     njs_typed_array_t  *buffer;
 
     static const njs_value_t  string_length = njs_string("length");
@@ -379,7 +378,8 @@ next:
         }
 
         if (njs_is_object(&retval)) {
-            value = &retval;
+            njs_value_assign(&data, &retval);
+            value = &data;
             goto next;
         }
 
@@ -398,23 +398,6 @@ next:
 
     p = njs_typed_array_buffer(buffer)->u.u8;
 
-    if (njs_is_fast_array(value)) {
-        array = njs_array(value);
-
-        for (i = 0; i < array->length; i++) {
-            ret = njs_value_to_number(vm, &array->start[i], &num);
-            if (njs_slow_path(ret != NJS_OK)) {
-                return ret;
-            }
-
-            *p++ = njs_number_to_int32(num);
-        }
-
-        njs_set_typed_array(&vm->retval, buffer);
-
-        return NJS_OK;
-    }
-
     for (i = 0; i < len; i++) {
         ret = njs_value_property_i64(vm, value, i, &retval);
         if (njs_slow_path(ret == NJS_ERROR)) {
diff -r 2e544ef59092 -r 752d3d8ab217 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Sat Dec 25 22:45:30 2021 +0300
+++ b/src/test/njs_unit_test.c	Sat Dec 25 22:45:30 2021 +0300
@@ -19926,6 +19926,14 @@ static njs_unit_test_t  njs_buffer_modul
     { njs_str("Buffer.from({ type: 'Buffer', get data() { throw new Error('test'); } })"),
       njs_str("Error: test") },
 
+    { njs_str("var a = [1,2,3,4]; a[1] = { valueOf() { a.length = 3; return 1; } };"
+              "njs.dump(Buffer.from(a))"),
+      njs_str("Buffer [1,1,3,0]") },
+
+    { njs_str("var a = [1,2,3,4]; a[1] = { valueOf() { a.length = 4096; a.fill(13); return 1; } };"
+              "njs.dump(Buffer.from(a))"),
+      njs_str("Buffer [1,1,13,13]") },
+
     { njs_str("["
              " ['6576696c', 'hex'],"
              " ['ZXZpbA==', 'base64'],"

From mdounin at mdounin.ru  Mon Dec 27 18:48:30 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 27 Dec 2021 18:48:30 +0000
Subject: [nginx] Removed "aio sendfile", deprecated since 1.7.11.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/ae992b5a27b2
branches:  
changeset: 7984:ae992b5a27b2
user:      Maxim Dounin 
date:      Mon Dec 27 19:47:05 2021 +0300
description:
Removed "aio sendfile", deprecated since 1.7.11.

diffstat:

 src/http/ngx_http_core_module.c |  13 -------------
 1 files changed, 0 insertions(+), 13 deletions(-)

diffs (23 lines):

diff -r d07456044b61 -r ae992b5a27b2 src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c	Sat Dec 25 01:07:18 2021 +0300
+++ b/src/http/ngx_http_core_module.c	Mon Dec 27 19:47:05 2021 +0300
@@ -4568,19 +4568,6 @@ ngx_http_core_set_aio(ngx_conf_t *cf, ng
 #endif
     }
 
-#if (NGX_HAVE_AIO_SENDFILE)
-
-    if (ngx_strcmp(value[1].data, "sendfile") == 0) {
-        clcf->aio = NGX_HTTP_AIO_ON;
-
-        ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
-                           "the \"sendfile\" parameter of "
-                           "the \"aio\" directive is deprecated");
-        return NGX_CONF_OK;
-    }
-
-#endif
-
     if (ngx_strncmp(value[1].data, "threads", 7) == 0
         && (value[1].len == 7 || value[1].data[7] == '='))
     {

From mdounin at mdounin.ru  Mon Dec 27 18:48:33 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 27 Dec 2021 18:48:33 +0000
Subject: [nginx] Simplified sendfile(SF_NODISKIO) usage.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/ec2e6893caaa
branches:  
changeset: 7985:ec2e6893caaa
user:      Maxim Dounin 
date:      Mon Dec 27 19:48:33 2021 +0300
description:
Simplified sendfile(SF_NODISKIO) usage.

Starting with FreeBSD 11, there is no need to use AIO operations to preload
data into cache for sendfile(SF_NODISKIO) to work.  Instead, sendfile()
handles non-blocking loading data from disk by itself.  It still can, however,
return EBUSY if a page is already being loaded (for example, by a different
process).  If this happens, we now post an event for the next event loop
iteration, so sendfile() is retried "after a short period", as manpage
recommends.

The limit of the number of EBUSY tolerated without any progress is preserved,
but now it does not result in an alert, since on an idle system event loop
iteration might be very short and EBUSY can happen many times in a row.
Instead, SF_NODISKIO is simply disabled for one call once the limit is
reached.

With this change, sendfile(SF_NODISKIO) is now used automatically as long as
sendfile() is enabled, and no longer requires "aio on;".

diffstat:

 auto/os/freebsd                          |   8 +-
 src/core/ngx_buf.h                       |   3 -
 src/core/ngx_connection.h                |   2 +-
 src/core/ngx_module.h                    |   2 +-
 src/core/ngx_output_chain.c              |  32 ------------
 src/event/ngx_event.h                    |   4 -
 src/http/ngx_http_copy_filter_module.c   |  82 --------------------------------
 src/os/unix/ngx_freebsd_sendfile_chain.c |  74 ++++++++++-----------------
 8 files changed, 33 insertions(+), 174 deletions(-)

diffs (367 lines):

diff -r ae992b5a27b2 -r ec2e6893caaa auto/os/freebsd
--- a/auto/os/freebsd	Mon Dec 27 19:47:05 2021 +0300
+++ b/auto/os/freebsd	Mon Dec 27 19:48:33 2021 +0300
@@ -44,12 +44,10 @@ if [ $osreldate -gt 300007 ]; then
     CORE_SRCS="$CORE_SRCS $FREEBSD_SENDFILE_SRCS"
 fi
 
-if [ $NGX_FILE_AIO = YES ]; then
-    if [ $osreldate -gt 502103 ]; then
-        echo " + sendfile()'s SF_NODISKIO found"
+if [ $osreldate -gt 1100093 ]; then
+    echo " + sendfile()'s SF_NODISKIO found"
 
-        have=NGX_HAVE_AIO_SENDFILE . auto/have
-    fi
+    have=NGX_HAVE_SENDFILE_NODISKIO . auto/have
 fi
 
 # POSIX semaphores
diff -r ae992b5a27b2 -r ec2e6893caaa src/core/ngx_buf.h
--- a/src/core/ngx_buf.h	Mon Dec 27 19:47:05 2021 +0300
+++ b/src/core/ngx_buf.h	Mon Dec 27 19:48:33 2021 +0300
@@ -90,9 +90,6 @@ struct ngx_output_chain_ctx_s {
 
 #if (NGX_HAVE_FILE_AIO || NGX_COMPAT)
     ngx_output_chain_aio_pt      aio_handler;
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
-    ssize_t                    (*aio_preload)(ngx_buf_t *file);
-#endif
 #endif
 
 #if (NGX_THREADS || NGX_COMPAT)
diff -r ae992b5a27b2 -r ec2e6893caaa src/core/ngx_connection.h
--- a/src/core/ngx_connection.h	Mon Dec 27 19:47:05 2021 +0300
+++ b/src/core/ngx_connection.h	Mon Dec 27 19:48:33 2021 +0300
@@ -185,7 +185,7 @@ struct ngx_connection_s {
 
     unsigned            need_last_buf:1;
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
+#if (NGX_HAVE_SENDFILE_NODISKIO || NGX_COMPAT)
     unsigned            busy_count:2;
 #endif
 
diff -r ae992b5a27b2 -r ec2e6893caaa src/core/ngx_module.h
--- a/src/core/ngx_module.h	Mon Dec 27 19:47:05 2021 +0300
+++ b/src/core/ngx_module.h	Mon Dec 27 19:48:33 2021 +0300
@@ -41,7 +41,7 @@
 #define NGX_MODULE_SIGNATURE_3   "0"
 #endif
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
+#if (NGX_HAVE_SENDFILE_NODISKIO || NGX_COMPAT)
 #define NGX_MODULE_SIGNATURE_4   "1"
 #else
 #define NGX_MODULE_SIGNATURE_4   "0"
diff -r ae992b5a27b2 -r ec2e6893caaa src/core/ngx_output_chain.c
--- a/src/core/ngx_output_chain.c	Mon Dec 27 19:47:05 2021 +0300
+++ b/src/core/ngx_output_chain.c	Mon Dec 27 19:48:33 2021 +0300
@@ -29,10 +29,6 @@
 
 static ngx_inline ngx_int_t
     ngx_output_chain_as_is(ngx_output_chain_ctx_t *ctx, ngx_buf_t *buf);
-#if (NGX_HAVE_AIO_SENDFILE)
-static ngx_int_t ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx,
-    ngx_file_t *file);
-#endif
 static ngx_int_t ngx_output_chain_add_copy(ngx_pool_t *pool,
     ngx_chain_t **chain, ngx_chain_t *in);
 static ngx_int_t ngx_output_chain_align_file_buf(ngx_output_chain_ctx_t *ctx,
@@ -283,12 +279,6 @@ ngx_output_chain_as_is(ngx_output_chain_
         buf->in_file = 0;
     }
 
-#if (NGX_HAVE_AIO_SENDFILE)
-    if (ctx->aio_preload && buf->in_file) {
-        (void) ngx_output_chain_aio_setup(ctx, buf->file);
-    }
-#endif
-
     if (ctx->need_in_memory && !ngx_buf_in_memory(buf)) {
         return 0;
     }
@@ -301,28 +291,6 @@ ngx_output_chain_as_is(ngx_output_chain_
 }
 
 
-#if (NGX_HAVE_AIO_SENDFILE)
-
-static ngx_int_t
-ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx, ngx_file_t *file)
-{
-    ngx_event_aio_t  *aio;
-
-    if (file->aio == NULL && ngx_file_aio_init(file, ctx->pool) != NGX_OK) {
-        return NGX_ERROR;
-    }
-
-    aio = file->aio;
-
-    aio->data = ctx->filter_ctx;
-    aio->preload_handler = ctx->aio_preload;
-
-    return NGX_OK;
-}
-
-#endif
-
-
 static ngx_int_t
 ngx_output_chain_add_copy(ngx_pool_t *pool, ngx_chain_t **chain,
     ngx_chain_t *in)
diff -r ae992b5a27b2 -r ec2e6893caaa src/event/ngx_event.h
--- a/src/event/ngx_event.h	Mon Dec 27 19:47:05 2021 +0300
+++ b/src/event/ngx_event.h	Mon Dec 27 19:48:33 2021 +0300
@@ -147,10 +147,6 @@ struct ngx_event_aio_s {
 
     ngx_fd_t                   fd;
 
-#if (NGX_HAVE_AIO_SENDFILE || NGX_COMPAT)
-    ssize_t                  (*preload_handler)(ngx_buf_t *file);
-#endif
-
 #if (NGX_HAVE_EVENTFD)
     int64_t                    res;
 #endif
diff -r ae992b5a27b2 -r ec2e6893caaa src/http/ngx_http_copy_filter_module.c
--- a/src/http/ngx_http_copy_filter_module.c	Mon Dec 27 19:47:05 2021 +0300
+++ b/src/http/ngx_http_copy_filter_module.c	Mon Dec 27 19:48:33 2021 +0300
@@ -19,10 +19,6 @@ typedef struct {
 static void ngx_http_copy_aio_handler(ngx_output_chain_ctx_t *ctx,
     ngx_file_t *file);
 static void ngx_http_copy_aio_event_handler(ngx_event_t *ev);
-#if (NGX_HAVE_AIO_SENDFILE)
-static ssize_t ngx_http_copy_aio_sendfile_preload(ngx_buf_t *file);
-static void ngx_http_copy_aio_sendfile_event_handler(ngx_event_t *ev);
-#endif
 #endif
 #if (NGX_THREADS)
 static ngx_int_t ngx_http_copy_thread_handler(ngx_thread_task_t *task,
@@ -128,9 +124,6 @@ ngx_http_copy_filter(ngx_http_request_t 
 #if (NGX_HAVE_FILE_AIO)
         if (ngx_file_aio && clcf->aio == NGX_HTTP_AIO_ON) {
             ctx->aio_handler = ngx_http_copy_aio_handler;
-#if (NGX_HAVE_AIO_SENDFILE)
-            ctx->aio_preload = ngx_http_copy_aio_sendfile_preload;
-#endif
         }
 #endif
 
@@ -207,81 +200,6 @@ ngx_http_copy_aio_event_handler(ngx_even
     ngx_http_run_posted_requests(c);
 }
 
-
-#if (NGX_HAVE_AIO_SENDFILE)
-
-static ssize_t
-ngx_http_copy_aio_sendfile_preload(ngx_buf_t *file)
-{
-    ssize_t                  n;
-    static u_char            buf[1];
-    ngx_event_aio_t         *aio;
-    ngx_http_request_t      *r;
-    ngx_output_chain_ctx_t  *ctx;
-
-    aio = file->file->aio;
-    r = aio->data;
-
-    if (r->aio) {
-        /*
-         * tolerate sendfile() calls if another operation is already
-         * running; this can happen due to subrequests, multiple calls
-         * of the next body filter from a filter, or in HTTP/2 due to
-         * a write event on the main connection
-         */
-
-        return NGX_AGAIN;
-    }
-
-    n = ngx_file_aio_read(file->file, buf, 1, file->file_pos, NULL);
-
-    if (n == NGX_AGAIN) {
-        aio->handler = ngx_http_copy_aio_sendfile_event_handler;
-
-        r->main->blocked++;
-        r->aio = 1;
-
-        ctx = ngx_http_get_module_ctx(r, ngx_http_copy_filter_module);
-        ctx->aio = 1;
-    }
-
-    return n;
-}
-
-
-static void
-ngx_http_copy_aio_sendfile_event_handler(ngx_event_t *ev)
-{
-    ngx_event_aio_t     *aio;
-    ngx_connection_t    *c;
-    ngx_http_request_t  *r;
-
-    aio = ev->data;
-    r = aio->data;
-    c = r->connection;
-
-    r->main->blocked--;
-    r->aio = 0;
-    ev->complete = 0;
-
-#if (NGX_HTTP_V2)
-
-    if (r->stream) {
-        /*
-         * for HTTP/2, update write event to make sure processing will
-         * reach the main connection to handle sendfile() preload
-         */
-
-        c->write->ready = 1;
-        c->write->active = 0;
-    }
-
-#endif
-
-    c->write->handler(c->write);
-}
-
-#endif
 #endif
 
 
diff -r ae992b5a27b2 -r ec2e6893caaa src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c	Mon Dec 27 19:47:05 2021 +0300
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c	Mon Dec 27 19:48:33 2021 +0300
@@ -32,23 +32,22 @@
 ngx_chain_t *
 ngx_freebsd_sendfile_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
 {
-    int               rc, flags;
-    off_t             send, prev_send, sent;
-    size_t            file_size;
-    ssize_t           n;
-    ngx_uint_t        eintr, eagain;
-    ngx_err_t         err;
-    ngx_buf_t        *file;
-    ngx_event_t      *wev;
-    ngx_chain_t      *cl;
-    ngx_iovec_t       header, trailer;
-    struct sf_hdtr    hdtr;
-    struct iovec      headers[NGX_IOVS_PREALLOCATE];
-    struct iovec      trailers[NGX_IOVS_PREALLOCATE];
-#if (NGX_HAVE_AIO_SENDFILE)
-    ngx_uint_t        ebusy;
-    ngx_event_aio_t  *aio;
+    int              rc, flags;
+    off_t            send, prev_send, sent;
+    size_t           file_size;
+    ssize_t          n;
+    ngx_err_t        err;
+    ngx_buf_t       *file;
+    ngx_uint_t       eintr, eagain;
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+    ngx_uint_t       ebusy;
 #endif
+    ngx_event_t     *wev;
+    ngx_chain_t     *cl;
+    ngx_iovec_t      header, trailer;
+    struct sf_hdtr   hdtr;
+    struct iovec     headers[NGX_IOVS_PREALLOCATE];
+    struct iovec     trailers[NGX_IOVS_PREALLOCATE];
 
     wev = c->write;
 
@@ -77,11 +76,6 @@ ngx_freebsd_sendfile_chain(ngx_connectio
     eagain = 0;
     flags = 0;
 
-#if (NGX_HAVE_AIO_SENDFILE && NGX_SUPPRESS_WARN)
-    aio = NULL;
-    file = NULL;
-#endif
-
     header.iovs = headers;
     header.nalloc = NGX_IOVS_PREALLOCATE;
 
@@ -90,7 +84,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
     for ( ;; ) {
         eintr = 0;
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
         ebusy = 0;
 #endif
         prev_send = send;
@@ -179,9 +173,8 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
             sent = 0;
 
-#if (NGX_HAVE_AIO_SENDFILE)
-            aio = file->file->aio;
-            flags = (aio && aio->preload_handler) ? SF_NODISKIO : 0;
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+            flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
 #endif
 
             rc = sendfile(file->file->fd, c->fd, file->file_pos,
@@ -199,7 +192,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
                     eintr = 1;
                     break;
 
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
                 case NGX_EBUSY:
                     ebusy = 1;
                     break;
@@ -252,41 +245,30 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
         in = ngx_chain_update_sent(in, sent);
 
-#if (NGX_HAVE_AIO_SENDFILE)
+#if (NGX_HAVE_SENDFILE_NODISKIO)
 
         if (ebusy) {
             if (sent == 0) {
                 c->busy_count++;
 
-                if (c->busy_count > 2) {
-                    ngx_log_error(NGX_LOG_ALERT, c->log, 0,
-                                  "sendfile(%V) returned busy again",
-                                  &file->file->name);
-
-                    c->busy_count = 0;
-                    aio->preload_handler = NULL;
-
-                    send = prev_send;
-                    continue;
-                }
+                ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
+                               "sendfile() busy, count:%d", c->busy_count);
 
             } else {
                 c->busy_count = 0;
             }
 
-            n = aio->preload_handler(file);
-
-            if (n > 0) {
-                send = prev_send + sent;
-                continue;
+            if (wev->posted) {
+                ngx_delete_posted_event(wev);
             }
 
+            ngx_post_event(wev, &ngx_posted_next_events);
+
+            wev->ready = 0;
             return in;
         }
 
-        if (flags == SF_NODISKIO) {
-            c->busy_count = 0;
-        }
+        c->busy_count = 0;
 
 #endif
 

From mdounin at mdounin.ru  Mon Dec 27 18:48:35 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 27 Dec 2021 18:48:35 +0000
Subject: [nginx] SSL: SSL_sendfile(SF_NODISKIO) support.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/f1fcb0fe6975
branches:  
changeset: 7986:f1fcb0fe6975
user:      Maxim Dounin 
date:      Mon Dec 27 19:48:42 2021 +0300
description:
SSL: SSL_sendfile(SF_NODISKIO) support.

diffstat:

 src/event/ngx_event_openssl.c |  31 +++++++++++++++++++++++++++++--
 1 files changed, 29 insertions(+), 2 deletions(-)

diffs (63 lines):

diff -r ec2e6893caaa -r f1fcb0fe6975 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Mon Dec 27 19:48:33 2021 +0300
+++ b/src/event/ngx_event_openssl.c	Mon Dec 27 19:48:42 2021 +0300
@@ -2942,7 +2942,7 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
 {
 #ifdef BIO_get_ktls_send
 
-    int        sslerr;
+    int        sslerr, flags;
     ssize_t    n;
     ngx_err_t  err;
 
@@ -2954,8 +2954,14 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
 
     ngx_set_errno(0);
 
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+    flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
+#else
+    flags = 0;
+#endif
+
     n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,
-                     size, 0);
+                     size, flags);
 
     ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_sendfile: %d", n);
 
@@ -2974,6 +2980,10 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
             ngx_post_event(c->read, &ngx_posted_events);
         }
 
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+        c->busy_count = 0;
+#endif
+
         c->sent += n;
 
         return n;
@@ -3038,6 +3048,23 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
             ngx_post_event(c->read, &ngx_posted_events);
         }
 
+#if (NGX_HAVE_SENDFILE_NODISKIO)
+
+        if (ngx_errno == EBUSY) {
+            c->busy_count++;
+
+            ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0,
+                           "SSL_sendfile() busy, count:%d", c->busy_count);
+
+            if (c->write->posted) {
+                ngx_delete_posted_event(c->write);
+            }
+
+            ngx_post_event(c->write, &ngx_posted_next_events);
+        }
+
+#endif
+
         c->write->ready = 0;
         return NGX_AGAIN;
     }

From mdounin at mdounin.ru  Mon Dec 27 18:48:38 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 27 Dec 2021 18:48:38 +0000
Subject: [nginx] Support for sendfile(SF_NOCACHE).
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/b002ad258f1d
branches:  
changeset: 7987:b002ad258f1d
user:      Maxim Dounin 
date:      Mon Dec 27 19:49:26 2021 +0300
description:
Support for sendfile(SF_NOCACHE).

The SF_NOCACHE flag, introduced in FreeBSD 11 along with the new non-blocking
sendfile() implementation by glebius@, makes it possible to use sendfile()
along with the "directio" directive.

diffstat:

 src/core/ngx_output_chain.c              |  17 +++++++++++++----
 src/event/ngx_event_openssl.c            |   6 ++++++
 src/os/unix/ngx_freebsd_sendfile_chain.c |   6 ++++++
 3 files changed, 25 insertions(+), 4 deletions(-)

diffs (68 lines):

diff -r f1fcb0fe6975 -r b002ad258f1d src/core/ngx_output_chain.c
--- a/src/core/ngx_output_chain.c	Mon Dec 27 19:48:42 2021 +0300
+++ b/src/core/ngx_output_chain.c	Mon Dec 27 19:49:26 2021 +0300
@@ -256,10 +256,6 @@ ngx_output_chain_as_is(ngx_output_chain_
     }
 #endif
 
-    if (buf->in_file && buf->file->directio) {
-        return 0;
-    }
-
     sendfile = ctx->sendfile;
 
 #if (NGX_SENDFILE_LIMIT)
@@ -270,6 +266,19 @@ ngx_output_chain_as_is(ngx_output_chain_
 
 #endif
 
+#if !(NGX_HAVE_SENDFILE_NODISKIO)
+
+    /*
+     * With DIRECTIO, disable sendfile() unless sendfile(SF_NOCACHE)
+     * is available.
+     */
+
+    if (buf->in_file && buf->file->directio) {
+        sendfile = 0;
+    }
+
+#endif
+
     if (!sendfile) {
 
         if (!ngx_buf_in_memory(buf)) {
diff -r f1fcb0fe6975 -r b002ad258f1d src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Mon Dec 27 19:48:42 2021 +0300
+++ b/src/event/ngx_event_openssl.c	Mon Dec 27 19:49:26 2021 +0300
@@ -2955,7 +2955,13 @@ ngx_ssl_sendfile(ngx_connection_t *c, ng
     ngx_set_errno(0);
 
 #if (NGX_HAVE_SENDFILE_NODISKIO)
+
     flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
+
+    if (file->file->directio) {
+        flags |= SF_NOCACHE;
+    }
+
 #else
     flags = 0;
 #endif
diff -r f1fcb0fe6975 -r b002ad258f1d src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c	Mon Dec 27 19:48:42 2021 +0300
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c	Mon Dec 27 19:49:26 2021 +0300
@@ -174,7 +174,13 @@ ngx_freebsd_sendfile_chain(ngx_connectio
             sent = 0;
 
 #if (NGX_HAVE_SENDFILE_NODISKIO)
+
             flags = (c->busy_count <= 2) ? SF_NODISKIO : 0;
+
+            if (file->file->directio) {
+                flags |= SF_NOCACHE;
+            }
+
 #endif
 
             rc = sendfile(file->file->fd, c->fd, file->file_pos,

From mdounin at mdounin.ru  Mon Dec 27 18:48:55 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 27 Dec 2021 21:48:55 +0300
Subject: [PATCH 2 of 4] Simplified sendfile(SF_NODISKIO) usage
In-Reply-To: <1FA7B6FA-A7A3-49F9-89B8-6CF9C2610508@nginx.com>
References: 
 <4a954e89b1ae8539bbe0.1636604470@vm-bsd.mdounin.ru>
 <20211130120503.552wawqq52udi3ag@MacBook-Air-Sergey.local>
  
 <1FA7B6FA-A7A3-49F9-89B8-6CF9C2610508@nginx.com>
Message-ID: 

Hello!

On Mon, Dec 27, 2021 at 05:12:39PM +0300, Sergey Kandaurov wrote:

> > On 14 Dec 2021, at 18:28, Maxim Dounin  wrote:
> > 
> > Hello!
> > 
> > On Tue, Dec 14, 2021 at 05:15:47PM +0300, Maxim Dounin wrote:
> > 
> >> On Tue, Nov 30, 2021 at 03:05:03PM +0300, Sergey Kandaurov wrote:
> >> 
> >>> On Thu, Nov 11, 2021 at 07:21:10AM +0300, Maxim Dounin wrote:
> >>>> # HG changeset patch
> >>>> # User Maxim Dounin 
> >>>> # Date 1636603886 -10800
> >>>> #      Thu Nov 11 07:11:26 2021 +0300
> >>>> # Node ID 4a954e89b1ae8539bbe08c5afc1d5c9828d82d6f
> >>>> # Parent  0fb75ef9dbca698e5e855145cf6a12180a36d400
> >>>> Simplified sendfile(SF_NODISKIO) usage.
> >>>> 
> >>>> Starting with FreeBSD 11, there is no need to use AIO operations to preload
> >>>> data into cache for sendfile(SF_NODISKIO) to work.  Instead, sendfile()
> >>>> handles non-blocking loading data from disk by itself.  It still can, however,
> >>>> return EBUSY if a page is already being loaded (for example, by a different
> >>>> process).  If this happens, we now post an event for the next event loop
> >>>> iteration, so sendfile() is retried "after a short period", as manpage
> >>>> recommends.
> >>>> 
> >>>> The limit of the number of EBUSY tolerated without any progress is preserved,
> >>>> but now it does not result in an alert, since on an idle system event loop
> >>>> iteration might be very short and EBUSY can happen many times in a row.
> >>>> Instead, SF_NODISKIO is simply disabled for one call once the limit is
> >>>> reached.
> >>>> 
> >>>> With this change, sendfile(SF_NODISKIO) is now used automatically as long as
> >>>> sendfile() is enabled, and no longer requires "aio on;".
> >>>> 
> >>>> diff --git a/auto/os/freebsd b/auto/os/freebsd
> >>>> --- a/auto/os/freebsd
> >>>> +++ b/auto/os/freebsd
> >>>> @@ -44,12 +44,10 @@ if [ $osreldate -gt 300007 ]; then
> >>>>     CORE_SRCS="$CORE_SRCS $FREEBSD_SENDFILE_SRCS"
> >>>> fi
> >>>> 
> >>>> -if [ $NGX_FILE_AIO = YES ]; then
> >>>> -    if [ $osreldate -gt 502103 ]; then
> >>>> -        echo " + sendfile()'s SF_NODISKIO found"
> >>>> +if [ $osreldate -gt 1100000 ]; then
> >>>> +    echo " + sendfile()'s SF_NODISKIO found"
> >>>> 
> >>>> -        have=NGX_HAVE_AIO_SENDFILE . auto/have
> >>>> -    fi
> >>>> +    have=NGX_HAVE_SENDFILE_NODISKIO . auto/have
> >>>> fi
> >>>> 
> >>>> # POSIX semaphores
> >>> 
> >>> We could check the exact __FreeBSD_version number 1100093 that was
> >>> at the time the new sendfile() appeared, which is more accurate.
> >>> 
> >>> https://cgit.freebsd.org/src/commit/?id=2bab0c553588
> >>> https://cgit.freebsd.org/src/tree/sys/sys/param.h?id=2bab0c553588#n48
> >>> 
> >>> Unfortunately, it was not bumped (same as with SF_NODISKIO in 5.2.1).
> >> 
> >> Yes, probably.  But the next version bump would be more 
> >> appropriate, changed to 1100093.  Similarly to 502103, which is 
> >> the next version after SF_NODISKIO introduction:
> >> 
> >> https://cgit.freebsd.org/src/commit/?id=b49d824e8bc1
> >> https://cgit.freebsd.org/src/tree/sys/sys/param.h?id=b49d824e8bc1
> >> 
> 
> I'm fine with it.
> 
> >> [...]
> >> 
> >>>> -static ngx_int_t
> >>>> -ngx_output_chain_aio_setup(ngx_output_chain_ctx_t *ctx, ngx_file_t *file)
> >>>> -{
> >>>> -    ngx_event_aio_t  *aio;
> >>>> -
> >>>> -    if (file->aio == NULL && ngx_file_aio_init(file, ctx->pool) != NGX_OK) {
> >>>> -        return NGX_ERROR;
> >>>> -    }
> >>>> -
> >>>> -    aio = file->aio;
> >>>> -
> >>>> -    aio->data = ctx->filter_ctx;
> >>>> -    aio->preload_handler = ctx->aio_preload;
> >>>> -
> >>>> -    return NGX_OK;
> >>>> -}
> >>>> -
> >>>> -#endif
> >>>> -
> >>>> -
> >>>> static ngx_int_t
> >>>> ngx_output_chain_add_copy(ngx_pool_t *pool, ngx_chain_t **chain,
> >>>>     ngx_chain_t *in)
> >>> 
> >>> After this change, ngx_file_aio_init() doesn't need to be external.
> >>> It is only used in ngx_file_aio_read() in corresponding ngx_*_aio_read.c.
> >> 
> >> In practice, yes.  In theory, it might be needed if we'll ever 
> >> implement other AIO operations, such as aio_write(), and/or will 
> >> reconsider thread-based interface to use the aio structure.  So I 
> >> would rather  preserve it as is, at least for now.
> >> 
> 
> Ok, I won't insist.
> 
> >> [...]
> >> 
> >>>> ngx_chain_t *
> >>>> ngx_freebsd_sendfile_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
> >>>> {
> >>>> -    int               rc, flags;
> >>>> -    off_t             send, prev_send, sent;
> >>>> -    size_t            file_size;
> >>>> -    ssize_t           n;
> >>>> -    ngx_uint_t        eintr, eagain;
> >>>> -    ngx_err_t         err;
> >>>> -    ngx_buf_t        *file;
> >>>> -    ngx_event_t      *wev;
> >>>> -    ngx_chain_t      *cl;
> >>>> -    ngx_iovec_t       header, trailer;
> >>>> -    struct sf_hdtr    hdtr;
> >>>> -    struct iovec      headers[NGX_IOVS_PREALLOCATE];
> >>>> -    struct iovec      trailers[NGX_IOVS_PREALLOCATE];
> >>>> -#if (NGX_HAVE_AIO_SENDFILE)
> >>>> -    ngx_uint_t        ebusy;
> >>>> -    ngx_event_aio_t  *aio;
> >>>> +    int              rc, flags;
> >>>> +    off_t            send, prev_send, sent;
> >>>> +    size_t           file_size;
> >>>> +    ssize_t          n;
> >>>> +    ngx_uint_t       eintr, eagain;
> >>>> +    ngx_err_t        err;
> >>>> +    ngx_buf_t       *file;
> >>>> +    ngx_event_t     *wev;
> >>>> +    ngx_chain_t     *cl;
> >>>> +    ngx_iovec_t      header, trailer;
> >>>> +    struct sf_hdtr   hdtr;
> >>>> +    struct iovec     headers[NGX_IOVS_PREALLOCATE];
> >>>> +    struct iovec     trailers[NGX_IOVS_PREALLOCATE];
> >>>> +#if (NGX_HAVE_SENDFILE_NODISKIO)
> >>>> +    ngx_uint_t       ebusy;
> >>>> #endif
> >>> 
> >>> After ngx_event_aio_t *aio variable removal,
> >>> this block could be placed under "eintr, eagain" line
> >>> (which by itself looks unsorted).
> >>> 
> >>> The remaining part looks good to me.
> >> 
> >> Yes, thanks.  Changed to:
> >> 
> >> diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
> >> --- a/src/os/unix/ngx_freebsd_sendfile_chain.c
> >> +++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
> >> @@ -36,18 +36,18 @@ ngx_freebsd_sendfile_chain(ngx_connectio
> >>     off_t            send, prev_send, sent;
> >>     size_t           file_size;
> >>     ssize_t          n;
> >> -    ngx_uint_t       eintr, eagain;
> >>     ngx_err_t        err;
> >>     ngx_buf_t       *file;
> >> +    ngx_uint_t       eintr, eagain;
> >> +#if (NGX_HAVE_SENDFILE_NODISKIO)
> >> +    ngx_uint_t       ebusy;
> >> +#endif
> >>     ngx_event_t     *wev;
> >>     ngx_chain_t     *cl;
> >>     ngx_iovec_t      header, trailer;
> >>     struct sf_hdtr   hdtr;
> >>     struct iovec     headers[NGX_IOVS_PREALLOCATE];
> >>     struct iovec     trailers[NGX_IOVS_PREALLOCATE];
> >> -#if (NGX_HAVE_SENDFILE_NODISKIO)
> >> -    ngx_uint_t       ebusy;
> >> -#endif
> >> 
> >>     wev = c->write;
> >> 
> >> 
> 
> Looks good to me.
> 
> >> Full patch, updated:
> > 
> > [...]
> > 
> >> diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
> >> --- a/src/os/unix/ngx_freebsd_sendfile_chain.c
> >> +++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
> > 
> > [...]
> > 
> >> @@ -199,7 +192,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
> >>                     eintr = 1;
> >>                     break;
> >> 
> >> -#if (NGX_HAVE_AIO_SENDFILE)
> >> +#if (NGX_HAVE_SENDFILE_NODISKIO)
> >>                 case NGX_EBUSY:
> >>                     ebusy = 1;
> >>                     break;
> >> @@ -258,35 +251,24 @@ ngx_freebsd_sendfile_chain(ngx_connectio
> >>             if (sent == 0) {
> >>                 c->busy_count++;
> >> 
> > 
> > Err, mismerge here.  Should be:
> > 
> > diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
> > --- a/src/os/unix/ngx_freebsd_sendfile_chain.c
> > +++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
> > @@ -245,7 +245,7 @@ ngx_freebsd_sendfile_chain(ngx_connectio
> > 
> >         in = ngx_chain_update_sent(in, sent);
> > 
> > -#if (NGX_HAVE_AIO_SENDFILE)
> > +#if (NGX_HAVE_SENDFILE_NODISKIO)
> > 
> >         if (ebusy) {
> >             if (sent == 0) {
> > 
> 
> Looks like this hunk appears in the updated patch 3,
> should belong to patch 2 (SF_NODISKIO refactoring).

Sure, thanks for noticing.  Fixed and committed.

-- 
Maxim Dounin
http://mdounin.ru/

From mandeep-singh.chhabra at thalesgroup.com  Tue Dec 28 11:56:50 2021
From: mandeep-singh.chhabra at thalesgroup.com (CHHABRA Mandeep Singh)
Date: Tue, 28 Dec 2021 11:56:50 +0000
Subject: [PATCH] Add provision to fetch certificate chain from Nginx
Message-ID: 

# HG changeset patch
# User Mandeep Singh Chhabra 
# Date 1640691269 -19800
#      Tue Dec 28 17:04:29 2021 +0530
# Node ID 9baaef976ac80f05107b60801ebe6559cdb2cbc6
# Parent  b002ad258f1d70924dc13d8f4bc0cc44362f0d0a
Add provision to fetch certificate chain from Nginx

The change adds a new variable ('ssl_client_cert_chain') to the
existing set of variables. It is being part of the http's SSL
module. With this, the middleware can fetch the certificate chain
from Nginx using the variable mentioned. The variable returns
a verified chain of certificates.
If the trust anchor is a root certificate (self signed) which has
issued an intermediate certificate and the client certificate is
issued by the intermediate certificate. The variable ('ssl_client_cert_chain')
will return three certificates (rootCert -> intermediateCert -> clientCert)

diff -r b002ad258f1d -r 9baaef976ac8 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c         Mon Dec 27 19:49:26 2021 +0300
+++ b/src/event/ngx_event_openssl.c      Tue Dec 28 17:04:29 2021 +0530
@@ -5048,6 +5048,99 @@
     return NGX_ERROR;
}
+ngx_int_t
+ngx_ssl_get_verified_certificate_chain(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s)
+{
+    size_t   len;
+    BIO     *bio;
+    X509    *cert;
+    STACK_OF(X509)    *certs;
+
+    s->len = 0;
+
+    bio = BIO_new(BIO_s_mem());
+    if (bio == NULL) {
+        ngx_ssl_error(NGX_LOG_ALERT, c->log, 0, "BIO_new() failed");
+        return NGX_ERROR;
+    }
+
+    certs = SSL_get0_verified_chain(c->ssl->connection);
+    if (certs == NULL) {
+        ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "SSL_get0_verified_chain failed");
+        goto failed;
+    } else {
+        for (int i = 0; i < sk_X509_num(certs); i++) {
+            cert = sk_X509_value(certs, i);
+
+            if (PEM_write_bio_X509(bio, cert) == 0) {
+                ngx_ssl_error(NGX_LOG_ALERT, c->log, 0, "PEM_write_bio_X509() failed");
+                goto failed;
+            }
+        }
+    }
+
+    len = BIO_pending(bio);
+    s->len = len;
+
+    s->data = ngx_pnalloc(pool, len);
+    if (s->data == NULL) {
+        goto failed;
+    }
+
+    BIO_read(bio, s->data, len);
+    BIO_free(bio);
+
+    return NGX_OK;
+
+failed:
+    BIO_free(bio);
+
+    return NGX_ERROR;
+}
+
+ngx_int_t
+ngx_ssl_get_client_certificate_chain(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s)
+{
+    u_char      *p;
+    size_t       len;
+    ngx_uint_t   i;
+    ngx_str_t    cert_chain;
+
+    if (ngx_ssl_get_verified_certificate_chain(c, pool, &cert_chain) != NGX_OK) {
+        return NGX_ERROR;
+    }
+
+    if (cert_chain.len == 0) {
+        s->len = 0;
+        return NGX_OK;
+    }
+
+    len = cert_chain.len - 1;
+
+    for (i = 0; i < cert_chain.len - 1; i++) {
+        if (cert_chain.data[i] == LF) {
+            len++;
+        }
+    }
+
+    s->len = len;
+    s->data = ngx_pnalloc(pool, len);
+    if (s->data == NULL) {
+        ngx_ssl_error(NGX_LOG_ERR, c->log, 0, "ngx_pnalloc failed");
+        return NGX_ERROR;
+    }
+
+    p = s->data;
+
+    for (i = 0; i < cert_chain.len - 1; i++) {
+        *p++ = cert_chain.data[i];
+        if (cert_chain.data[i] == LF) {
+            *p++ = '\t';
+        }
+    }
+
+    return NGX_OK;
+}
 ngx_int_t
ngx_ssl_get_certificate(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s)
diff -r b002ad258f1d -r 9baaef976ac8 src/event/ngx_event_openssl.h
--- a/src/event/ngx_event_openssl.h        Mon Dec 27 19:49:26 2021 +0300
+++ b/src/event/ngx_event_openssl.h     Tue Dec 28 17:04:29 2021 +0530
@@ -276,6 +276,8 @@
     ngx_str_t *s);
ngx_int_t ngx_ssl_get_escaped_certificate(ngx_connection_t *c, ngx_pool_t *pool,
     ngx_str_t *s);
+ngx_int_t ngx_ssl_get_client_certificate_chain(ngx_connection_t *c, ngx_pool_t *pool,
+    ngx_str_t *s);
ngx_int_t ngx_ssl_get_subject_dn(ngx_connection_t *c, ngx_pool_t *pool,
     ngx_str_t *s);
ngx_int_t ngx_ssl_get_issuer_dn(ngx_connection_t *c, ngx_pool_t *pool,
diff -r b002ad258f1d -r 9baaef976ac8 src/http/modules/ngx_http_ssl_module.c
--- a/src/http/modules/ngx_http_ssl_module.c    Mon Dec 27 19:49:26 2021 +0300
+++ b/src/http/modules/ngx_http_ssl_module.c                Tue Dec 28 17:04:29 2021 +0530
@@ -370,6 +370,10 @@
     { ngx_string("ssl_client_raw_cert"), NULL, ngx_http_ssl_variable,
       (uintptr_t) ngx_ssl_get_raw_certificate,
       NGX_HTTP_VAR_CHANGEABLE, 0 },
+
+    { ngx_string("ssl_client_cert_chain"), NULL, ngx_http_ssl_variable,
+      (uintptr_t) ngx_ssl_get_client_certificate_chain,
+      NGX_HTTP_VAR_CHANGEABLE, 0 },
     { ngx_string("ssl_client_escaped_cert"), NULL, ngx_http_ssl_variable,
       (uintptr_t) ngx_ssl_get_escaped_certificate,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From xeioex at nginx.com  Tue Dec 28 14:25:08 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 28 Dec 2021 14:25:08 +0000
Subject: [njs] Version 0.7.1.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/35aca5cc5ea7
branches:  
changeset: 1790:35aca5cc5ea7
user:      Dmitry Volyntsev 
date:      Tue Dec 28 13:55:44 2021 +0000
description:
Version 0.7.1.

diffstat:

 CHANGES |  43 +++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 43 insertions(+), 0 deletions(-)

diffs (50 lines):

diff -r 752d3d8ab217 -r 35aca5cc5ea7 CHANGES
--- a/CHANGES	Sat Dec 25 22:45:30 2021 +0300
+++ b/CHANGES	Tue Dec 28 13:55:44 2021 +0000
@@ -1,3 +1,46 @@
+Changes with njs 0.7.1                                        28 Dec 2021
+
+    nginx modules:
+
+    *) Change: the "js_include" directive deprecated since 0.4.0 was
+       removed.
+
+    *) Change: PCRE/PCRE2-specific code was moved to the modules.
+       This ensures that njs uses the same RegExp library as nginx.
+
+    Core:
+
+    *) Feature: extended "fs" module. Added stat(), fstat()
+       and friends.
+
+    *) Change: default RegExp engine for CLI is switched
+       to PCRE2.
+
+    *) Bugfix: fixed decodeURI() and decodeURIComponent() with
+       invalid byte strings. The bug was introduced in 0.4.3.
+
+    *) Bugfix: fixed heap-use-after-free in await frame.
+       The bug was introduced in 0.7.0.
+
+    *) Bugfix: fixed WebCrypto sign() and verify() methods
+       with OpenSSL 3.0.
+
+    *) Bugfix: fixed exception throwing when RegExp match fails.
+       The bug was introduced in 0.1.15.
+
+    *) Bugfix: fixed catching of exception thrown in try block
+       of async function. The bug was introduced in 0.7.0.
+
+    *) Bugfix: fixed execution of async function in synchronous
+       context. The bug was introduced in 0.7.0.
+
+    *) Bugfix: fixed function redeclaration in CLI when interactive
+       mode is on. The bug was introduced in 0.6.2.
+
+    *) Bugfix: fixed typeof operator with DataView object.
+
+    *) Bugfix: eliminated information leak in Buffer.from().
+
 Changes with njs 0.7.0                                        19 Oct 2021
 
     nginx modules:

From xeioex at nginx.com  Tue Dec 28 14:25:09 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 28 Dec 2021 14:25:09 +0000
Subject: [njs] Added tag 0.7.1 for changeset 35aca5cc5ea7
Message-ID: 

details:   https://hg.nginx.org/njs/rev/25310e8f0ef5
branches:  
changeset: 1791:25310e8f0ef5
user:      Dmitry Volyntsev 
date:      Tue Dec 28 14:24:53 2021 +0000
description:
Added tag 0.7.1 for changeset 35aca5cc5ea7

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff -r 35aca5cc5ea7 -r 25310e8f0ef5 .hgtags
--- a/.hgtags	Tue Dec 28 13:55:44 2021 +0000
+++ b/.hgtags	Tue Dec 28 14:24:53 2021 +0000
@@ -46,3 +46,4 @@ 742ebceef2b5d15febc093172fe6174e427b26c8
 4adbe67b292af2adc0a6fde4ec6cb95dbba9470a 0.6.1
 dfba7f61745c7454ffdd55303a793206d0a9a84a 0.6.2
 8418bd4a4ce3114d57b4d75f913e8c4912bf4b5d 0.7.0
+35aca5cc5ea7582b80947caa1e3f4a4fb8ee232d 0.7.1

From mdounin at mdounin.ru  Tue Dec 28 15:33:52 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 28 Dec 2021 15:33:52 +0000
Subject: [nginx] Updated OpenSSL and PCRE used for win32 builds.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/8af85c66da94
branches:  
changeset: 7988:8af85c66da94
user:      Maxim Dounin 
date:      Tue Dec 28 17:56:16 2021 +0300
description:
Updated OpenSSL and PCRE used for win32 builds.

diffstat:

 misc/GNUmakefile |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (15 lines):

diff -r b002ad258f1d -r 8af85c66da94 misc/GNUmakefile
--- a/misc/GNUmakefile	Mon Dec 27 19:49:26 2021 +0300
+++ b/misc/GNUmakefile	Tue Dec 28 17:56:16 2021 +0300
@@ -6,9 +6,9 @@ TEMP =		tmp
 
 CC =		cl
 OBJS =		objs.msvc8
-OPENSSL =	openssl-1.1.1l
+OPENSSL =	openssl-1.1.1m
 ZLIB =		zlib-1.2.11
-PCRE =		pcre-8.44
+PCRE =		pcre2-10.39
 
 
 release: export

From mdounin at mdounin.ru  Tue Dec 28 15:33:55 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 28 Dec 2021 15:33:55 +0000
Subject: [nginx] nginx-1.21.5-RELEASE
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/d986378168fd
branches:  
changeset: 7989:d986378168fd
user:      Maxim Dounin 
date:      Tue Dec 28 18:28:37 2021 +0300
description:
nginx-1.21.5-RELEASE

diffstat:

 docs/xml/nginx/changes.xml |  52 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 52 insertions(+), 0 deletions(-)

diffs (62 lines):

diff -r 8af85c66da94 -r d986378168fd docs/xml/nginx/changes.xml
--- a/docs/xml/nginx/changes.xml	Tue Dec 28 17:56:16 2021 +0300
+++ b/docs/xml/nginx/changes.xml	Tue Dec 28 18:28:37 2021 +0300
@@ -5,6 +5,58 @@
 
 
 
+
+
+
+
+?????? nginx ?? ????????? ?????????? ? ??????????? PCRE2.
+
+
+now nginx is built with the PCRE2 library by default.
+
+
+
+
+
+?????? nginx ?????? ?????????? sendfile(SF_NODISKIO) ?? FreeBSD.
+
+
+now nginx always uses sendfile(SF_NODISKIO) on FreeBSD.
+
+
+
+
+
+????????? sendfile(SF_NOCACHE) ?? FreeBSD.
+
+
+support for sendfile(SF_NOCACHE) on FreeBSD.
+
+
+
+
+
+?????????? $ssl_curve.
+
+
+the $ssl_curve variable.
+
+
+
+
+
+??? ????????????? HTTP/2 ??? SSL ?????? ? ??????????? sendfile ? aio
+?????????? ????? ????????.
+
+
+connections might hang
+when using HTTP/2 without SSL with the "sendfile" and "aio" directives.
+
+
+
+
+
+
 
 
 

From mdounin at mdounin.ru  Tue Dec 28 15:33:58 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 28 Dec 2021 15:33:58 +0000
Subject: [nginx] release-1.21.5 tag
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/67408b4a12c0
branches:  
changeset: 7990:67408b4a12c0
user:      Maxim Dounin 
date:      Tue Dec 28 18:28:38 2021 +0300
description:
release-1.21.5 tag

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff -r d986378168fd -r 67408b4a12c0 .hgtags
--- a/.hgtags	Tue Dec 28 18:28:37 2021 +0300
+++ b/.hgtags	Tue Dec 28 18:28:38 2021 +0300
@@ -465,3 +465,4 @@ a68ac0677f8553b1f84d357bc9da114731ab5f47
 bfbc52374adcbf2f9060afd62de940f6fab3bba5 release-1.21.2
 2217a9c1d0b86026f22700b3c089545db1964f55 release-1.21.3
 39be8a682c58308d9399cddd57e37f9fdb7bdf3e release-1.21.4
+d986378168fd4d70e0121cabac274c560cca9bdf release-1.21.5

From fengyoulin at live.com  Tue Dec 28 15:36:22 2021
From: fengyoulin at live.com (=?utf-8?B?5bCBIOW5vOm6nw==?=)
Date: Tue, 28 Dec 2021 15:36:22 +0000
Subject: [PATCH] HTTP: keepalive_graceful_close support
In-Reply-To: 
References: 
 
Message-ID: 

Hello!

In my scenario, nginx is used with consul-template as a load balancer that implements service discovery. Each time a service instance is registered to or unregistered from consul?consul-template will trigger all nginx instances to reload. In a busy system, this happens very frequently, and each reload will cause many "keepalived" connections to be closed. The client based on golang got a lot of "server closed idle connection" and "EOF" errors.

Of course, the client should implement the "auto retry" logic, but I think we have a better way to completely solve this problem in nginx. My idea is, don't treat the keepalived http connections as idle connections, and send a "Connection: close" header in the next response. If no more requests come, the worker process will wait until "keepalive_timeout", then exit, and won't wait long. So, if we give the client a smaller "IdleConnTimeout", there is no race now. If someone doesn't want to use this patch, just leave "keepalive_graceful_close" unset.

According to your suggestion, I optimized my patch:

# HG changeset patch
# User Youlin Feng 
# Date 1640704599 -28800
# ? ? ?Tue Dec 28 23:16:39 2021 +0800
# Node ID b763dec013e5e26e8801614df74b45a85a540e8e
# Parent ?b002ad258f1d70924dc13d8f4bc0cc44362f0d0a
HTTP: keepalive_graceful_close support.

Previously, keepalived connections will be suddenly closed during worker
process exiting, without notification to the client. The client may be
sending a request when the connection is closed, resulting in an error.

With "keepalive_graceful_close on;", the client will receive an
"Connection: close" response header in the next request, then the
connection can be closed gracefully. If no more requests come, the worker
process will wait until "keepalive_timeout" and then exit.

diff -r b002ad258f1d -r b763dec013e5 src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c	Mon Dec 27 19:49:26 2021 +0300
+++ b/src/http/ngx_http_core_module.c	Tue Dec 28 23:16:39 2021 +0800
@@ -523,6 +523,13 @@
? ? ? ?offsetof(ngx_http_core_loc_conf_t, keepalive_disable),
? ? ? ?&ngx_http_core_keepalive_disable },
?
+ ? ?{ ngx_string("keepalive_graceful_close"),
+ ? ? ?NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG,
+ ? ? ?ngx_conf_set_flag_slot,
+ ? ? ?NGX_HTTP_LOC_CONF_OFFSET,
+ ? ? ?offsetof(ngx_http_core_loc_conf_t, keepalive_graceful_close),
+ ? ? ?NULL },
+
? ? ?{ ngx_string("satisfy"),
? ? ? ?NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
? ? ? ?ngx_conf_set_enum_slot,
@@ -3517,6 +3524,7 @@
? ? ?clcf->keepalive_timeout = NGX_CONF_UNSET_MSEC;
? ? ?clcf->keepalive_header = NGX_CONF_UNSET;
? ? ?clcf->keepalive_requests = NGX_CONF_UNSET_UINT;
+ ? ?clcf->keepalive_graceful_close = NGX_CONF_UNSET;
? ? ?clcf->lingering_close = NGX_CONF_UNSET_UINT;
? ? ?clcf->lingering_time = NGX_CONF_UNSET_MSEC;
? ? ?clcf->lingering_timeout = NGX_CONF_UNSET_MSEC;
@@ -3756,6 +3764,8 @@
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?prev->keepalive_header, 0);
? ? ?ngx_conf_merge_uint_value(conf->keepalive_requests,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?prev->keepalive_requests, 1000);
+ ? ?ngx_conf_merge_value(conf->keepalive_graceful_close,
+ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?prev->keepalive_graceful_close, 0);
? ? ?ngx_conf_merge_uint_value(conf->lingering_close,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?prev->lingering_close, NGX_HTTP_LINGERING_ON);
? ? ?ngx_conf_merge_msec_value(conf->lingering_time,
diff -r b002ad258f1d -r b763dec013e5 src/http/ngx_http_core_module.h
--- a/src/http/ngx_http_core_module.h	Mon Dec 27 19:49:26 2021 +0300
+++ b/src/http/ngx_http_core_module.h	Tue Dec 28 23:16:39 2021 +0800
@@ -381,6 +381,7 @@
?
? ? ?ngx_flag_t ? ?client_body_in_single_buffer;
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /* client_body_in_singe_buffer */
+ ? ?ngx_flag_t ? ?keepalive_graceful_close; /* keepalive_graceful_close */
? ? ?ngx_flag_t ? ?internal; ? ? ? ? ? ? ? ?/* internal */
? ? ?ngx_flag_t ? ?sendfile; ? ? ? ? ? ? ? ?/* sendfile */
? ? ?ngx_flag_t ? ?aio; ? ? ? ? ? ? ? ? ? ? /* aio */
diff -r b002ad258f1d -r b763dec013e5 src/http/ngx_http_header_filter_module.c
--- a/src/http/ngx_http_header_filter_module.c	Mon Dec 27 19:49:26 2021 +0300
+++ b/src/http/ngx_http_header_filter_module.c	Tue Dec 28 23:16:39 2021 +0800
@@ -197,6 +197,10 @@
? ? ? ? ?}
? ? ?}
?
+ ? ?if (r->keepalive && (ngx_terminate || ngx_exiting)) {
+ ? ? ? ?r->keepalive = 0;
+ ? ?}
+
? ? ?len = sizeof("HTTP/1.x ") - 1 + sizeof(CRLF) - 1
? ? ? ? ? ?/* the end of the header */
? ? ? ? ? ?+ sizeof(CRLF) - 1;
diff -r b002ad258f1d -r b763dec013e5 src/http/ngx_http_request.c
--- a/src/http/ngx_http_request.c	Mon Dec 27 19:49:26 2021 +0300
+++ b/src/http/ngx_http_request.c	Tue Dec 28 23:16:39 2021 +0800
@@ -2767,7 +2767,7 @@
? ? ?}
?
? ? ?if (!ngx_terminate
- ? ? ? ? && !ngx_exiting
+ ? ? ? ? && (!ngx_exiting || clcf->keepalive_graceful_close)
? ? ? ? ? && r->keepalive
? ? ? ? ? && clcf->keepalive_timeout > 0)
? ? ?{
@@ -3252,7 +3252,7 @@
? ? ?r->http_state = NGX_HTTP_KEEPALIVE_STATE;
?#endif
?
- ? ?c->idle = 1;
+ ? ?c->idle = !clcf->keepalive_graceful_close;
? ? ?ngx_reusable_connection(c, 1);
?
? ? ?ngx_add_timer(rev, clcf->keepalive_timeout);


From: nginx-devel  on behalf of Maxim Dounin 
Sent: Monday, December 27, 2021 22:08
To: nginx-devel at nginx.org 
Subject: Re: [PATCH] HTTP: keepalive_graceful_close support 
?
Hello!

On Fri, Dec 17, 2021 at 01:59:40PM +0800, Youlin Feng wrote:

> # HG changeset patch
> # User Youlin Feng 
> # Date 1639718067 -28800
> #????? Fri Dec 17 13:14:27 2021 +0800
> # Node ID 54db7272bb0c9040eaf657f92bb02c9147d927e6
> # Parent? a7a77549265ef46f1f0fdb3897f4beabf9e09c40
> HTTP: keepalive_graceful_close support.
> 
> Previously, keepalived connections will be suddenly closed during worker
> process exiting, without notification to the client. The client may be
> sending a request when the connection is closed, resulting in an error.
> 
> With "keepalive_graceful_close on;", the client will receive an
> "Connection: close" response header in the last request, then the
> connection can be closed gracefully.

Thanks for the patch.

First of all, you may want to clarify which problem you are trying 
to solve.? Note that clients are expected to be prepared for a 
connection close by the server, and retry requests as appropriate.? 
If the client cannot handle this, probably there is a room for 
improvement in the client.? Further, there is an unavoidable race, 
and a client which cannot handle connection close is likely to see 
errors regardless of server behaviour.? See ticket #1022 
(https://trac.nginx.org/nginx/ticket/1022) for detailed 
explanation.

What probably nginx should do here is to disable keepalive and 
avoid sending "Connection: keep-alive" if it already knows that 
keepalive cannot be used when sending response headers.? This 
should reduce unneeded request failures and retries at no 
additional cost.

Patch:

# HG changeset patch
# User Maxim Dounin 
# Date 1640613750 -10800
#????? Mon Dec 27 17:02:30 2021 +0300
# Node ID 0430ee0554a19281abcc074981702ac201f2922e
# Parent? 4fa260f60bac03037017a33672da1ac75ac31408
Avoid sending "Connection: keep-alive" when shutting down.

When a worker process is shutting down, keepalive is not used: this is checked
before the ngx_http_set_keepalive() call in ngx_http_finalize_connection().
Yet the "Connection: keep-alive" header was still sent, even if we know that
the worker process is shutting down, potentially resulting in additional
requests being sent to the connection which is going to be closed anyway.
While clients are expected to be able to handle asynchronous close events
(see ticket #1022), it is certainly possible to send the "Connection: close"
header instead, informing the client that the connection is going to be closed
and potentially saving some unneeded work.

With this change, we additionally check for worker process shutdown just
before sending response headers, and disable keepalive accordingly.

diff --git a/src/http/ngx_http_header_filter_module.c b/src/http/ngx_http_header_filter_module.c
--- a/src/http/ngx_http_header_filter_module.c
+++ b/src/http/ngx_http_header_filter_module.c
@@ -197,6 +197,10 @@ ngx_http_header_filter(ngx_http_request_
???????? }
???? }
?
+??? if (r->keepalive && (ngx_terminate || ngx_exiting)) {
+??????? r->keepalive = 0;
+??? }
+
???? len = sizeof("HTTP/1.x ") - 1 + sizeof(CRLF) - 1
?????????? /* the end of the header */
?????????? + sizeof(CRLF) - 1;

-- 
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

From mdounin at mdounin.ru  Tue Dec 28 15:58:27 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 28 Dec 2021 18:58:27 +0300
Subject: [PATCH] Add provision to fetch certificate chain from Nginx
In-Reply-To: 
References: 
Message-ID: 

Hello!

On Tue, Dec 28, 2021 at 11:56:50AM +0000, CHHABRA Mandeep Singh wrote:

> # HG changeset patch
> # User Mandeep Singh Chhabra 
> # Date 1640691269 -19800
> #      Tue Dec 28 17:04:29 2021 +0530
> # Node ID 9baaef976ac80f05107b60801ebe6559cdb2cbc6
> # Parent  b002ad258f1d70924dc13d8f4bc0cc44362f0d0a
> Add provision to fetch certificate chain from Nginx
> 
> The change adds a new variable ('ssl_client_cert_chain') to the
> existing set of variables. It is being part of the http's SSL
> module. With this, the middleware can fetch the certificate chain
> from Nginx using the variable mentioned. The variable returns
> a verified chain of certificates.
> If the trust anchor is a root certificate (self signed) which has
> issued an intermediate certificate and the client certificate is
> issued by the intermediate certificate. The variable ('ssl_client_cert_chain')
> will return three certificates (rootCert -> intermediateCert -> clientCert)

Thanks for the patch.

You may want to be more specific about which problem you are 
trying to solve.  In particular, all root and intermediate 
certificates are expected to be known on the server.  If they 
aren't for some reason, it might be a good idea to clarify why 
they aren't known or reconsider particular configuration.

[...]

> +    p = s->data;
> +
> +    for (i = 0; i < cert_chain.len - 1; i++) {
> +        *p++ = cert_chain.data[i];
> +        if (cert_chain.data[i] == LF) {
> +            *p++ = '\t';
> +        }

Just a side note: certainly we are not going to introduce new 
variables using this syntax.

Also it might be a good idea to fix various style issues in the 
patch, but probably it make sense to resolve the "why it should be 
needed" question first.

[...]

-- 
Maxim Dounin
http://mdounin.ru/

From mdounin at mdounin.ru  Tue Dec 28 16:20:13 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 28 Dec 2021 19:20:13 +0300
Subject: [PATCH] HTTP: keepalive_graceful_close support
In-Reply-To: 
References: 
 
 
Message-ID: 

Hello!

On Tue, Dec 28, 2021 at 03:36:22PM +0000, ? ?? wrote:

> In my scenario, nginx is used with consul-template as a load 
> balancer that implements service discovery. Each time a service 
> instance is registered to or unregistered from 
> consul?consul-template will trigger all nginx instances to 
> reload. In a busy system, this happens very frequently, and each 
> reload will cause many "keepalived" connections to be closed. 
> The client based on golang got a lot of "server closed idle 
> connection" and "EOF" errors.

The "server closed idle connection" doesn't look like an error at 
all.  Not sure what "EOF" means, but either way it looks more like 
an issue in the client than something to address in nginx.

> Of course, the client should implement the "auto retry" logic, 
> but I think we have a better way to completely solve this 
> problem in nginx. My idea is, don't treat the keepalived http 
> connections as idle connections, and send a "Connection: close" 
> header in the next response. If no more requests come, the 
> worker process will wait until "keepalive_timeout", then exit, 
> and won't wait long. So, if we give the client a smaller 
> "IdleConnTimeout", there is no race now. If someone doesn't want 
> to use this patch, just leave "keepalive_graceful_close" unset.

To re-iterate: even with your patch (which basically returns the 
nginx behaviour before 0.5.15), the race is still here, since 
there are cases when keepalive connections are closed (notably, 
when keepalive_timeout expires, or when there are no free 
connections).  That is, the patch basically tries to hide client 
issues in some common case at the cost of keeping old worker 
processes alive till keepalive_timeout expiration.  Still, these 
issues can re-appear in other cases, and the only complete 
solution can be implemented on the client.

-- 
Maxim Dounin
http://mdounin.ru/

From pgnet.dev at gmail.com  Tue Dec 28 17:45:19 2021
From: pgnet.dev at gmail.com (PGNet Dev)
Date: Tue, 28 Dec 2021 12:45:19 -0500
Subject: nginx 1.21.5 + PCRE2 build fail @ naxsi
Message-ID: <7f645af7-bb34-34fa-8677-729c2ce97781@gmail.com>

with pcre2 enabled for new-version build, builds with naxsi now fail

i'd reported @ naxsi upstream issue,

   https://github.com/nbs-system/naxsi/issues/580

comment there suggests,

"Possibly connected commits:

     nginx/nginx at 931acbf
     nginx/nginx at d5f1f16 (mentions NAXSI)
     nginx/nginx at c6fec0b
"

2nd issue,

   https://github.com/nginx/nginx/commit/d5f1f169bc71d32b96960266d54e189c69af00ba

specifically mentions NAXSI, suggesting it was tested/working before commit to release 1.21.5

unclear if i've missed something required, or still an issue :-/

has anyone here yet been successful witjh v1.21.5 + PCRE2 + naxsi ?

From mdounin at mdounin.ru  Tue Dec 28 18:14:18 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 28 Dec 2021 21:14:18 +0300
Subject: nginx 1.21.5 + PCRE2 build fail @ naxsi
In-Reply-To: <7f645af7-bb34-34fa-8677-729c2ce97781@gmail.com>
References: <7f645af7-bb34-34fa-8677-729c2ce97781@gmail.com>
Message-ID: 

Hello!

On Tue, Dec 28, 2021 at 12:45:19PM -0500, PGNet Dev wrote:

> with pcre2 enabled for new-version build, builds with naxsi now fail
> 
> i'd reported @ naxsi upstream issue,
> 
>    https://github.com/nbs-system/naxsi/issues/580
> 
> comment there suggests,
> 
> "Possibly connected commits:
> 
>      nginx/nginx at 931acbf
>      nginx/nginx at d5f1f16 (mentions NAXSI)
>      nginx/nginx at c6fec0b
> "
> 
> 2nd issue,
> 
>    https://github.com/nginx/nginx/commit/d5f1f169bc71d32b96960266d54e189c69af00ba
> 
> specifically mentions NAXSI, suggesting it was tested/working before commit to release 1.21.5
> 
> unclear if i've missed something required, or still an issue :-/
> 
> has anyone here yet been successful witjh v1.21.5 + PCRE2 + naxsi ?

The NAXSI bug mentioned in the second commit needs to be fixed 
before it will be possible to build NAXSI with PCRE2.

-- 
Maxim Dounin
http://mdounin.ru/

From pgnet.dev at gmail.com  Tue Dec 28 19:12:50 2021
From: pgnet.dev at gmail.com (PGNet Dev)
Date: Tue, 28 Dec 2021 14:12:50 -0500
Subject: nginx 1.21.5 + PCRE2 build fail @ naxsi
In-Reply-To: 
References: <7f645af7-bb34-34fa-8677-729c2ce97781@gmail.com>
 
Message-ID: 

On 12/28/21 13:14, Maxim Dounin wrote:
> The NAXSI bug mentioned in the second commit needs to be fixed
> before it will be possible to build NAXSI with PCRE2.

Noted, & mentioned @ upstream bug (afaict, no prior relevant bug ?)

thanks!

fwiw, without naxsi, nginx 1.21.5 + pcre2 fails also @ ngx_http_lua_module

   https://docs.nginx.com/nginx/admin-guide/dynamic-modules/lua/

with

   grep lua_module nginx.conf

     load_module /usr/local/nginx-modules/ngx_http_lua_module.so;


seems that's a separate issue, and just new/noted here

   https://github.com/openresty/lua-nginx-module/issues/1984


From devashi.tandon at appsentinels.ai  Wed Dec 29 06:30:11 2021
From: devashi.tandon at appsentinels.ai (Devashi Tandon)
Date: Wed, 29 Dec 2021 06:30:11 +0000
Subject: Using single persistent socket to send subrequests
Message-ID: 

Hi,

We have a auth module in our code that sends requests to a server and waits for response to approve the request before proceeding to forward the request to a proxy server.

We use the function ngx_http_post_request to post the subrequest.

As I understand, this function adds the request to a queue which is then processed by ngx_http_run_posted_requests function.

We observe that every single subrequest is sent over a new socket connection. Unfortunately, when we scale to more than 1000 concurrent subrequests, we start seeing socket failures.

Is there a way to specify to this function, to maintain a persistent socket connection with the auth server and reuse the same socket connection for sending multiple subrequests?

Any help is appreciated.

Thanks,
Devashi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From mdounin at mdounin.ru  Wed Dec 29 14:37:53 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 29 Dec 2021 17:37:53 +0300
Subject: Using single persistent socket to send subrequests
In-Reply-To: 
References: 
Message-ID: 

Hello!

On Wed, Dec 29, 2021 at 06:30:11AM +0000, Devashi Tandon wrote:

> We have a auth module in our code that sends requests to a 
> server and waits for response to approve the request before 
> proceeding to forward the request to a proxy server.
> 
> We use the function ngx_http_post_request to post the 
> subrequest.
> 
> As I understand, this function adds the request to a queue which 
> is then processed by ngx_http_run_posted_requests function.
> 
> We observe that every single subrequest is sent over a new 
> socket connection. Unfortunately, when we scale to more than 
> 1000 concurrent subrequests, we start seeing socket failures.
> 
> Is there a way to specify to this function, to maintain a 
> persistent socket connection with the auth server and reuse the 
> same socket connection for sending multiple subrequests?

In no particular order:

- Using ngx_http_post_request() directly might not be a good idea.  
  Consider using ngx_http_subrequest() instead.

- For the particular task there is the auth_request module 
  (https://nginx.org/en/docs/http/ngx_http_auth_request_module.html).  
  It might be a good idea to use it instead of rolling your own 
  module.

- Every subrequest is processed according to the configuration 
  specified in the configuration file, much like any other 
  request.  To use persistent connections to upstream servers you 
  have to configure nginx to do so as usual, see 
  http://nginx.org/r/keepalive.

Hope this helps.

-- 
Maxim Dounin
http://mdounin.ru/

From xeioex at nginx.com  Wed Dec 29 16:44:34 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 29 Dec 2021 16:44:34 +0000
Subject: [njs] Version bump.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/e82abbb43383
branches:  
changeset: 1792:e82abbb43383
user:      Dmitry Volyntsev 
date:      Wed Dec 29 16:03:11 2021 +0000
description:
Version bump.

diffstat:

 src/njs.h |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r 25310e8f0ef5 -r e82abbb43383 src/njs.h
--- a/src/njs.h	Tue Dec 28 14:24:53 2021 +0000
+++ b/src/njs.h	Wed Dec 29 16:03:11 2021 +0000
@@ -11,7 +11,7 @@
 
 #include 
 
-#define NJS_VERSION                 "0.7.1"
+#define NJS_VERSION                 "0.7.2"
 
 
 #include                  /* STDOUT_FILENO, STDERR_FILENO */

From xeioex at nginx.com  Wed Dec 29 16:44:36 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 29 Dec 2021 16:44:36 +0000
Subject: [njs] Types: added TS definitions for fs.stat() and fs.lstat()
 methods.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/e172cd35cb8c
branches:  
changeset: 1793:e172cd35cb8c
user:      Dmitry Volyntsev 
date:      Wed Dec 29 16:03:12 2021 +0000
description:
Types: added TS definitions for fs.stat() and fs.lstat() methods.

diffstat:

 ts/njs_modules/fs.d.ts |  176 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 176 insertions(+), 0 deletions(-)

diffs (214 lines):

diff -r e82abbb43383 -r e172cd35cb8c ts/njs_modules/fs.d.ts
--- a/ts/njs_modules/fs.d.ts	Wed Dec 29 16:03:11 2021 +0000
+++ b/ts/njs_modules/fs.d.ts	Wed Dec 29 16:03:12 2021 +0000
@@ -70,6 +70,136 @@ declare module "fs" {
         name: string;
     }
 
+    /**
+     * Stats object provides information about a file.
+     *
+     * The objects is returned from fs.stat(), fs.lstat() and friends.
+     */
+    export interface Stats {
+        /**
+         * @returns `true` if the object describes a block device.
+         */
+        isBlockDevice(): boolean;
+        /**
+         * @returns `true` if the object describes a character device.
+         */
+        isCharacterDevice(): boolean;
+        /**
+         * @returns `true` if the object describes a file system directory.
+         */
+        isDirectory(): boolean;
+        /**
+         * @returns `true` if the object describes a first-in-first-out (FIFO) pipe.
+         */
+        isFIFO(): boolean;
+        /**
+         * @returns `true` if the object describes a regular file.
+         */
+        isFile(): boolean;
+        /**
+         * @returns `true` if the object describes a socket.
+         */
+        isSocket(): boolean;
+        /**
+         * @returns `true` if the object describes a symbolic link.
+         */
+        isSymbolicLink(): boolean;
+
+        /**
+         * The numeric identifier of the device containing the file.
+         */
+        dev: number;
+
+        /**
+         * The file system specific "Inode" number for the file.
+         */
+        ino: number;
+
+        /**
+         * A bit-field describing the file type and mode.
+         */
+        mode: number;
+
+        /**
+         * The number of hard-links that exist for the file.
+         */
+        nlink: number;
+
+        /**
+         * The numeric user identifier of the user that owns the file (POSIX).
+         */
+        uid: number;
+
+        /**
+         * The numeric group identifier of the group that owns the file (POSIX).
+         */
+        gid: number;
+
+        /**
+         * A numeric device identifier if the file represents a device.
+         */
+        rdev: number;
+
+        /**
+         * The size of the file in bytes.
+         */
+        size: number;
+
+        /**
+         * The file system block size for i/o operations.
+         */
+        blksize: number;
+
+        /**
+         * The number of blocks allocated for this file.
+         */
+        blocks: number;
+
+        /**
+         * The timestamp indicating the last time this file was accessed expressed
+         * in milliseconds since the POSIX Epoch.
+         */
+        atimeMs: number;
+
+        /**
+         * The timestamp indicating the last time this file was modified expressed
+         * in milliseconds since the POSIX Epoch.
+         */
+        mtimeMs: number;
+
+        /**
+         * The timestamp indicating the last time this file was changed expressed
+         * in milliseconds since the POSIX Epoch.
+         */
+        ctimeMs: number;
+
+        /**
+         * The timestamp indicating the creation time of this file expressed
+         * in milliseconds since the POSIX Epoch.
+         */
+        birthtimeMs: number;
+
+        /**
+         * The timestamp indicating the last time this file was accessed.
+         */
+        atime: Date;
+
+        /**
+         * The timestamp indicating the last time this file was modified.
+         */
+        mtime: Date;
+
+        /**
+         * The timestamp indicating the last time this file was changed.
+         */
+        ctime: Date;
+
+        /**
+         * The timestamp indicating the creation time of this file.
+         */
+        birthtime: Date;
+    }
+
     type WriteFileOptions = {
         mode?: number;
         flag?: OpenMode;
@@ -127,6 +257,18 @@ declare module "fs" {
         appendFile(path: PathLike, data: NjsStringOrBuffer, options?: WriteFileOptions): Promise;
 
         /**
+         * Asynchronously retrieves `fs.Stats` object for the symbolic link referred to by `path`.
+         * See `lstat(2)` for more details.
+         *
+         * @since 0.7.1
+         * @param path A path to a file.
+         * @param options An object with the following optional keys:
+         *   - `throwIfNoEntry` - Whether an exception will be thrown if no file system entry exists,
+         *      rather than returning undefined, defaults to `true`.
+         */
+        lstat(path: PathLike, options?: { throwIfNoEntry?: boolean; }): Promise;
+
+        /**
          * Asynchronously creates a directory at the specified `path`.
          *
          * @since 0.4.2
@@ -191,6 +333,17 @@ declare module "fs" {
         rmdir(path: PathLike): Promise;
 
         /**
+         * Asynchronously retrieves `fs.Stats` object for the specified `path`.
+         *
+         * @since 0.7.1
+         * @param path A path to a file.
+         * @param options An object with the following optional keys:
+         *   - `throwIfNoEntry` - Whether an exception will be thrown if no file system entry exists,
+         *      rather than returning undefined, defaults to `true`.
+         */
+        stat(path: PathLike, options?: { throwIfNoEntry?: boolean; }): Promise;
+
+        /**
          * Asynchronously creates the link called `path` pointing to `target` using `symlink(2)`.
          * Relative targets are relative to the link?s parent directory.
          *
@@ -267,6 +420,18 @@ declare module "fs" {
         appendFileSync(path: PathLike, data: NjsStringOrBuffer, options?: WriteFileOptions): void;
 
         /**
+         * Synchronously retrieves `fs.Stats` object for the symbolic link referred to by path.
+         * See `lstat(2)` for more details.
+         *
+         * @since 0.7.1
+         * @param path A path to a file.
+         * @param options An object with the following optional keys:
+         *   - `throwIfNoEntry` - Whether an exception will be thrown if no file system entry exists,
+         *      rather than returning undefined, defaults to `true`.
+         */
+        lstatSync(path: PathLike, options?: { throwIfNoEntry?: boolean; }): Stats;
+
+        /**
          * Synchronously creates a directory at the specified `path`.
          *
          * @since 0.4.2
@@ -341,6 +506,17 @@ declare module "fs" {
         rmdirSync(path: PathLike): void;
 
         /**
+         * Synchronously retrieves `fs.Stats` object for the specified path.
+         *
+         * @since 0.7.1
+         * @param path A path to a file.
+         * @param options An object with the following optional keys:
+         *   - `throwIfNoEntry` - Whether an exception will be thrown if no file system entry exists,
+         *      rather than returning undefined, defaults to `true`.
+         */
+        statSync(path: PathLike, options?: { throwIfNoEntry?: boolean; }): Stats;
+
+        /**
          * Synchronously creates the link called `path` pointing to `target` using `symlink(2)`.
          * Relative targets are relative to the link?s parent directory.
          *

From xeioex at nginx.com  Wed Dec 29 16:44:38 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 29 Dec 2021 16:44:38 +0000
Subject: [njs] Added description for recently added configure options.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/001bb59dc3a8
branches:  
changeset: 1794:001bb59dc3a8
user:      Dmitry Volyntsev 
date:      Wed Dec 29 16:03:13 2021 +0000
description:
Added description for recently added configure options.

diffstat:

 auto/help |  15 +++++++++++++++
 1 files changed, 15 insertions(+), 0 deletions(-)

diffs (25 lines):

diff -r e172cd35cb8c -r 001bb59dc3a8 auto/help
--- a/auto/help	Wed Dec 29 16:03:12 2021 +0000
+++ b/auto/help	Wed Dec 29 16:03:13 2021 +0000
@@ -14,6 +14,21 @@ default: "$NJS_CC_OPT"
 default: "$NJS_LD_OPT"
   --ar=FILE                 set static linking program, default: "$AR"
 
+  --no-pcre                 disables PCRE/PCRE2 discovery for RegExp
+                            backend. This flag allows to build PCRE/PCRE2
+                            outside of libnjs.a.  When this option is enabled
+                            functions described in njs_regex.h are not built.
+                            Instead this functions are expected to be provided
+                            while linking.
+
+  --no-pcre2                disables PCRE2 discovery for RegExp backend.
+                            When this option is enabled only PCRE library
+                            is discovered.
+
+  --no-openssl              disables OpenSSL discovery. When this option is
+                            enabled OpenSSL dependant code is not built as a
+                            part of libnjs.a.
+
   --address-sanitizer=YES   enables build with address sanitizer, \
 default: "$NJS_ADDRESS_SANITIZER"
   --debug=YES               enables additional runtime checks, \

From devashi.tandon at appsentinels.ai  Thu Dec 30 07:58:33 2021
From: devashi.tandon at appsentinels.ai (Devashi Tandon)
Date: Thu, 30 Dec 2021 07:58:33 +0000
Subject: Using single persistent socket to send subrequests
In-Reply-To: 
References: 
 
Message-ID: 

Hi Maxim,

Is HTTP Pipelining supported in NGINX? How can I pipeline requests?

I have the following configuration:

location /auth {
                internal;
                proxy_connect_timeout 5000ms;
                proxy_read_timeout    5000ms;
                proxy_http_version 1.1;
                proxy_set_header Connection "keep-alive";
                proxy_pass http://ext-authz-upstream-server;
        }
    }

    upstream ext-authz-upstream-server {
                server 172.20.10.6:9006;
                keepalive 4;
    }


However, when I create 100 simultaneous connections, they are all sent via a different source port which means that a new socket connection is created everytime. How can I pipeline requests over 4 connections with keepalive configuration set to 4?

Thanks,
Devashi

________________________________
From: nginx-devel  on behalf of Maxim Dounin 
Sent: Wednesday, December 29, 2021 8:07 PM
To: nginx-devel at nginx.org 
Subject: Re: Using single persistent socket to send subrequests

Hello!

On Wed, Dec 29, 2021 at 06:30:11AM +0000, Devashi Tandon wrote:

> We have a auth module in our code that sends requests to a
> server and waits for response to approve the request before
> proceeding to forward the request to a proxy server.
>
> We use the function ngx_http_post_request to post the
> subrequest.
>
> As I understand, this function adds the request to a queue which
> is then processed by ngx_http_run_posted_requests function.
>
> We observe that every single subrequest is sent over a new
> socket connection. Unfortunately, when we scale to more than
> 1000 concurrent subrequests, we start seeing socket failures.
>
> Is there a way to specify to this function, to maintain a
> persistent socket connection with the auth server and reuse the
> same socket connection for sending multiple subrequests?

In no particular order:

- Using ngx_http_post_request() directly might not be a good idea.
  Consider using ngx_http_subrequest() instead.

- For the particular task there is the auth_request module
  (https://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
  It might be a good idea to use it instead of rolling your own
  module.

- Every subrequest is processed according to the configuration
  specified in the configuration file, much like any other
  request.  To use persistent connections to upstream servers you
  have to configure nginx to do so as usual, see
  http://nginx.org/r/keepalive.

Hope this helps.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From mdounin at mdounin.ru  Thu Dec 30 08:17:48 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 30 Dec 2021 11:17:48 +0300
Subject: Using single persistent socket to send subrequests
In-Reply-To: 
References: 
 
 
Message-ID: 

Hello!

On Thu, Dec 30, 2021 at 07:58:33AM +0000, Devashi Tandon wrote:

>     upstream ext-authz-upstream-server {
>                 server 172.20.10.6:9006;
>                 keepalive 4;
>     }

[...]

> However, when I create 100 simultaneous connections, they are 
> all sent via a different source port which means that a new 
> socket connection is created everytime.

That's expected behaviour: the keepalive directive specifies the 
number of connections to cache, not the limit on the number of 
connections to the upstream server.  With many simultaneous 
requests nginx will open additional connections as needed.

> How can I pipeline requests over 4 connections with keepalive 
> configuration set to 4?

You cannot, pipelining is not supported by the proxy module.

If the goal is not pipelining but to limit the number of 
connections to upstream servers, the "server ... max_conns=..." and 
the "queue" directive as available in nginx-plus might be what you 
want, see here:

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#queue

Note well that such questions do not look like something related 
to nginx development.  A better mailing list for user-level 
question would be nginx at nginx.org, see here:

http://nginx.org/en/support.html

Hope this helps.

-- 
Maxim Dounin
http://mdounin.ru/

From mandeep-singh.chhabra at thalesgroup.com  Thu Dec 30 09:35:26 2021
From: mandeep-singh.chhabra at thalesgroup.com (CHHABRA Mandeep Singh)
Date: Thu, 30 Dec 2021 09:35:26 +0000
Subject: [PATCH] Add provision to fetch certificate chain from Nginx
In-Reply-To: 
References: 
 
Message-ID: 

Hi Maxim, 

Thanks for giving time to this.

As far as my understanding goes, the intermediate CA certificates are not required to be known to the server.
It is only the trust anchor(the root CA certificate) which is required to be known and trusted on the sever.
And in our case also, the root CA certificate is trusted for the web. 

I have tried to give a brief of the problem in the following section.

We have a product which supports multi-tenancy and uses Nginx as a reverse proxy.
There are different isolated domains which share the same trust anchor. But there could be difference
in the client certificate chain in different domains. There is a need to do some extra validations 
based on the CAs in the chain. To be more precise, we have option to specify if a CA could be used to
do client or user authentication. There is a possibility that in one domain, a CA is enabled for client authentication and in another , the same CA is disabled.

So, we need a way to get the certificate chain from Nginx, to do these extra validations, apart from what Nginx does i.e. checking if the chain could be verified.
But there is no way to get the chain, today.

This could be a common problem applicable to multiple use cases, depending upon how a product wants its CA to behave.
And we think, it could be a good to have feature in Nginx.   

Please let me know if I should be specify more details on the problem.

Regards
Mandeep

-----Original Message-----
From: nginx-devel  On Behalf Of Maxim Dounin
Sent: Tuesday, December 28, 2021 9:28 PM
To: nginx-devel at nginx.org
Subject: Re: [PATCH] Add provision to fetch certificate chain from Nginx

Hello!

On Tue, Dec 28, 2021 at 11:56:50AM +0000, CHHABRA Mandeep Singh wrote:

> # HG changeset patch
> # User Mandeep Singh Chhabra 
> # Date 1640691269 -19800
> #      Tue Dec 28 17:04:29 2021 +0530
> # Node ID 9baaef976ac80f05107b60801ebe6559cdb2cbc6
> # Parent  b002ad258f1d70924dc13d8f4bc0cc44362f0d0a
> Add provision to fetch certificate chain from Nginx
> 
> The change adds a new variable ('ssl_client_cert_chain') to the 
> existing set of variables. It is being part of the http's SSL module. 
> With this, the middleware can fetch the certificate chain from Nginx 
> using the variable mentioned. The variable returns a verified chain of 
> certificates.
> If the trust anchor is a root certificate (self signed) which has 
> issued an intermediate certificate and the client certificate is 
> issued by the intermediate certificate. The variable 
> ('ssl_client_cert_chain') will return three certificates (rootCert -> 
> intermediateCert -> clientCert)

Thanks for the patch.

You may want to be more specific about which problem you are trying to solve.  In particular, all root and intermediate certificates are expected to be known on the server.  If they aren't for some reason, it might be a good idea to clarify why they aren't known or reconsider particular configuration.

[...]

> +    p = s->data;
> +
> +    for (i = 0; i < cert_chain.len - 1; i++) {
> +        *p++ = cert_chain.data[i];
> +        if (cert_chain.data[i] == LF) {
> +            *p++ = '\t';
> +        }

Just a side note: certainly we are not going to introduce new variables using this syntax.

Also it might be a good idea to fix various style issues in the patch, but probably it make sense to resolve the "why it should be needed" question first.

[...]

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

From richagaur586 at gmail.com  Thu Dec 30 11:47:23 2021
From: richagaur586 at gmail.com (Richa Gaur)
Date: Thu, 30 Dec 2021 17:17:23 +0530
Subject: Stream metrics at request level
In-Reply-To: 
References: 
Message-ID: 

Hello,

I am using nginx as an L4 load balancer. I need to expose some stream level
metrics like throughput etc. I am using nginx-module-stream-sts
 for calculating metrics.

However, this module operates at nginx-stream-log-phase which is called
just before closing the connection. In case, if the underlying connection
is persistent and several requests are being made on the same connection,
the metrics are calculated at the end which does not depict the true
picture.

I tried looking at the stream-module code and based on preliminary
observations it looks like the stream phases are called for each stream
session and not for each tcp request (analogous to http request). So, even
if I register the handler at an earlier phase, let's say
nginx-stream-content-phase, it would be called only once in the connection
lifecycle instead of being called for every request.

Please let me know if my understanding is correct and if there is any
mechanism by which I can calculate metrics for each TCP request and show
real time data.

Any help would be appreciated.

Thanks and Regards,
Richa Gaur
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From mdounin at mdounin.ru  Thu Dec 30 13:26:10 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 30 Dec 2021 13:26:10 +0000
Subject: [nginx] Version bump.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/57581198e51e
branches:  
changeset: 7991:57581198e51e
user:      Maxim Dounin 
date:      Wed Dec 29 22:59:53 2021 +0300
description:
Version bump.

diffstat:

 src/core/nginx.h |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff -r 67408b4a12c0 -r 57581198e51e src/core/nginx.h
--- a/src/core/nginx.h	Tue Dec 28 18:28:38 2021 +0300
+++ b/src/core/nginx.h	Wed Dec 29 22:59:53 2021 +0300
@@ -9,8 +9,8 @@
 #define _NGINX_H_INCLUDED_
 
 
-#define nginx_version      1021005
-#define NGINX_VERSION      "1.21.5"
+#define nginx_version      1021006
+#define NGINX_VERSION      "1.21.6"
 #define NGINX_VER          "nginx/" NGINX_VERSION
 
 #ifdef NGX_BUILD

From mdounin at mdounin.ru  Thu Dec 30 13:26:13 2021
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 30 Dec 2021 13:26:13 +0000
Subject: [nginx] Events: fixed balancing between workers with EPOLLEXCLUSIVE.
Message-ID: 

details:   https://hg.nginx.org/nginx/rev/e2d07e4ec636
branches:  
changeset: 7992:e2d07e4ec636
user:      Maxim Dounin 
date:      Thu Dec 30 01:08:46 2021 +0300
description:
Events: fixed balancing between workers with EPOLLEXCLUSIVE.

Linux with EPOLLEXCLUSIVE usually notifies only the process which was first
to add the listening socket to the epoll instance.  As a result most of the
connections are handled by the first worker process (ticket #2285).  To fix
this, we re-add the socket periodically, so other workers will get a chance
to accept connections.

diffstat:

 src/event/ngx_event.c        |   5 +++
 src/event/ngx_event.h        |   1 +
 src/event/ngx_event_accept.c |  58 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+), 0 deletions(-)

diffs (122 lines):

diff -r 57581198e51e -r e2d07e4ec636 src/event/ngx_event.c
--- a/src/event/ngx_event.c	Wed Dec 29 22:59:53 2021 +0300
+++ b/src/event/ngx_event.c	Thu Dec 30 01:08:46 2021 +0300
@@ -55,6 +55,7 @@ ngx_uint_t            ngx_accept_events;
 ngx_uint_t            ngx_accept_mutex_held;
 ngx_msec_t            ngx_accept_mutex_delay;
 ngx_int_t             ngx_accept_disabled;
+ngx_uint_t            ngx_use_exclusive_accept;
 
 
 #if (NGX_STAT_STUB)
@@ -644,6 +645,8 @@ ngx_event_process_init(ngx_cycle_t *cycl
 
 #endif
 
+    ngx_use_exclusive_accept = 0;
+
     ngx_queue_init(&ngx_posted_accept_events);
     ngx_queue_init(&ngx_posted_next_events);
     ngx_queue_init(&ngx_posted_events);
@@ -889,6 +892,8 @@ ngx_event_process_init(ngx_cycle_t *cycl
         if ((ngx_event_flags & NGX_USE_EPOLL_EVENT)
             && ccf->worker_processes > 1)
         {
+            ngx_use_exclusive_accept = 1;
+
             if (ngx_add_event(rev, NGX_READ_EVENT, NGX_EXCLUSIVE_EVENT)
                 == NGX_ERROR)
             {
diff -r 57581198e51e -r e2d07e4ec636 src/event/ngx_event.h
--- a/src/event/ngx_event.h	Wed Dec 29 22:59:53 2021 +0300
+++ b/src/event/ngx_event.h	Thu Dec 30 01:08:46 2021 +0300
@@ -462,6 +462,7 @@ extern ngx_uint_t             ngx_accept
 extern ngx_uint_t             ngx_accept_mutex_held;
 extern ngx_msec_t             ngx_accept_mutex_delay;
 extern ngx_int_t              ngx_accept_disabled;
+extern ngx_uint_t             ngx_use_exclusive_accept;
 
 
 #if (NGX_STAT_STUB)
diff -r 57581198e51e -r e2d07e4ec636 src/event/ngx_event_accept.c
--- a/src/event/ngx_event_accept.c	Wed Dec 29 22:59:53 2021 +0300
+++ b/src/event/ngx_event_accept.c	Thu Dec 30 01:08:46 2021 +0300
@@ -11,6 +11,9 @@
 
 
 static ngx_int_t ngx_disable_accept_events(ngx_cycle_t *cycle, ngx_uint_t all);
+#if (NGX_HAVE_EPOLLEXCLUSIVE)
+static void ngx_reorder_accept_events(ngx_listening_t *ls);
+#endif
 static void ngx_close_accepted_connection(ngx_connection_t *c);
 
 
@@ -314,6 +317,10 @@ ngx_event_accept(ngx_event_t *ev)
         }
 
     } while (ev->available);
+
+#if (NGX_HAVE_EPOLLEXCLUSIVE)
+    ngx_reorder_accept_events(ls);
+#endif
 }
 
 
@@ -420,6 +427,57 @@ ngx_disable_accept_events(ngx_cycle_t *c
 }
 
 
+#if (NGX_HAVE_EPOLLEXCLUSIVE)
+
+static void
+ngx_reorder_accept_events(ngx_listening_t *ls)
+{
+    ngx_connection_t  *c;
+
+    /*
+     * Linux with EPOLLEXCLUSIVE usually notifies only the process which
+     * was first to add the listening socket to the epoll instance.  As
+     * a result most of the connections are handled by the first worker
+     * process.  To fix this, we re-add the socket periodically, so other
+     * workers will get a chance to accept connections.
+     */
+
+    if (!ngx_use_exclusive_accept) {
+        return;
+    }
+
+#if (NGX_HAVE_REUSEPORT)
+
+    if (ls->reuseport) {
+        return;
+    }
+
+#endif
+
+    c = ls->connection;
+
+    if (c->requests++ % 16 != 0
+        && ngx_accept_disabled <= 0)
+    {
+        return;
+    }
+
+    if (ngx_del_event(c->read, NGX_READ_EVENT, NGX_DISABLE_EVENT)
+        == NGX_ERROR)
+    {
+        return;
+    }
+
+    if (ngx_add_event(c->read, NGX_READ_EVENT, NGX_EXCLUSIVE_EVENT)
+        == NGX_ERROR)
+    {
+        return;
+    }
+}
+
+#endif
+
+
 static void
 ngx_close_accepted_connection(ngx_connection_t *c)
 {

From xeioex at nginx.com  Thu Dec 30 13:30:33 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 30 Dec 2021 13:30:33 +0000
Subject: [njs] Configure: fixed custom NJS_BUILD_DIR values support.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/1ad65f9d50f5
branches:  
changeset: 1795:1ad65f9d50f5
user:      Dmitry Volyntsev 
date:      Wed Dec 29 16:56:47 2021 +0000
description:
Configure: fixed custom NJS_BUILD_DIR values support.

The issue was introduced in 4d4657128baf (0.7.1).

This closes #454 issue on Github.

diffstat:

 auto/make |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (13 lines):

diff -r 001bb59dc3a8 -r 1ad65f9d50f5 auto/make
--- a/auto/make	Wed Dec 29 16:03:13 2021 +0000
+++ b/auto/make	Wed Dec 29 16:56:47 2021 +0000
@@ -7,8 +7,8 @@
 echo "creating $NJS_MAKEFILE"
 
 mkdir -p $NJS_BUILD_DIR/src
-mkdir -p $NJS_BUILD_DIR/build
 mkdir -p $NJS_BUILD_DIR/external
+mkdir -p $NJS_BUILD_DIR/$NJS_BUILD_DIR
 mkdir -p $NJS_BUILD_DIR/test
 
 njs_modules_c=$NJS_BUILD_DIR/njs_modules.c

From xeioex at nginx.com  Thu Dec 30 13:30:35 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 30 Dec 2021 13:30:35 +0000
Subject: [njs] Tests: fixed "fs" tests with custom build directory.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/d11d962c40cd
branches:  
changeset: 1796:d11d962c40cd
user:      Dmitry Volyntsev 
date:      Wed Dec 29 17:19:55 2021 +0000
description:
Tests: fixed "fs" tests with custom build directory.

diffstat:

 test/fs/methods.t.js     |  16 ++++++++--------
 test/fs/promises_01.t.js |   2 +-
 test/fs/promises_02.t.js |   2 +-
 test/fs/promises_03.t.js |   2 +-
 test/fs/promises_04.t.js |   7 +++----
 test/fs/promises_05.t.js |   5 ++---
 test/fs/promises_06.t.js |   5 ++++-
 test/fs/promises_07.t.js |   4 ++--
 test/fs/promises_08.t.js |   2 +-
 test/fs/promises_09.t.js |   2 +-
 test/harness/compatFs.js |   3 +++
 test/help                |   2 --
 test/test262             |   2 +-
 13 files changed, 28 insertions(+), 26 deletions(-)

diffs (220 lines):

diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/methods.t.js
--- a/test/fs/methods.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/methods.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -9,7 +9,7 @@ function p(args, default_opts) {
     let fname = params.args[0];
 
     if (fname[0] == '@') {
-        let gen = `build/test/fs_test_${Math.round(Math.random() * 1000000)}`;
+        let gen = `${test_dir}/fs_test_${Math.round(Math.random() * 1000000)}`;
         params.args = params.args.map(v => v);
         params.args[0] = gen + fname.slice(1);
     }
@@ -368,9 +368,9 @@ async function realpath_test(params) {
 }
 
 let realpath_tests = () => [
-    { args: ["build/test/.."],
-      check: (data) => data.endsWith("build") },
-    { args: ["build/test/", {encoding:'buffer'}],
+    { args: ["test/fs/.."],
+      check: (data) => data.endsWith("test") },
+    { args: ["test/fs/ascii", {encoding:'buffer'}],
       check: (data) => data instanceof Buffer },
 ];
 
@@ -478,14 +478,14 @@ let stat_tests = () => [
           return true;
       } },
 
-    { args: ["build/"],
+    { args: ["test/fs/ascii"],
       check: (st) => contains(Object.keys(st),
                               [ "atime", "atimeMs", "birthtime", "birthtimeMs",
                                 "blksize", "blocks", "ctime", "ctimeMs", "dev",
                                 "gid", "ino", "mode", "mtime", "mtimeMs","nlink",
                                 "rdev", "size", "uid" ]) },
 
-    { args: ["build/"],
+    { args: ["test/fs/ascii"],
       check: (st) => Object.keys(st).every(p => {
         let v = st[p];
         if (p == 'atime' || p == 'ctime' || p == 'mtime' || p == 'birthtime') {
@@ -503,7 +503,7 @@ let stat_tests = () => [
         return true;
       }) },
 
-    { args: ["build/"],
+    { args: ["test/fs/ascii"],
       check: (st) => ['atime', 'birthtime', 'ctime', 'mtime'].every(p => {
           let date = st[p].valueOf();
           let num = st[p + 'Ms'];
@@ -515,7 +515,7 @@ let stat_tests = () => [
           return true;
       }) },
 
-    { args: ["build/"],
+    { args: [test_dir],
       check: (st) => ['isBlockDevice',
                       'isCharacterDevice',
                       'isDirectory',
diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/promises_01.t.js
--- a/test/fs/promises_01.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/promises_01.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -3,7 +3,7 @@ includes: [compareArray.js, compatFs.js]
 flags: [async]
 ---*/
 
-var fname = 'build/test/fs_promises_01';
+var fname = `${test_dir}/fs_promises_01`;
 
 let stages = [];
 
diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/promises_02.t.js
--- a/test/fs/promises_02.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/promises_02.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -3,7 +3,7 @@ includes: [compareArray.js, compatFs.js]
 flags: [async]
 ---*/
 
-var fname = 'build/test/fs_promises_02';
+var fname = `${test_dir}/fs_promises_02`;
 
 var testSync = new Promise((resolve, reject) => {
     var failed = false;
diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/promises_03.t.js
--- a/test/fs/promises_03.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/promises_03.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -3,7 +3,7 @@ includes: [compareArray.js, compatFs.js]
 flags: [async]
 ---*/
 
-var fname = 'build/test/fs_promises_03';
+var fname = `${test_dir}/fs_promises_03`;
 
 var testSync = () => new Promise((resolve, reject) => {
     try {
diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/promises_04.t.js
--- a/test/fs/promises_04.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/promises_04.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -3,10 +3,9 @@ includes: [compareArray.js, compatFs.js]
 flags: [async]
 ---*/
 
-var dname = 'build/test/';
-var fname = dname + 'fs_promises_04';
-var fname_utf8 = dname + 'fs_promises_???_04';
-var lname = dname + 'fs_promises_04_lnk';
+var fname = `${test_dir}/fs_promises_04`;
+var fname_utf8 = `${test_dir}/fs_promises_???_04`;
+var lname = `${test_dir}/fs_promises_lnk_04`;
 
 var testSync = () => new Promise((resolve, reject) => {
     try {
diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/promises_05.t.js
--- a/test/fs/promises_05.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/promises_05.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -3,9 +3,8 @@ includes: [compareArray.js, compatFs.js]
 flags: [async]
 ---*/
 
-var rname = 'build/test/';
-var dname = rname + 'fs_promises_05';
-var dname_utf8 = rname + 'fs_promises_???_05';
+var dname = `${test_dir}/fs_promises_05`;
+var dname_utf8 = `${test_dir}/fs_promises_???_05`;
 var fname = (d) => d + '/fs_promises_05_file';
 
 var testSync = () => new Promise((resolve, reject) => {
diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/promises_06.t.js
--- a/test/fs/promises_06.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/promises_06.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -3,7 +3,7 @@ includes: [compareArray.js, compatFs.js]
 flags: [async]
 ---*/
 
-var dname = 'build/test/';
+var dname = `${test_dir}/`;
 var fname = (d) => d + '/fs_promises_06_file';
 var fname_utf8 = (d) => d + '/fs_promises_???_06';
 
@@ -73,6 +73,9 @@ Promise.resolve()
 .then(() => {
     stages.push("renameSync");
 })
+.catch((e) => {
+    console.log('test fs.renameSync failed', JSON.stringify(e));
+})
 
 .then(testCallback)
 .then(() => {
diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/promises_07.t.js
--- a/test/fs/promises_07.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/promises_07.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -3,8 +3,8 @@ includes: [compareArray.js, compatFs.js]
 flags: [async]
 ---*/
 
-var dname = 'build/test/fs_promises_07';
-var dname_utf8 = 'build/test/fs_promises_???_07';
+var dname = `${test_dir}/fs_promises_07`;
+var dname_utf8 = `${test_dir}/fs_promises_???_07`;
 var fname = (d) => d + '/fs_promises_07_file';
 var lname = (d) => d + '/fs_promises_07_link';
 var cname = (d) => d + '/fs_promises_???_07_dir';
diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/promises_08.t.js
--- a/test/fs/promises_08.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/promises_08.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -3,7 +3,7 @@ includes: [compareArray.js, compatFs.js]
 flags: [async]
 ---*/
 
-var dname = 'build/test/fs_promises_???_08/';
+var dname = `${test_dir}/fs_promises_???_08/`;
 var path = 'one/two/three/???';
 
 var wipePath = (root, path, nofail) => {
diff -r 1ad65f9d50f5 -r d11d962c40cd test/fs/promises_09.t.js
--- a/test/fs/promises_09.t.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/fs/promises_09.t.js	Wed Dec 29 17:19:55 2021 +0000
@@ -3,7 +3,7 @@ includes: [compareArray.js, compatFs.js]
 flags: [async]
 ---*/
 
-var root = 'build/test/';
+var root = test_dir;
 var dname = 'fs_promises_???_09/';
 var lname = 'fs_promises_???_09_lnk';
 var path = 'one/two/three/???';
diff -r 1ad65f9d50f5 -r d11d962c40cd test/harness/compatFs.js
--- a/test/harness/compatFs.js	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/harness/compatFs.js	Wed Dec 29 17:19:55 2021 +0000
@@ -9,3 +9,6 @@ if (typeof require == 'function') {
 function has_fs() {
     return fs;
 }
+
+let test_dir = process.env && process.env['NJS_TEST_DIR'] || 'build';
+test_dir = `${test_dir}/test`;
diff -r 1ad65f9d50f5 -r d11d962c40cd test/help
--- a/test/help	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/help	Wed Dec 29 17:19:55 2021 +0000
@@ -17,6 +17,4 @@ default: "$NJS_TEST_LOG"
 default: "$NJS_TEST_DIR"
   --verbose=YES             enables verbose output, \
 default: "$NJS_TEST_VERBOSE"
-  --leave=YES               disables removing of "$NJS_TEST_DIR", \
-default: "$NJS_TEST_LEAVE"
 END
diff -r 1ad65f9d50f5 -r d11d962c40cd test/test262
--- a/test/test262	Wed Dec 29 16:56:47 2021 +0000
+++ b/test/test262	Wed Dec 29 17:19:55 2021 +0000
@@ -19,7 +19,7 @@ for njs_test in $NJS_TESTS; do
 running $njs_test $njs_log
 END
 
-    if /bin/sh -c "($NJS_TEST_BINARY $NJS_TEST_DIR/$njs_test)" > $njs_log 2>&1; then
+    if /bin/sh -c "(NJS_TEST_DIR=$NJS_TEST_DIR $NJS_TEST_BINARY $NJS_TEST_DIR/$njs_test)" > $njs_log 2>&1; then
         njs_success=yes
     else
         njs_success=no

From xeioex at nginx.com  Thu Dec 30 13:30:37 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 30 Dec 2021 13:30:37 +0000
Subject: [njs] Configure: added --build-dir option.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/c714088503bc
branches:  
changeset: 1797:c714088503bc
user:      Dmitry Volyntsev 
date:      Wed Dec 29 17:20:09 2021 +0000
description:
Configure: added --build-dir option.

diffstat:

 auto/help    |   2 ++
 auto/init    |  16 ----------------
 auto/make    |   2 +-
 auto/options |   2 ++
 configure    |  20 +++++++++++++++++++-
 5 files changed, 24 insertions(+), 18 deletions(-)

diffs (94 lines):

diff -r d11d962c40cd -r c714088503bc auto/help
--- a/auto/help	Wed Dec 29 17:19:55 2021 +0000
+++ b/auto/help	Wed Dec 29 17:20:09 2021 +0000
@@ -14,6 +14,8 @@ default: "$NJS_CC_OPT"
 default: "$NJS_LD_OPT"
   --ar=FILE                 set static linking program, default: "$AR"
 
+  --build-dir=DIR           set build directory, default: "$NJS_BUILD_DIR"
+
   --no-pcre                 disables PCRE/PCRE2 discovery for RegExp
                             backend. This flag allows to build PCRE/PCRE2
                             outside of libnjs.a.  When this option is enabled
diff -r d11d962c40cd -r c714088503bc auto/init
--- a/auto/init	Wed Dec 29 17:19:55 2021 +0000
+++ b/auto/init	Wed Dec 29 17:20:09 2021 +0000
@@ -15,23 +15,7 @@ NJS_CFLAGS=${NJS_CFLAGS=}
 
 NJS_BUILD_DIR=${NJS_BUILD_DIR:-build}
 
-NJS_AUTOTEST=$NJS_BUILD_DIR/autotest
-NJS_AUTOCONF_ERR=$NJS_BUILD_DIR/autoconf.err
-NJS_AUTO_CONFIG_H=$NJS_BUILD_DIR/njs_auto_config.h
-NJS_MAKEFILE=$NJS_BUILD_DIR/Makefile
-
 NJS_LIB_MODULES=
-NJS_LIB_INCS="src $NJS_BUILD_DIR"
-
-test -d $NJS_BUILD_DIR || mkdir $NJS_BUILD_DIR
-
-> $NJS_AUTOCONF_ERR
-
-cat << END > $NJS_AUTO_CONFIG_H
-
-/* This file is auto-generated by configure */
-
-END
 
 NJS_LIBRT=
 
diff -r d11d962c40cd -r c714088503bc auto/make
--- a/auto/make	Wed Dec 29 17:19:55 2021 +0000
+++ b/auto/make	Wed Dec 29 17:20:09 2021 +0000
@@ -242,7 +242,7 @@ lib_test: $NJS_BUILD_DIR/njs_auto_config
 
 test262: njs
 
-	test/test262
+	test/test262 --binary=$NJS_BUILD_DIR/njs
 
 unit_test: $NJS_BUILD_DIR/njs_auto_config.h \\
 	$NJS_BUILD_DIR/njs_unit_test
diff -r d11d962c40cd -r c714088503bc auto/options
--- a/auto/options	Wed Dec 29 17:19:55 2021 +0000
+++ b/auto/options	Wed Dec 29 17:20:09 2021 +0000
@@ -31,6 +31,8 @@ do
         --ld-opt=*)                      NJS_LD_OPT="$value"                 ;;
         --ar=*)                          AR="$value"                         ;;
 
+        --build-dir=*)                   NJS_BUILD_DIR="$value"              ;;
+
         --address-sanitizer=*)           NJS_ADDRESS_SANITIZER="$value"      ;;
         --debug=*)                       NJS_DEBUG="$value"                  ;;
         --debug-memory=*)                NJS_DEBUG_MEMORY="$value"           ;;
diff -r d11d962c40cd -r c714088503bc configure
--- a/configure	Wed Dec 29 17:19:55 2021 +0000
+++ b/configure	Wed Dec 29 17:20:09 2021 +0000
@@ -14,8 +14,26 @@ set -e
 set -u
 
 . auto/init
+. auto/options
+
+NJS_AUTOTEST=$NJS_BUILD_DIR/autotest
+NJS_AUTOCONF_ERR=$NJS_BUILD_DIR/autoconf.err
+NJS_AUTO_CONFIG_H=$NJS_BUILD_DIR/njs_auto_config.h
+NJS_MAKEFILE=$NJS_BUILD_DIR/Makefile
+
+NJS_LIB_INCS="src $NJS_BUILD_DIR"
+
+test -d $NJS_BUILD_DIR || mkdir $NJS_BUILD_DIR
+
+> $NJS_AUTOCONF_ERR
+
+cat << END > $NJS_AUTO_CONFIG_H
+
+/* This file is auto-generated by configure */
+
+END
+
 . auto/os
-. auto/options
 . auto/cc
 . auto/types
 . auto/endianness

From xeioex at nginx.com  Thu Dec 30 13:30:38 2021
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 30 Dec 2021 13:30:38 +0000
Subject: [njs] Improved discovery of OpenSSL libraries.
Message-ID: 

details:   https://hg.nginx.org/njs/rev/9b112a44e540
branches:  
changeset: 1798:9b112a44e540
user:      Dmitry Volyntsev 
date:      Wed Dec 29 18:26:40 2021 +0000
description:
Improved discovery of OpenSSL libraries.

Trying to link again the library using --cc-opt and --ld-opt before
attempting to use the default.

This change is similar to 9e2e4d04dfc4 (0.7.1) made for PCRE.

diffstat:

 auto/openssl |  9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)

diffs (26 lines):

diff -r c714088503bc -r 9b112a44e540 auto/openssl
--- a/auto/openssl	Wed Dec 29 17:20:09 2021 +0000
+++ b/auto/openssl	Wed Dec 29 18:26:40 2021 +0000
@@ -13,7 +13,7 @@ if [ $NJS_OPENSSL = YES ]; then
     njs_feature_name=NJS_HAVE_OPENSSL
     njs_feature_run=yes
     njs_feature_incs=
-    njs_feature_libs="-lcrypto"
+    njs_feature_libs=""
     njs_feature_test="#include 
 
                       int main() {
@@ -24,6 +24,13 @@ if [ $NJS_OPENSSL = YES ]; then
                      }"
     . auto/feature
 
+    if [ $njs_found = no ]; then
+        njs_feature="OpenSSL library -lcrypto"
+        njs_feature_libs="-lcrypto"
+
+        . auto/feature
+    fi
+
 
     if [ $njs_found = yes ]; then
         njs_feature="OpenSSL version"

From fengyoulin at live.com  Fri Dec 31 14:24:13 2021
From: fengyoulin at live.com (=?iso-2022-jp?B?GyRCTUROWxsoQiAbJEJJdRsoQg==?=)
Date: Fri, 31 Dec 2021 14:24:13 +0000
Subject: [PATCH] HTTP: keepalive_graceful_close support
In-Reply-To: 
References: 
 
 
 
Message-ID: 

Hello, Maxim!

You are right, we can?t completely avoid this kind of problem, but we can reduce some. Maybe "keepalive_graceful_close" is not exact, it should be "wait_keepalive_on_exit". I mean, with this patch, I want to optimize the connection closure problem caused by the exit of the work process, is an optimization for a specific scene. By setting a shorter expiration time for the keepalived connection on the client side than the "keepalive_timeout" on the nginx side, we can really reduce the number of client retries.

Regards and happy new year.

From: Maxim Dounin
Sent: 2021?12?29? 0:20
To: nginx-devel at nginx.org
Subject: Re: [PATCH] HTTP: keepalive_graceful_close support

Hello!

On Tue, Dec 28, 2021 at 03:36:22PM +0000, ? ?? wrote:

> In my scenario, nginx is used with consul-template as a load
> balancer that implements service discovery. Each time a service
> instance is registered to or unregistered from
> consul?consul-template will trigger all nginx instances to
> reload. In a busy system, this happens very frequently, and each
> reload will cause many "keepalived" connections to be closed.
> The client based on golang got a lot of "server closed idle
> connection" and "EOF" errors.

The "server closed idle connection" doesn't look like an error at
all.  Not sure what "EOF" means, but either way it looks more like
an issue in the client than something to address in nginx.

> Of course, the client should implement the "auto retry" logic,
> but I think we have a better way to completely solve this
> problem in nginx. My idea is, don't treat the keepalived http
> connections as idle connections, and send a "Connection: close"
> header in the next response. If no more requests come, the
> worker process will wait until "keepalive_timeout", then exit,
> and won't wait long. So, if we give the client a smaller
> "IdleConnTimeout", there is no race now. If someone doesn't want
> to use this patch, just leave "keepalive_graceful_close" unset.

To re-iterate: even with your patch (which basically returns the
nginx behaviour before 0.5.15), the race is still here, since
there are cases when keepalive connections are closed (notably,
when keepalive_timeout expires, or when there are no free
connections).  That is, the patch basically tries to hide client
issues in some common case at the cost of keeping old worker
processes alive till keepalive_timeout expiration.  Still, these
issues can re-appear in other cases, and the only complete
solution can be implemented on the client.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: