From nginx at rastos.org Sun Apr 4 07:52:03 2021 From: nginx at rastos.org (Rastislav Stanik) Date: Sun, 4 Apr 2021 09:52:03 +0200 Subject: fsize and flastmod SSI commands Message-ID: <6682f478-e82a-0436-4d7b-02b3f8407f30@rastos.org> Hi A year ago there was a patch posted that adds support for fsize and flastmod SSI commands: http://mailman.nginx.org/pipermail/nginx-devel/2020-February/013003.html What's needs to happen to get that into official sources? -- Sincerely Rastislav Stanik From mdounin at mdounin.ru Sun Apr 4 15:56:25 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sun, 4 Apr 2021 18:56:25 +0300 Subject: fsize and flastmod SSI commands In-Reply-To: <6682f478-e82a-0436-4d7b-02b3f8407f30@rastos.org> References: <6682f478-e82a-0436-4d7b-02b3f8407f30@rastos.org> Message-ID: Hello! On Sun, Apr 04, 2021 at 09:52:03AM +0200, Rastislav Stanik wrote: > A year ago there was a patch posted that adds support for fsize and flastmod SSI > commands: > http://mailman.nginx.org/pipermail/nginx-devel/2020-February/013003.html > What's needs to happen to get that into official sources? Firt of all, questions and concerns from the previous review need answers. -- Maxim Dounin http://mdounin.ru/ From nginx at rastos.org Sun Apr 4 19:19:30 2021 From: nginx at rastos.org (Rastislav Stanik) Date: Sun, 4 Apr 2021 21:19:30 +0200 Subject: fsize and flastmod SSI commands In-Reply-To: References: <6682f478-e82a-0436-4d7b-02b3f8407f30@rastos.org> Message-ID: On 04/04/2021 17.56, Maxim Dounin wrote: > Hello! > > On Sun, Apr 04, 2021 at 09:52:03AM +0200, Rastislav Stanik wrote: > >> A year ago there was a patch posted that adds support for fsize and flastmod SSI >> commands: >> http://mailman.nginx.org/pipermail/nginx-devel/2020-February/013003.html >> What's needs to happen to get that into official sources? > > Firt of all, questions and concerns from the previous review need > answers. > Hi As far as I can see, the patches posted on February 2020 (linked above) are addressing your concerns expressed in May 2017 - perhaps with exception of your concern about handling multiple config commands? I'm not sure about that one. I did not find more questions and concerns expressed on patch proposed on February 2020. -- Sincerely Rastislav Stanik From mdounin at mdounin.ru Sun Apr 4 23:39:09 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 5 Apr 2021 02:39:09 +0300 Subject: fsize and flastmod SSI commands In-Reply-To: References: <6682f478-e82a-0436-4d7b-02b3f8407f30@rastos.org> Message-ID: Hello! On Sun, Apr 04, 2021 at 09:19:30PM +0200, Rastislav Stanik wrote: > On 04/04/2021 17.56, Maxim Dounin wrote: > > Hello! > > > > On Sun, Apr 04, 2021 at 09:52:03AM +0200, Rastislav Stanik wrote: > > > >> A year ago there was a patch posted that adds support for fsize and flastmod SSI > >> commands: > >> http://mailman.nginx.org/pipermail/nginx-devel/2020-February/013003.html > >> What's needs to happen to get that into official sources? > > > > Firt of all, questions and concerns from the previous review need > > answers. > > As far as I can see, the patches posted on February 2020 (linked above) are > addressing your concerns expressed in May 2017 - perhaps with exception of your > concern about handling multiple config commands? I'm not sure about that one. > I did not find more questions and concerns expressed on patch proposed on > February 2020. I don't see answers to questions asked in the review made in 2017 (actually, two reviews, both without any answers), and obviously not motivated to do additional reviews. Even if all concerns are addressed in the new patch. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Mon Apr 5 13:00:00 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 05 Apr 2021 13:00:00 +0000 Subject: [nginx] Version bump. Message-ID: details: https://hg.nginx.org/nginx/rev/19799b290812 branches: changeset: 7815:19799b290812 user: Maxim Dounin date: Mon Apr 05 04:03:10 2021 +0300 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r eb23d58bfd6b -r 19799b290812 src/core/nginx.h --- a/src/core/nginx.h Tue Mar 30 17:47:11 2021 +0300 +++ b/src/core/nginx.h Mon Apr 05 04:03:10 2021 +0300 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1019009 -#define NGINX_VERSION "1.19.9" +#define nginx_version 1019010 +#define NGINX_VERSION "1.19.10" #define NGINX_VER "nginx/" NGINX_VERSION #ifdef NGX_BUILD From mdounin at mdounin.ru Mon Apr 5 13:00:04 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 05 Apr 2021 13:00:04 +0000 Subject: [nginx] Gzip: support for zlib-ng. Message-ID: details: https://hg.nginx.org/nginx/rev/1f3d0d9f893f branches: changeset: 7816:1f3d0d9f893f user: Maxim Dounin date: Mon Apr 05 04:06:58 2021 +0300 description: Gzip: support for zlib-ng. diffstat: src/http/modules/ngx_http_gzip_filter_module.c | 23 +++++++++++++++++++++-- 1 files changed, 21 insertions(+), 2 deletions(-) diffs (65 lines): diff -r 19799b290812 -r 1f3d0d9f893f src/http/modules/ngx_http_gzip_filter_module.c --- a/src/http/modules/ngx_http_gzip_filter_module.c Mon Apr 05 04:03:10 2021 +0300 +++ b/src/http/modules/ngx_http_gzip_filter_module.c Mon Apr 05 04:06:58 2021 +0300 @@ -57,6 +57,7 @@ typedef struct { unsigned nomem:1; unsigned buffering:1; unsigned intel:1; + unsigned zlib_ng:1; size_t zin; size_t zout; @@ -214,6 +215,7 @@ static ngx_http_output_header_filter_pt static ngx_http_output_body_filter_pt ngx_http_next_body_filter; static ngx_uint_t ngx_http_gzip_assume_intel; +static ngx_uint_t ngx_http_gzip_assume_zlib_ng; static ngx_int_t @@ -506,7 +508,7 @@ ngx_http_gzip_filter_memory(ngx_http_req if (!ngx_http_gzip_assume_intel) { ctx->allocated = 8192 + (1 << (wbits + 2)) + (1 << (memlevel + 9)); - } else { + } else if (!ngx_http_gzip_assume_zlib_ng) { /* * A zlib variant from Intel, https://github.com/jtkukunas/zlib. * It can force window bits to 13 for fast compression level, @@ -523,6 +525,20 @@ ngx_http_gzip_filter_memory(ngx_http_req + (1 << (ngx_max(memlevel, 8) + 8)) + (1 << (memlevel + 8)); ctx->intel = 1; + + } else { + /* + * Another zlib variant, https://github.com/zlib-ng/zlib-ng. + * Similar to Intel's variant, though uses 128K hash. + */ + + if (conf->level == 1) { + wbits = ngx_max(wbits, 13); + } + + ctx->allocated = 8192 + 16 + (1 << (wbits + 2)) + + 131072 + (1 << (memlevel + 8)); + ctx->zlib_ng = 1; } } @@ -945,11 +961,14 @@ ngx_http_gzip_filter_alloc(void *opaque, return p; } - if (ctx->intel) { + if (ctx->zlib_ng) { ngx_log_error(NGX_LOG_ALERT, ctx->request->connection->log, 0, "gzip filter failed to use preallocated memory: " "%ud of %ui", items * size, ctx->allocated); + } else if (ctx->intel) { + ngx_http_gzip_assume_zlib_ng = 1; + } else { ngx_http_gzip_assume_intel = 1; } From mdounin at mdounin.ru Mon Apr 5 13:00:06 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 05 Apr 2021 13:00:06 +0000 Subject: [nginx] Gzip: updated handling of zlib variant from Intel. Message-ID: details: https://hg.nginx.org/nginx/rev/c297c2c252d8 branches: changeset: 7817:c297c2c252d8 user: Maxim Dounin date: Mon Apr 05 04:07:17 2021 +0300 description: Gzip: updated handling of zlib variant from Intel. In current versions (all versions based on zlib 1.2.11, at least since 2018) it no longer uses 64K hash and does not force window bits to 13 if it is less than 13. That is, it needs just 16 bytes more memory than normal zlib, so these bytes are simply added to the normal size calculation. diffstat: src/http/modules/ngx_http_gzip_filter_module.c | 35 ++++++------------------- 1 files changed, 9 insertions(+), 26 deletions(-) diffs (74 lines): diff -r 1f3d0d9f893f -r c297c2c252d8 src/http/modules/ngx_http_gzip_filter_module.c --- a/src/http/modules/ngx_http_gzip_filter_module.c Mon Apr 05 04:06:58 2021 +0300 +++ b/src/http/modules/ngx_http_gzip_filter_module.c Mon Apr 05 04:07:17 2021 +0300 @@ -56,7 +56,6 @@ typedef struct { unsigned done:1; unsigned nomem:1; unsigned buffering:1; - unsigned intel:1; unsigned zlib_ng:1; size_t zin; @@ -214,7 +213,6 @@ static ngx_str_t ngx_http_gzip_ratio = static ngx_http_output_header_filter_pt ngx_http_next_header_filter; static ngx_http_output_body_filter_pt ngx_http_next_body_filter; -static ngx_uint_t ngx_http_gzip_assume_intel; static ngx_uint_t ngx_http_gzip_assume_zlib_ng; @@ -503,33 +501,21 @@ ngx_http_gzip_filter_memory(ngx_http_req * 8K is for zlib deflate_state, it takes * *) 5816 bytes on i386 and sparc64 (32-bit mode) * *) 5920 bytes on amd64 and sparc64 + * + * A zlib variant from Intel (https://github.com/jtkukunas/zlib) + * uses additional 16-byte padding in one of window-sized buffers. */ - if (!ngx_http_gzip_assume_intel) { - ctx->allocated = 8192 + (1 << (wbits + 2)) + (1 << (memlevel + 9)); - - } else if (!ngx_http_gzip_assume_zlib_ng) { - /* - * A zlib variant from Intel, https://github.com/jtkukunas/zlib. - * It can force window bits to 13 for fast compression level, - * on processors with SSE 4.2 it uses 64K hash instead of scaling - * it from the specified memory level, and also introduces - * 16-byte padding in one out of the two window-sized buffers. - */ - - if (conf->level == 1) { - wbits = ngx_max(wbits, 13); - } - + if (!ngx_http_gzip_assume_zlib_ng) { ctx->allocated = 8192 + 16 + (1 << (wbits + 2)) - + (1 << (ngx_max(memlevel, 8) + 8)) - + (1 << (memlevel + 8)); - ctx->intel = 1; + + (1 << (memlevel + 9)); } else { /* * Another zlib variant, https://github.com/zlib-ng/zlib-ng. - * Similar to Intel's variant, though uses 128K hash. + * It forces window bits to 13 for fast compression level, + * uses 16-byte padding in one of window-sized buffers, and + * uses 128K hash. */ if (conf->level == 1) { @@ -966,11 +952,8 @@ ngx_http_gzip_filter_alloc(void *opaque, "gzip filter failed to use preallocated memory: " "%ud of %ui", items * size, ctx->allocated); - } else if (ctx->intel) { + } else { ngx_http_gzip_assume_zlib_ng = 1; - - } else { - ngx_http_gzip_assume_intel = 1; } p = ngx_palloc(ctx->request->pool, items * size); From mdounin at mdounin.ru Mon Apr 5 18:20:16 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 05 Apr 2021 18:20:16 +0000 Subject: [nginx] Configure: fixed --test-build-epoll on FreeBSD 13. Message-ID: details: https://hg.nginx.org/nginx/rev/e2e9e0fae747 branches: changeset: 7818:e2e9e0fae747 user: Maxim Dounin date: Mon Apr 05 20:14:16 2021 +0300 description: Configure: fixed --test-build-epoll on FreeBSD 13. In FreeBSD 13, eventfd(2) was added, and this breaks build with --test-build-epoll and without --with-file-aio. Fix is to move eventfd(2) detection to auto/os/linux, as it is used only on Linux as a notification mechanism for epoll(). diffstat: auto/os/linux | 25 +++++++++++++++++++++++++ auto/unix | 23 ----------------------- 2 files changed, 25 insertions(+), 23 deletions(-) diffs (68 lines): diff -r c297c2c252d8 -r e2e9e0fae747 auto/os/linux --- a/auto/os/linux Mon Apr 05 04:07:17 2021 +0300 +++ b/auto/os/linux Mon Apr 05 20:14:16 2021 +0300 @@ -86,6 +86,31 @@ if [ $ngx_found = yes ]; then ee.data.ptr = NULL; epoll_ctl(efd, EPOLL_CTL_ADD, fd, &ee)" . auto/feature + + + # eventfd() + + ngx_feature="eventfd()" + ngx_feature_name="NGX_HAVE_EVENTFD" + ngx_feature_run=no + ngx_feature_incs="#include " + ngx_feature_path= + ngx_feature_libs= + ngx_feature_test="(void) eventfd(0, 0)" + . auto/feature + + if [ $ngx_found = yes ]; then + have=NGX_HAVE_SYS_EVENTFD_H . auto/have + fi + + + if [ $ngx_found = no ]; then + + ngx_feature="eventfd() (SYS_eventfd)" + ngx_feature_incs="#include " + ngx_feature_test="(void) SYS_eventfd" + . auto/feature + fi fi diff -r c297c2c252d8 -r e2e9e0fae747 auto/unix --- a/auto/unix Mon Apr 05 04:07:17 2021 +0300 +++ b/auto/unix Mon Apr 05 20:14:16 2021 +0300 @@ -582,29 +582,6 @@ Currently file AIO is supported on FreeB END exit 1 fi - -else - - ngx_feature="eventfd()" - ngx_feature_name="NGX_HAVE_EVENTFD" - ngx_feature_run=no - ngx_feature_incs="#include " - ngx_feature_path= - ngx_feature_libs= - ngx_feature_test="(void) eventfd(0, 0)" - . auto/feature - - if [ $ngx_found = yes ]; then - have=NGX_HAVE_SYS_EVENTFD_H . auto/have - fi - - if [ $ngx_found = no ]; then - - ngx_feature="eventfd() (SYS_eventfd)" - ngx_feature_incs="#include " - ngx_feature_test="(void) SYS_eventfd" - . auto/feature - fi fi From jusmaki at gmail.com Wed Apr 7 17:51:38 2021 From: jusmaki at gmail.com (Jussi Maki) Date: Wed, 7 Apr 2021 20:51:38 +0300 Subject: PATCH Upstream: new "keepalive_max_connection_duration" directive Message-ID: # HG changeset patch # User Jussi Maki # Date 1617816597 -10800 # Wed Apr 07 20:29:57 2021 +0300 # Node ID 3699288ff20a3e51ee4b7689898ce0241f64f0f5 # Parent e2e9e0fae74734b28974c64daacc492d751b4781 Upstream: new "keepalive_max_connection_duration" directive Added a new keepalive_max_connection duration which provides the time in milliseconds for the upstream block on how long the connection should be kept connected. The current keepalive directives either define the idle time or the number of requests but there is no elapsed time-based parameter. The elapsed time-based connection parameter is useful in a case when there are multiple backends and the connection should be evenly load balanced to them and the response times for upstream requests vary. diff -r e2e9e0fae747 -r 3699288ff20a src/core/ngx_connection.h --- a/src/core/ngx_connection.h Mon Apr 05 20:14:16 2021 +0300 +++ b/src/core/ngx_connection.h Wed Apr 07 20:29:57 2021 +0300 @@ -191,6 +191,8 @@ #if (NGX_THREADS || NGX_COMPAT) ngx_thread_task_t *sendfile_task; #endif + + ngx_msec_t connection_started_time; }; diff -r e2e9e0fae747 -r 3699288ff20a src/http/modules/ngx_http_upstream_keepalive_module.c --- a/src/http/modules/ngx_http_upstream_keepalive_module.c Mon Apr 05 20:14:16 2021 +0300 +++ b/src/http/modules/ngx_http_upstream_keepalive_module.c Wed Apr 07 20:29:57 2021 +0300 @@ -14,6 +14,7 @@ ngx_uint_t max_cached; ngx_uint_t requests; ngx_msec_t timeout; + ngx_msec_t duration; ngx_queue_t cache; ngx_queue_t free; @@ -100,6 +101,13 @@ offsetof(ngx_http_upstream_keepalive_srv_conf_t, requests), NULL }, + { ngx_string("keepalive_max_connection_duration"), + NGX_HTTP_UPS_CONF|NGX_CONF_TAKE1, + ngx_conf_set_num_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_upstream_keepalive_srv_conf_t, duration), + NULL }, + ngx_null_command }; @@ -387,7 +395,13 @@ item->socklen = pc->socklen; ngx_memcpy(&item->sockaddr, pc->sockaddr, pc->socklen); - if (c->read->ready) { + if (kp->conf->duration != NGX_CONF_UNSET_MSEC && + ngx_current_msec - c->connection_started_time > kp->conf->duration) { + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, pc->log, 0, + "free keepalive peer: expired connection %p", c); + c->close = 1; + } + if (c->close || c->read->ready) { ngx_http_upstream_keepalive_close_handler(c->read); } @@ -515,6 +529,7 @@ conf->timeout = NGX_CONF_UNSET_MSEC; conf->requests = NGX_CONF_UNSET_UINT; + conf->duration = NGX_CONF_UNSET_MSEC; return conf; } diff -r e2e9e0fae747 -r 3699288ff20a src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c Mon Apr 05 20:14:16 2021 +0300 +++ b/src/http/ngx_http_upstream.c Wed Apr 07 20:29:57 2021 +0300 @@ -2017,6 +2017,10 @@ u->state->connect_time = ngx_current_msec - u->start_time; } + if (!c->connection_started_time) { + c->connection_started_time = ngx_current_msec; + } + if (!u->request_sent && ngx_http_upstream_test_connect(c) != NGX_OK) { ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_ERROR); return; -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Wed Apr 7 22:35:20 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 07 Apr 2021 22:35:20 +0000 Subject: [nginx] HTTP/2: relaxed PRIORITY frames limit. Message-ID: details: https://hg.nginx.org/nginx/rev/3674d5b7174e branches: changeset: 7819:3674d5b7174e user: Maxim Dounin date: Wed Apr 07 02:03:29 2021 +0300 description: HTTP/2: relaxed PRIORITY frames limit. Firefox uses several idle streams for PRIORITY frames[1], and "http2_max_concurrent_streams 1;" results in "client sent too many PRIORITY frames" errors when a connection is established by Firefox. Fix is to relax the PRIORITY frames limit to use at least 100 as the initial value (which is the recommended by the HTTP/2 protocol minimum limit on the number of concurrent streams, so it is not unreasonable for clients to assume that similar number of idle streams can be used for prioritization). [1] https://hg.mozilla.org/mozilla-central/file/32a9e6e145d6e3071c3993a20bb603a2f388722b/netwerk/protocol/http/Http2Stream.cpp#l1270 diffstat: src/http/v2/ngx_http_v2.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r e2e9e0fae747 -r 3674d5b7174e src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c Mon Apr 05 20:14:16 2021 +0300 +++ b/src/http/v2/ngx_http_v2.c Wed Apr 07 02:03:29 2021 +0300 @@ -277,7 +277,7 @@ ngx_http_v2_init(ngx_event_t *rev) h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module); h2c->concurrent_pushes = h2scf->concurrent_pushes; - h2c->priority_limit = h2scf->concurrent_streams; + h2c->priority_limit = ngx_max(h2scf->concurrent_streams, 100); h2c->pool = ngx_create_pool(h2scf->pool_size, h2c->connection->log); if (h2c->pool == NULL) { From mdounin at mdounin.ru Wed Apr 7 22:35:23 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 07 Apr 2021 22:35:23 +0000 Subject: [nginx] Introduced the "keepalive_time" directive. Message-ID: details: https://hg.nginx.org/nginx/rev/fdc3d40979b0 branches: changeset: 7820:fdc3d40979b0 user: Maxim Dounin date: Thu Apr 08 00:15:48 2021 +0300 description: Introduced the "keepalive_time" directive. Similar to lingering_time, it limits total connection lifetime before keepalive is switched off. The default is 1 hour, which is close to the total maximum connection lifetime possible with default keepalive_requests and keepalive_timeout. diffstat: src/core/ngx_connection.h | 1 + src/core/ngx_resolver.c | 4 ++++ src/event/ngx_event_accept.c | 2 ++ src/event/ngx_event_acceptex.c | 2 ++ src/event/ngx_event_connect.c | 2 ++ src/event/ngx_event_udp.c | 2 ++ src/http/modules/ngx_http_upstream_keepalive_module.c | 14 ++++++++++++++ src/http/ngx_http_core_module.c | 15 +++++++++++++++ src/http/ngx_http_core_module.h | 1 + src/http/v2/ngx_http_v2.c | 4 +++- 10 files changed, 46 insertions(+), 1 deletions(-) diffs (203 lines): diff -r 3674d5b7174e -r fdc3d40979b0 src/core/ngx_connection.h --- a/src/core/ngx_connection.h Wed Apr 07 02:03:29 2021 +0300 +++ b/src/core/ngx_connection.h Thu Apr 08 00:15:48 2021 +0300 @@ -162,6 +162,7 @@ struct ngx_connection_s { ngx_atomic_uint_t number; + ngx_msec_t start_time; ngx_uint_t requests; unsigned buffered:8; diff -r 3674d5b7174e -r fdc3d40979b0 src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Wed Apr 07 02:03:29 2021 +0300 +++ b/src/core/ngx_resolver.c Thu Apr 08 00:15:48 2021 +0300 @@ -4459,6 +4459,8 @@ ngx_udp_connect(ngx_resolver_connection_ c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); + c->start_time = ngx_current_msec; + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, &rec->log, 0, "connect to %V, fd:%d #%uA", &rec->server, s, c->number); @@ -4545,6 +4547,8 @@ ngx_tcp_connect(ngx_resolver_connection_ c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); + c->start_time = ngx_current_msec; + if (ngx_add_conn) { if (ngx_add_conn(c) == NGX_ERROR) { goto failed; diff -r 3674d5b7174e -r fdc3d40979b0 src/event/ngx_event_accept.c --- a/src/event/ngx_event_accept.c Wed Apr 07 02:03:29 2021 +0300 +++ b/src/event/ngx_event_accept.c Thu Apr 08 00:15:48 2021 +0300 @@ -256,6 +256,8 @@ ngx_event_accept(ngx_event_t *ev) c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); + c->start_time = ngx_current_msec; + #if (NGX_STAT_STUB) (void) ngx_atomic_fetch_add(ngx_stat_handled, 1); #endif diff -r 3674d5b7174e -r fdc3d40979b0 src/event/ngx_event_acceptex.c --- a/src/event/ngx_event_acceptex.c Wed Apr 07 02:03:29 2021 +0300 +++ b/src/event/ngx_event_acceptex.c Thu Apr 08 00:15:48 2021 +0300 @@ -80,6 +80,8 @@ ngx_event_acceptex(ngx_event_t *rev) c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); + c->start_time = ngx_current_msec; + ls->handler(c); return; diff -r 3674d5b7174e -r fdc3d40979b0 src/event/ngx_event_connect.c --- a/src/event/ngx_event_connect.c Wed Apr 07 02:03:29 2021 +0300 +++ b/src/event/ngx_event_connect.c Thu Apr 08 00:15:48 2021 +0300 @@ -193,6 +193,8 @@ ngx_event_connect_peer(ngx_peer_connecti c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); + c->start_time = ngx_current_msec; + if (ngx_add_conn) { if (ngx_add_conn(c) == NGX_ERROR) { goto failed; diff -r 3674d5b7174e -r fdc3d40979b0 src/event/ngx_event_udp.c --- a/src/event/ngx_event_udp.c Wed Apr 07 02:03:29 2021 +0300 +++ b/src/event/ngx_event_udp.c Thu Apr 08 00:15:48 2021 +0300 @@ -363,6 +363,8 @@ ngx_event_recvmsg(ngx_event_t *ev) c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); + c->start_time = ngx_current_msec; + #if (NGX_STAT_STUB) (void) ngx_atomic_fetch_add(ngx_stat_handled, 1); #endif diff -r 3674d5b7174e -r fdc3d40979b0 src/http/modules/ngx_http_upstream_keepalive_module.c --- a/src/http/modules/ngx_http_upstream_keepalive_module.c Wed Apr 07 02:03:29 2021 +0300 +++ b/src/http/modules/ngx_http_upstream_keepalive_module.c Thu Apr 08 00:15:48 2021 +0300 @@ -13,6 +13,7 @@ typedef struct { ngx_uint_t max_cached; ngx_uint_t requests; + ngx_msec_t time; ngx_msec_t timeout; ngx_queue_t cache; @@ -86,6 +87,13 @@ static ngx_command_t ngx_http_upstream_ 0, NULL }, + { ngx_string("keepalive_time"), + NGX_HTTP_UPS_CONF|NGX_CONF_TAKE1, + ngx_conf_set_msec_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_upstream_keepalive_srv_conf_t, time), + NULL }, + { ngx_string("keepalive_timeout"), NGX_HTTP_UPS_CONF|NGX_CONF_TAKE1, ngx_conf_set_msec_slot, @@ -149,6 +157,7 @@ ngx_http_upstream_init_keepalive(ngx_con kcf = ngx_http_conf_upstream_srv_conf(us, ngx_http_upstream_keepalive_module); + ngx_conf_init_msec_value(kcf->time, 3600000); ngx_conf_init_msec_value(kcf->timeout, 60000); ngx_conf_init_uint_value(kcf->requests, 100); @@ -326,6 +335,10 @@ ngx_http_upstream_free_keepalive_peer(ng goto invalid; } + if (ngx_current_msec - c->start_time > kp->conf->time) { + goto invalid; + } + if (!u->keepalive) { goto invalid; } @@ -513,6 +526,7 @@ ngx_http_upstream_keepalive_create_conf( * conf->max_cached = 0; */ + conf->time = NGX_CONF_UNSET_MSEC; conf->timeout = NGX_CONF_UNSET_MSEC; conf->requests = NGX_CONF_UNSET_UINT; diff -r 3674d5b7174e -r fdc3d40979b0 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Wed Apr 07 02:03:29 2021 +0300 +++ b/src/http/ngx_http_core_module.c Thu Apr 08 00:15:48 2021 +0300 @@ -495,6 +495,13 @@ static ngx_command_t ngx_http_core_comm offsetof(ngx_http_core_loc_conf_t, limit_rate_after), NULL }, + { ngx_string("keepalive_time"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_msec_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_core_loc_conf_t, keepalive_time), + NULL }, + { ngx_string("keepalive_timeout"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE12, ngx_http_core_keepalive, @@ -1335,6 +1342,11 @@ ngx_http_update_location_config(ngx_http } else if (r->connection->requests >= clcf->keepalive_requests) { r->keepalive = 0; + } else if (ngx_current_msec - r->connection->start_time + > clcf->keepalive_time) + { + r->keepalive = 0; + } else if (r->headers_in.msie6 && r->method == NGX_HTTP_POST && (clcf->keepalive_disable @@ -3500,6 +3512,7 @@ ngx_http_core_create_loc_conf(ngx_conf_t clcf->send_timeout = NGX_CONF_UNSET_MSEC; clcf->send_lowat = NGX_CONF_UNSET_SIZE; clcf->postpone_output = NGX_CONF_UNSET_SIZE; + clcf->keepalive_time = NGX_CONF_UNSET_MSEC; clcf->keepalive_timeout = NGX_CONF_UNSET_MSEC; clcf->keepalive_header = NGX_CONF_UNSET; clcf->keepalive_requests = NGX_CONF_UNSET_UINT; @@ -3738,6 +3751,8 @@ ngx_http_core_merge_loc_conf(ngx_conf_t conf->limit_rate_after = prev->limit_rate_after; } + ngx_conf_merge_msec_value(conf->keepalive_time, + prev->keepalive_time, 3600000); ngx_conf_merge_msec_value(conf->keepalive_timeout, prev->keepalive_timeout, 75000); ngx_conf_merge_sec_value(conf->keepalive_header, diff -r 3674d5b7174e -r fdc3d40979b0 src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h Wed Apr 07 02:03:29 2021 +0300 +++ b/src/http/ngx_http_core_module.h Thu Apr 08 00:15:48 2021 +0300 @@ -359,6 +359,7 @@ struct ngx_http_core_loc_conf_s { ngx_msec_t client_body_timeout; /* client_body_timeout */ ngx_msec_t send_timeout; /* send_timeout */ + ngx_msec_t keepalive_time; /* keepalive_time */ ngx_msec_t keepalive_timeout; /* keepalive_timeout */ ngx_msec_t lingering_time; /* lingering_time */ ngx_msec_t lingering_timeout; /* lingering_timeout */ diff -r 3674d5b7174e -r fdc3d40979b0 src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c Wed Apr 07 02:03:29 2021 +0300 +++ b/src/http/v2/ngx_http_v2.c Thu Apr 08 00:15:48 2021 +0300 @@ -1369,7 +1369,9 @@ ngx_http_v2_state_headers(ngx_http_v2_co ngx_http_core_module); if (clcf->keepalive_timeout == 0 - || h2c->connection->requests >= clcf->keepalive_requests) + || h2c->connection->requests >= clcf->keepalive_requests + || ngx_current_msec - h2c->connection->start_time + > clcf->keepalive_time) { h2c->goaway = 1; From mdounin at mdounin.ru Wed Apr 7 22:35:26 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 07 Apr 2021 22:35:26 +0000 Subject: [nginx] Added $connection_time variable. Message-ID: details: https://hg.nginx.org/nginx/rev/6d4f7d5e279f branches: changeset: 7821:6d4f7d5e279f user: Maxim Dounin date: Thu Apr 08 00:16:17 2021 +0300 description: Added $connection_time variable. diffstat: src/http/ngx_http_variables.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-) diffs (54 lines): diff -r fdc3d40979b0 -r 6d4f7d5e279f src/http/ngx_http_variables.c --- a/src/http/ngx_http_variables.c Thu Apr 08 00:15:48 2021 +0300 +++ b/src/http/ngx_http_variables.c Thu Apr 08 00:16:17 2021 +0300 @@ -129,6 +129,8 @@ static ngx_int_t ngx_http_variable_conne ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_connection_requests(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); +static ngx_int_t ngx_http_variable_connection_time(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_nginx_version(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); @@ -342,6 +344,9 @@ static ngx_http_variable_t ngx_http_cor { ngx_string("connection_requests"), NULL, ngx_http_variable_connection_requests, 0, 0, 0 }, + { ngx_string("connection_time"), NULL, ngx_http_variable_connection_time, + 0, NGX_HTTP_VAR_NOCACHEABLE, 0 }, + { ngx_string("nginx_version"), NULL, ngx_http_variable_nginx_version, 0, 0, 0 }, @@ -2253,6 +2258,31 @@ ngx_http_variable_connection_requests(ng static ngx_int_t +ngx_http_variable_connection_time(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + u_char *p; + ngx_msec_int_t ms; + + p = ngx_pnalloc(r->pool, NGX_TIME_T_LEN + 4); + if (p == NULL) { + return NGX_ERROR; + } + + ms = ngx_current_msec - r->connection->start_time; + ms = ngx_max(ms, 0); + + v->len = ngx_sprintf(p, "%T.%03M", (time_t) ms / 1000, ms % 1000) - p; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + v->data = p; + + return NGX_OK; +} + + +static ngx_int_t ngx_http_variable_nginx_version(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) { From mdounin at mdounin.ru Wed Apr 7 22:35:29 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 07 Apr 2021 22:35:29 +0000 Subject: [nginx] Changed keepalive_requests default to 1000 (ticket #2155). Message-ID: details: https://hg.nginx.org/nginx/rev/82e174e47663 branches: changeset: 7822:82e174e47663 user: Maxim Dounin date: Thu Apr 08 00:16:30 2021 +0300 description: Changed keepalive_requests default to 1000 (ticket #2155). It turns out no browsers implement HTTP/2 GOAWAY handling properly, and large enough number of resources on a page results in failures to load some resources. In particular, Chrome seems to experience errors if loading of all resources requires more than 1 connection (while it is usually able to retry requests at least once, even with 2 connections there are occasional failures for some reason), Safari if loading requires more than 3 connections, and Firefox if loading requires more than 10 connections (can be configured with network.http.request.max-attempts, defaults to 10). It does not seem to be possible to resolve this on nginx side, even strict limiting of maximum concurrency does not help, and loading issues seems to be triggered by merely queueing of a request for a particular connection. The only available mitigation seems to use higher keepalive_requests value. The new default is 1000 and matches previously used default for http2_max_requests. It is expected to be enough for 99.98% of the pages (https://httparchive.org/reports/state-of-the-web?start=latest#reqTotal) even in Chrome. diffstat: src/http/modules/ngx_http_upstream_keepalive_module.c | 2 +- src/http/ngx_http_core_module.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diffs (24 lines): diff -r 6d4f7d5e279f -r 82e174e47663 src/http/modules/ngx_http_upstream_keepalive_module.c --- a/src/http/modules/ngx_http_upstream_keepalive_module.c Thu Apr 08 00:16:17 2021 +0300 +++ b/src/http/modules/ngx_http_upstream_keepalive_module.c Thu Apr 08 00:16:30 2021 +0300 @@ -159,7 +159,7 @@ ngx_http_upstream_init_keepalive(ngx_con ngx_conf_init_msec_value(kcf->time, 3600000); ngx_conf_init_msec_value(kcf->timeout, 60000); - ngx_conf_init_uint_value(kcf->requests, 100); + ngx_conf_init_uint_value(kcf->requests, 1000); if (kcf->original_init_upstream(cf, us) != NGX_OK) { return NGX_ERROR; diff -r 6d4f7d5e279f -r 82e174e47663 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Thu Apr 08 00:16:17 2021 +0300 +++ b/src/http/ngx_http_core_module.c Thu Apr 08 00:16:30 2021 +0300 @@ -3758,7 +3758,7 @@ ngx_http_core_merge_loc_conf(ngx_conf_t ngx_conf_merge_sec_value(conf->keepalive_header, prev->keepalive_header, 0); ngx_conf_merge_uint_value(conf->keepalive_requests, - prev->keepalive_requests, 100); + prev->keepalive_requests, 1000); ngx_conf_merge_uint_value(conf->lingering_close, prev->lingering_close, NGX_HTTP_LINGERING_ON); ngx_conf_merge_msec_value(conf->lingering_time, From mdounin at mdounin.ru Wed Apr 7 23:07:31 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 8 Apr 2021 02:07:31 +0300 Subject: PATCH Upstream: new "keepalive_max_connection_duration" directive In-Reply-To: References: Message-ID: Hello! On Wed, Apr 07, 2021 at 08:51:38PM +0300, Jussi Maki wrote: > # HG changeset patch > # User Jussi Maki > # Date 1617816597 -10800 > # Wed Apr 07 20:29:57 2021 +0300 > # Node ID 3699288ff20a3e51ee4b7689898ce0241f64f0f5 > # Parent e2e9e0fae74734b28974c64daacc492d751b4781 > Upstream: new "keepalive_max_connection_duration" directive > > Added a new keepalive_max_connection duration which provides > the time in milliseconds for the upstream block on how long > the connection should be kept connected. The current keepalive > directives either define the idle time or the number of requests > but there is no elapsed time-based parameter. > > The elapsed time-based connection parameter is useful in a case > when there are multiple backends and the connection should be > evenly load balanced to them and the response times for upstream > requests vary. Thanks for the patch. I've just committed a patch series which adds the "keepalive_time" directive[1] both for keepalive connections with clients and with upstream servers, as a part of a mitigation for browser issues identified in ticket #2155[2]. As far as I can see, it is basically identical to what you are trying to introduce for uptream servers, and should work for your use case as well. [1] https://hg.nginx.org/nginx/rev/fdc3d40979b0 [2] https://trac.nginx.org/nginx/ticket/2155 -- Maxim Dounin http://mdounin.ru/ From piotrsikora at google.com Fri Apr 9 02:56:07 2021 From: piotrsikora at google.com (Piotr Sikora) Date: Thu, 8 Apr 2021 19:56:07 -0700 Subject: [nginx] Changed keepalive_requests default to 1000 (ticket #2155). In-Reply-To: References: Message-ID: Hi Maxim, > It turns out no browsers implement HTTP/2 GOAWAY handling properly, and > large enough number of resources on a page results in failures to load > some resources. In particular, Chrome seems to experience errors if > loading of all resources requires more than 1 connection (while it > is usually able to retry requests at least once, even with 2 connections > there are occasional failures for some reason), Safari if loading requires > more than 3 connections, and Firefox if loading requires more than 10 > connections (can be configured with network.http.request.max-attempts, > defaults to 10). > > It does not seem to be possible to resolve this on nginx side, even strict > limiting of maximum concurrency does not help, and loading issues seems to > be triggered by merely queueing of a request for a particular connection. > The only available mitigation seems to use higher keepalive_requests value. Instead of blaming browsers, did you consider implementing graceful shutdown using 2-stage GOAWAY? The process is clearly described in RFC7540, sec. 6.8: [...] A server that is attempting to gracefully shut down a connection SHOULD send an initial GOAWAY frame with the last stream identifier set to 2^31-1 and a NO_ERROR code. This signals to the client that a shutdown is imminent and that initiating further requests is prohibited. After allowing time for any in-flight stream creation (at least one round-trip time), the server can send another GOAWAY frame with an updated last stream identifier. This ensures that a connection can be cleanly shut down without losing requests. This is a solved problem, and the solution was pointed out years ago: http://mailman.nginx.org/pipermail/nginx-devel/2017-August/010439.html http://mailman.nginx.org/pipermail/nginx-devel/2018-March/010930.html Best regards, Piotr Sikora From vasiliy.soshnikov at gmail.com Fri Apr 9 13:26:52 2021 From: vasiliy.soshnikov at gmail.com (Vasiliy Soshnikov) Date: Fri, 9 Apr 2021 16:26:52 +0300 Subject: [PATCH] Support of proxy v2 protocol for NGINX stream module Message-ID: diff -r 82e174e47663 src/core/ngx_proxy_protocol.c --- a/src/core/ngx_proxy_protocol.c Thu Apr 08 00:16:30 2021 +0300 +++ b/src/core/ngx_proxy_protocol.c Fri Apr 09 16:10:29 2021 +0300 @@ -13,6 +13,34 @@ #define NGX_PROXY_PROTOCOL_AF_INET6 2 +#define NGX_PROXY_PROTOCOL_V2_SIG "\x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A" +#define NGX_PROXY_PROTOCOL_V2_SIG_LEN 12 +#define NGX_PROXY_PROTOCOL_V2_HDR_LEN 16 +#define NGX_PROXY_PROTOCOL_V2_HDR_LEN_INET \ + (NGX_PROXY_PROTOCOL_V2_HDR_LEN + (4 + 4 + 2 + 2)) +#define NGX_PROXY_PROTOCOL_V2_HDR_LEN_INET6 \ + (NGX_PROXY_PROTOCOL_V2_HDR_LEN + (16 + 16 + 2 + 2)) + +#define NGX_PROXY_PROTOCOL_V2_CMD_PROXY (0x20 | 0x01) + +#define NGX_PROXY_PROTOCOL_V2_TRANS_STREAM 0x01 + +#define NGX_PROXY_PROTOCOL_V2_FAM_UNSPEC 0x00 +#define NGX_PROXY_PROTOCOL_V2_FAM_INET 0x10 +#define NGX_PROXY_PROTOCOL_V2_FAM_INET6 0x20 + +#define NGX_PROXY_PROTOCOL_V2_TYPE_ALPN 0x01 +#define NGX_PROXY_PROTOCOL_V2_TYPE_SSL 0x20 +#define NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_VERSION 0x21 +#define NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_CIPHER 0x23 +#define NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_SIG_ALG 0x24 +#define NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_KEY_ALG 0x25 + +#define NGX_PROXY_PROTOCOL_V2_CLIENT_SSL 0x01 +#define NGX_PROXY_PROTOCOL_V2_CLIENT_CERT_CONN 0x02 +#define NGX_PROXY_PROTOCOL_V2_CLIENT_CERT_SESS 0x04 + + #define ngx_proxy_protocol_parse_uint16(p) ((p)[0] << 8 | (p)[1]) @@ -40,12 +68,68 @@ } ngx_proxy_protocol_inet6_addrs_t; +typedef union { + struct { + uint32_t src_addr; + uint32_t dst_addr; + uint16_t src_port; + uint16_t dst_port; + } ip4; + struct { + uint8_t src_addr[16]; + uint8_t dst_addr[16]; + uint16_t src_port; + uint16_t dst_port; + } ip6; +} ngx_proxy_protocol_addrs_t; + + +typedef struct { + u_char signature[12]; + uint8_t version_command; + uint8_t family_transport; + uint16_t len; + ngx_proxy_protocol_addrs_t addr; +} ngx_proxy_protocol_v2_header_t; + + +struct ngx_tlv_s { + uint8_t type; + uint8_t length_hi; + uint8_t length_lo; + uint8_t value[0]; +} __attribute__((packed)); + +typedef struct ngx_tlv_s ngx_tlv_t; + + +#if (NGX_STREAM_SSL) +struct ngx_tlv_ssl_s { + ngx_tlv_t tlv; + uint8_t client; + uint32_t verify; + uint8_t sub_tlv[]; +} __attribute__((packed)); + +typedef struct ngx_tlv_ssl_s ngx_tlv_ssl_t; +#endif + + static u_char *ngx_proxy_protocol_read_addr(ngx_connection_t *c, u_char *p, u_char *last, ngx_str_t *addr); static u_char *ngx_proxy_protocol_read_port(u_char *p, u_char *last, in_port_t *port, u_char sep); static u_char *ngx_proxy_protocol_v2_read(ngx_connection_t *c, u_char *buf, u_char *last); +static u_char *ngx_proxy_protocol_v2_write(ngx_connection_t *c, u_char *buf, + u_char *last); +#if (NGX_HAVE_INET6) +static void ngx_v4tov6(struct in6_addr *sin6_addr, struct sockaddr *addr); +#endif +#if (NGX_STREAM_SSL) +static u_char *ngx_copy_tlv(u_char *pos, u_char *last, u_char type, + u_char *value, uint16_t value_len); +#endif u_char * @@ -223,7 +307,8 @@ u_char * -ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last) +ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, u_char *last, + ngx_uint_t pp_version) { ngx_uint_t port, lport; @@ -235,6 +320,10 @@ return NULL; } + if (pp_version == 2) { + return ngx_proxy_protocol_v2_write(c, buf, last); + } + switch (c->sockaddr->sa_family) { case AF_INET: @@ -420,3 +509,344 @@ return end; } + + +static u_char * +ngx_proxy_protocol_v2_write(ngx_connection_t *c, u_char *buf, u_char *last) +{ + struct sockaddr *src, *dst; + ngx_proxy_protocol_v2_header_t *header; +#if (NGX_HAVE_INET6) + struct in6_addr v6_tmp; + ngx_int_t v6_used; +#endif +#if (NGX_STREAM_SSL) + ngx_tlv_ssl_t *tlv; + u_char *value, *pos; + u_char kbuf[100]; + const unsigned char *data; + unsigned int data_len; + + X509 *crt; + EVP_PKEY *key; + const ASN1_OBJECT *algorithm; + const char *s; + + long rc; + size_t tlv_len; +#endif + size_t len; + + header = (ngx_proxy_protocol_v2_header_t *) buf; + + header->len = 0; + + src = c->sockaddr; + dst = c->local_sockaddr; + + len = 0; + +#if (NGX_HAVE_INET6) + v6_used = 0; +#endif + + ngx_memcpy(header->signature, NGX_PROXY_PROTOCOL_V2_SIG, + NGX_PROXY_PROTOCOL_V2_SIG_LEN); + + header->version_command = NGX_PROXY_PROTOCOL_V2_CMD_PROXY; + header->family_transport = NGX_PROXY_PROTOCOL_V2_TRANS_STREAM; + + /** Addrs */ + + switch (src->sa_family) { + + case AF_INET: + + if (dst->sa_family == AF_INET) { + + header->addr.ip4.src_addr = + ((struct sockaddr_in *) src)->sin_addr.s_addr; + header->addr.ip4.src_port = ((struct sockaddr_in *) src)->sin_port; + } +#if (NGX_HAVE_INET6) + else /** dst == AF_INET6 */{ + + ngx_v4tov6(&v6_tmp, src); + ngx_memcpy(header->addr.ip6.src_addr, &v6_tmp, 16); + header->addr.ip6.src_port = ((struct sockaddr_in *) src)->sin_port; + } +#endif + break; + +#if (NGX_HAVE_INET6) + case AF_INET6: + v6_used = 1; + + ngx_memcpy(header->addr.ip6.src_addr, + &((struct sockaddr_in6 *) src)->sin6_addr, 16); + header->addr.ip6.src_port = ((struct sockaddr_in6 *) src)->sin6_port; + + break; +#endif + + default: + ngx_log_debug1(NGX_LOG_DEBUG_CORE, c->log, 0, + "PROXY protocol v2 unsupported src address family %ui", + src->sa_family); + goto unspec; + }; + + switch (dst->sa_family) { + case AF_INET: + + if (src->sa_family == AF_INET) { + + header->addr.ip4.dst_addr = + ((struct sockaddr_in *) dst)->sin_addr.s_addr; + header->addr.ip4.dst_port = ((struct sockaddr_in *) dst)->sin_port; + } +#if (NGX_HAVE_INET6) + else /** src == AF_INET6 */{ + + ngx_v4tov6(&v6_tmp, dst); + ngx_memcpy(header->addr.ip6.dst_addr, &v6_tmp, 16); + header->addr.ip6.dst_port = ((struct sockaddr_in *) dst)->sin_port; + + } +#endif + break; + +#if (NGX_HAVE_INET6) + case AF_INET6: + v6_used = 1; + + ngx_memcpy(header->addr.ip6.dst_addr, + &((struct sockaddr_in6 *) dst)->sin6_addr, 16); + header->addr.ip6.dst_port = ((struct sockaddr_in6 *) dst)->sin6_port; + + break; +#endif + + default: + ngx_log_debug1(NGX_LOG_DEBUG_CORE, c->log, 0, + "PROXY protocol v2 unsupported dest address family %ui", + dst->sa_family); + goto unspec; + } + +#if (NGX_HAVE_INET6) + if (!v6_used) { + header->family_transport |= NGX_PROXY_PROTOCOL_V2_FAM_INET; + len = NGX_PROXY_PROTOCOL_V2_HDR_LEN_INET; + + } else { + header->family_transport |= NGX_PROXY_PROTOCOL_V2_FAM_INET6; + len = NGX_PROXY_PROTOCOL_V2_HDR_LEN_INET6; + + } +#else + header->family_transport |= NGX_PROXY_PROTOCOL_V2_FAM_INET; + len = NGX_PROXY_PROTOCOL_V2_HDR_LEN_INET; +#endif + + /** SSL TLVs */ + +#if (NGX_STREAM_SSL) + + data = NULL; + data_len = 0; + + tlv = (ngx_tlv_ssl_t *) (buf + len); + ngx_memzero(tlv, sizeof(ngx_tlv_ssl_t)); + + tlv->tlv.type = NGX_PROXY_PROTOCOL_V2_TYPE_SSL; + pos = buf + len + sizeof(ngx_tlv_ssl_t); + + tlv->client |= NGX_PROXY_PROTOCOL_V2_CLIENT_SSL; + + if (c->ssl != NULL) { + +#ifdef TLSEXT_TYPE_application_layer_protocol_negotiation + SSL_get0_alpn_selected(c->ssl->connection, &data, &data_len); + +#ifdef TLSEXT_TYPE_next_proto_neg + if (data_len == 0) { + SSL_get0_next_proto_negotiated(c->ssl->connection, + &data, &data_len); + } +#endif + +#else /* TLSEXT_TYPE_next_proto_neg */ + SSL_get0_next_proto_negotiated(c->ssl->connection, &data, &data_len); +#endif + + if (data_len) { + + pos = ngx_copy_tlv(pos, last, + NGX_PROXY_PROTOCOL_V2_TYPE_ALPN, + (u_char *) data, (uint16_t) data_len); + if (pos == NULL) { + return NULL; + } + } + + value = (u_char *) SSL_get_version(c->ssl->connection); + if (value != NULL) { + + pos = ngx_copy_tlv(pos, last, + NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_VERSION, + value, ngx_strlen(value)); + if (pos == NULL) { + return NULL; + } + } + + crt = SSL_get_peer_certificate(c->ssl->connection); + if (crt != NULL) { + + tlv->client |= NGX_PROXY_PROTOCOL_V2_CLIENT_CERT_SESS; + + rc = SSL_get_verify_result(c->ssl->connection); + tlv->verify = htonl(rc); + + if (rc == X509_V_OK) { + + if (ngx_ssl_ocsp_get_status(c, &s) == NGX_OK) { + tlv->client |= NGX_PROXY_PROTOCOL_V2_CLIENT_CERT_CONN; + } + } + + X509_free(crt); + } + + crt = SSL_get_certificate(c->ssl->connection); + if (crt != NULL) { + + key = X509_get_pubkey(crt); + + /** Key */ + if (key != NULL) { + + switch (EVP_PKEY_base_id(key)) { + case EVP_PKEY_RSA: + value = (u_char *) "RSA"; + break; + case EVP_PKEY_EC: + value = (u_char *) "EC"; + break; + case EVP_PKEY_DSA: + value = (u_char *) "DSA"; + break; + default: + value = NULL; + break; + } + + if (value != NULL) { + + value = ngx_snprintf(kbuf, sizeof(kbuf) - 1, "%s%d%Z", + value, EVP_PKEY_bits(key)); + + pos = ngx_copy_tlv(pos, last, + NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_KEY_ALG, + kbuf, ngx_strlen(kbuf)); + } + + EVP_PKEY_free(key); + + if (pos == NULL) { + return NULL; + } + } + + /* ALG */ + X509_ALGOR_get0(&algorithm, NULL, NULL, X509_get0_tbs_sigalg(crt)); + value = (u_char *) OBJ_nid2sn(OBJ_obj2nid(algorithm)); + + if (value != NULL) { + + pos = ngx_copy_tlv(pos, last, + NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_SIG_ALG, + value, ngx_strlen(value)); + if (pos == NULL) { + return NULL; + } + } + } + + value = (u_char *) SSL_get_cipher_name(c->ssl->connection); + if (value != NULL) { + + pos = ngx_copy_tlv(pos, last, + NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_CIPHER, + value, ngx_strlen(value)); + if (pos == NULL) { + return NULL; + } + } + } + + tlv_len = pos - (buf + len); + + tlv->tlv.length_hi = (uint16_t) (tlv_len - sizeof(ngx_tlv_t)) >> 8; + tlv->tlv.length_lo = (uint16_t) (tlv_len - sizeof(ngx_tlv_t)) & 0x00ff; + + len = len + tlv_len; + +#endif + + header->len = htons(len - NGX_PROXY_PROTOCOL_V2_HDR_LEN); + return buf + len; + +unspec: + header->family_transport |= NGX_PROXY_PROTOCOL_V2_FAM_UNSPEC; + header->len = 0; + + return buf + NGX_PROXY_PROTOCOL_V2_HDR_LEN; +} + + +#if (NGX_HAVE_INET6) +static void +ngx_v4tov6(struct in6_addr *sin6_addr, struct sockaddr *addr) +{ + static const char rfc4291[] = { 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0xFF, 0xFF }; + + struct in_addr tmp_addr, *sin_addr; + + sin_addr = &((struct sockaddr_in *) addr)->sin_addr; + + tmp_addr.s_addr = sin_addr->s_addr; + ngx_memcpy(sin6_addr->s6_addr, rfc4291, sizeof(rfc4291)); + ngx_memcpy(sin6_addr->s6_addr + 12, &tmp_addr.s_addr, 4); +} +#endif + + +#if (NGX_STREAM_SSL) + +static u_char * +ngx_copy_tlv(u_char *pos, u_char *last, u_char type, + u_char *value, uint16_t value_len) +{ + ngx_tlv_t *tlv; + + if (last - pos < (long) sizeof(*tlv)) { + return NULL; + } + + tlv = (ngx_tlv_t *) pos; + + tlv->type = type; + tlv->length_hi = (uint16_t) value_len >> 8; + tlv->length_lo = (uint16_t) value_len & 0x00ff; + ngx_memcpy(tlv->value, value, value_len); + + return pos + (value_len + sizeof(*tlv)); +} + +#endif + + diff -r 82e174e47663 src/core/ngx_proxy_protocol.h --- a/src/core/ngx_proxy_protocol.h Thu Apr 08 00:16:30 2021 +0300 +++ b/src/core/ngx_proxy_protocol.h Fri Apr 09 16:10:29 2021 +0300 @@ -13,7 +13,7 @@ #include -#define NGX_PROXY_PROTOCOL_MAX_HEADER 107 +#define NGX_PROXY_PROTOCOL_MAX_HEADER 214 struct ngx_proxy_protocol_s { @@ -27,7 +27,7 @@ u_char *ngx_proxy_protocol_read(ngx_connection_t *c, u_char *buf, u_char *last); u_char *ngx_proxy_protocol_write(ngx_connection_t *c, u_char *buf, - u_char *last); + u_char *last, ngx_uint_t pp_version); #endif /* _NGX_PROXY_PROTOCOL_H_INCLUDED_ */ diff -r 82e174e47663 src/stream/ngx_stream_proxy_module.c --- a/src/stream/ngx_stream_proxy_module.c Thu Apr 08 00:16:30 2021 +0300 +++ b/src/stream/ngx_stream_proxy_module.c Fri Apr 09 16:10:29 2021 +0300 @@ -30,7 +30,7 @@ ngx_uint_t responses; ngx_uint_t next_upstream_tries; ngx_flag_t next_upstream; - ngx_flag_t proxy_protocol; + ngx_uint_t proxy_protocol; ngx_stream_upstream_local_t *local; ngx_flag_t socket_keepalive; @@ -121,6 +121,14 @@ #endif +static ngx_conf_enum_t ngx_stream_proxy_protocol[] = { + { ngx_string("off"), 0 }, + { ngx_string("on"), 1 }, + { ngx_string("v2"), 2 }, + { ngx_null_string, 0 } +}; + + static ngx_conf_deprecated_t ngx_conf_deprecated_proxy_downstream_buffer = { ngx_conf_deprecated, "proxy_downstream_buffer", "proxy_buffer_size" }; @@ -239,10 +247,10 @@ { ngx_string("proxy_protocol"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_FLAG, - ngx_conf_set_flag_slot, + ngx_conf_set_enum_slot, NGX_STREAM_SRV_CONF_OFFSET, offsetof(ngx_stream_proxy_srv_conf_t, proxy_protocol), - NULL }, + &ngx_stream_proxy_protocol }, #if (NGX_STREAM_SSL) @@ -891,7 +899,8 @@ cl->buf->pos = p; - p = ngx_proxy_protocol_write(c, p, p + NGX_PROXY_PROTOCOL_MAX_HEADER); + p = ngx_proxy_protocol_write(c, p, p + NGX_PROXY_PROTOCOL_MAX_HEADER, + u->proxy_protocol); if (p == NULL) { ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR); return; @@ -942,14 +951,15 @@ ngx_log_debug0(NGX_LOG_DEBUG_STREAM, c->log, 0, "stream proxy send PROXY protocol header"); - p = ngx_proxy_protocol_write(c, buf, buf + NGX_PROXY_PROTOCOL_MAX_HEADER); + u = s->upstream; + + p = ngx_proxy_protocol_write(c, buf, buf + NGX_PROXY_PROTOCOL_MAX_HEADER, + u->proxy_protocol); if (p == NULL) { ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR); return NGX_ERROR; } - u = s->upstream; - pc = u->peer.connection; size = p - buf; @@ -1998,7 +2008,7 @@ conf->responses = NGX_CONF_UNSET_UINT; conf->next_upstream_tries = NGX_CONF_UNSET_UINT; conf->next_upstream = NGX_CONF_UNSET; - conf->proxy_protocol = NGX_CONF_UNSET; + conf->proxy_protocol = NGX_CONF_UNSET_UINT; conf->local = NGX_CONF_UNSET_PTR; conf->socket_keepalive = NGX_CONF_UNSET; @@ -2053,7 +2063,7 @@ ngx_conf_merge_value(conf->next_upstream, prev->next_upstream, 1); - ngx_conf_merge_value(conf->proxy_protocol, prev->proxy_protocol, 0); + ngx_conf_merge_uint_value(conf->proxy_protocol, prev->proxy_protocol, 0); ngx_conf_merge_ptr_value(conf->local, prev->local, NULL); diff -r 82e174e47663 src/stream/ngx_stream_upstream.h --- a/src/stream/ngx_stream_upstream.h Thu Apr 08 00:16:30 2021 +0300 +++ b/src/stream/ngx_stream_upstream.h Fri Apr 09 16:10:29 2021 +0300 @@ -141,7 +141,7 @@ ngx_stream_upstream_resolved_t *resolved; ngx_stream_upstream_state_t *state; unsigned connected:1; - unsigned proxy_protocol:1; + unsigned proxy_protocol:2; } ngx_stream_upstream_t; -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Fri Apr 9 13:49:57 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 Apr 2021 16:49:57 +0300 Subject: [nginx] Changed keepalive_requests default to 1000 (ticket #2155). In-Reply-To: References: Message-ID: Hello! On Thu, Apr 08, 2021 at 07:56:07PM -0700, Piotr Sikora wrote: > Hi Maxim, > > > It turns out no browsers implement HTTP/2 GOAWAY handling properly, and > > large enough number of resources on a page results in failures to load > > some resources. In particular, Chrome seems to experience errors if > > loading of all resources requires more than 1 connection (while it > > is usually able to retry requests at least once, even with 2 connections > > there are occasional failures for some reason), Safari if loading requires > > more than 3 connections, and Firefox if loading requires more than 10 > > connections (can be configured with network.http.request.max-attempts, > > defaults to 10). > > > > It does not seem to be possible to resolve this on nginx side, even strict > > limiting of maximum concurrency does not help, and loading issues seems to > > be triggered by merely queueing of a request for a particular connection. > > The only available mitigation seems to use higher keepalive_requests value. > > Instead of blaming browsers, did you consider implementing graceful shutdown > using 2-stage GOAWAY? The process is clearly described in RFC7540, sec. 6.8: > > [...] A server that is attempting to gracefully shut down a > connection SHOULD send an initial GOAWAY frame with the last stream > identifier set to 2^31-1 and a NO_ERROR code. This signals to the > client that a shutdown is imminent and that initiating further > requests is prohibited. After allowing time for any in-flight stream > creation (at least one round-trip time), the server can send another > GOAWAY frame with an updated last stream identifier. This ensures > that a connection can be cleanly shut down without losing requests. > > This is a solved problem, and the solution was pointed out years ago: > http://mailman.nginx.org/pipermail/nginx-devel/2017-August/010439.html > http://mailman.nginx.org/pipermail/nginx-devel/2018-March/010930.html As you can see from the commit log, as well as the details in the ticket, even limiting concurrency does not help, and this means that two-stage GOAWAY would be useless: its only benefit is to make it possible to process in-flight requests without rejecting them. Not to mention that all requests in tests can be easily retried by browsers (and many are actually retried, but not all). Nevertheless, I've tried implementing two-stage GOAWAY as well while working on this, just a quick hack to see if it helps. As expected from the above, it doesn't help. Further, it triggers the bug in Chrome (https://crbug.com/1030255), which basically stops any communication with the server after the first GOAWAY, and does nothing till the connection is closed by the server. That is, a simple approach of sending GOAWAY with 2^31-1 and waiting for keepalive_timeout to expire and then sending the real GOAWAY (or waiting for the client to close the connection) clearly does more harm than good. Probably it can be implemented in a way which doesn't hurt Chrome that much, but, given it doesn't help anyway, this wasn't considered. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Fri Apr 9 14:17:49 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 Apr 2021 17:17:49 +0300 Subject: [PATCH] Support of proxy v2 protocol for NGINX stream module In-Reply-To: References: Message-ID: Hello! On Fri, Apr 09, 2021 at 04:26:52PM +0300, Vasiliy Soshnikov wrote: [...] > + /** SSL TLVs */ > + > +#if (NGX_STREAM_SSL) > + > + data = NULL; > + data_len = 0; > + > + tlv = (ngx_tlv_ssl_t *) (buf + len); > + ngx_memzero(tlv, sizeof(ngx_tlv_ssl_t)); > + > + tlv->tlv.type = NGX_PROXY_PROTOCOL_V2_TYPE_SSL; > + pos = buf + len + sizeof(ngx_tlv_ssl_t); > + > + tlv->client |= NGX_PROXY_PROTOCOL_V2_CLIENT_SSL; > + > + if (c->ssl != NULL) { > + > +#ifdef TLSEXT_TYPE_application_layer_protocol_negotiation > + SSL_get0_alpn_selected(c->ssl->connection, &data, &data_len); > + > +#ifdef TLSEXT_TYPE_next_proto_neg > + if (data_len == 0) { > + SSL_get0_next_proto_negotiated(c->ssl->connection, > + &data, &data_len); > + } > +#endif > + > +#else /* TLSEXT_TYPE_next_proto_neg */ > + SSL_get0_next_proto_negotiated(c->ssl->connection, &data, > &data_len); > +#endif > + > + if (data_len) { > + > + pos = ngx_copy_tlv(pos, last, > + NGX_PROXY_PROTOCOL_V2_TYPE_ALPN, > + (u_char *) data, (uint16_t) data_len); > + if (pos == NULL) { > + return NULL; > + } > + } > + > + value = (u_char *) SSL_get_version(c->ssl->connection); > + if (value != NULL) { > + > + pos = ngx_copy_tlv(pos, last, > + NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_VERSION, > + value, ngx_strlen(value)); > + if (pos == NULL) { > + return NULL; > + } > + } [...] Thanks for the patch. For the record, as discussed privately: this is more or less proof-of-concept for the ticket #1639[1], used for tests with RabbitMQ[2]. A committable solution probably needs something similar to proxy_set_header / fastcgi_param to control TLVs sent to the upstream server instead of hardcoding them. [1] https://trac.nginx.org/nginx/ticket/1639 [2] https://www.rabbitmq.com/networking.html#proxy-protocol -- Maxim Dounin http://mdounin.ru/ From vasiliy.soshnikov at gmail.com Fri Apr 9 15:08:37 2021 From: vasiliy.soshnikov at gmail.com (Vasiliy Soshnikov) Date: Fri, 9 Apr 2021 18:08:37 +0300 Subject: [PATCH] Support of proxy v2 protocol for NGINX stream module In-Reply-To: References: Message-ID: Hello, Yeah. The proposed design would work well for me. On Fri, Apr 9, 2021 at 5:17 PM Maxim Dounin wrote: > Hello! > > On Fri, Apr 09, 2021 at 04:26:52PM +0300, Vasiliy Soshnikov wrote: > > [...] > > > + /** SSL TLVs */ > > + > > +#if (NGX_STREAM_SSL) > > + > > + data = NULL; > > + data_len = 0; > > + > > + tlv = (ngx_tlv_ssl_t *) (buf + len); > > + ngx_memzero(tlv, sizeof(ngx_tlv_ssl_t)); > > + > > + tlv->tlv.type = NGX_PROXY_PROTOCOL_V2_TYPE_SSL; > > + pos = buf + len + sizeof(ngx_tlv_ssl_t); > > + > > + tlv->client |= NGX_PROXY_PROTOCOL_V2_CLIENT_SSL; > > + > > + if (c->ssl != NULL) { > > + > > +#ifdef TLSEXT_TYPE_application_layer_protocol_negotiation > > + SSL_get0_alpn_selected(c->ssl->connection, &data, &data_len); > > + > > +#ifdef TLSEXT_TYPE_next_proto_neg > > + if (data_len == 0) { > > + SSL_get0_next_proto_negotiated(c->ssl->connection, > > + &data, &data_len); > > + } > > +#endif > > + > > +#else /* TLSEXT_TYPE_next_proto_neg */ > > + SSL_get0_next_proto_negotiated(c->ssl->connection, &data, > > &data_len); > > +#endif > > + > > + if (data_len) { > > + > > + pos = ngx_copy_tlv(pos, last, > > + NGX_PROXY_PROTOCOL_V2_TYPE_ALPN, > > + (u_char *) data, (uint16_t) data_len); > > + if (pos == NULL) { > > + return NULL; > > + } > > + } > > + > > + value = (u_char *) SSL_get_version(c->ssl->connection); > > + if (value != NULL) { > > + > > + pos = ngx_copy_tlv(pos, last, > > + NGX_PROXY_PROTOCOL_V2_SUBTYPE_SSL_VERSION, > > + value, ngx_strlen(value)); > > + if (pos == NULL) { > > + return NULL; > > + } > > + } > > [...] > > Thanks for the patch. > > For the record, as discussed privately: this is more or less > proof-of-concept for the ticket #1639[1], used for tests with > RabbitMQ[2]. A committable solution probably needs something similar > to proxy_set_header / fastcgi_param to control TLVs sent to the > upstream server instead of hardcoding them. > > [1] https://trac.nginx.org/nginx/ticket/1639 > [2] https://www.rabbitmq.com/networking.html#proxy-protocol > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Apr 13 15:34:25 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 13 Apr 2021 15:34:25 +0000 Subject: [nginx] nginx-1.19.10-RELEASE Message-ID: details: https://hg.nginx.org/nginx/rev/ffcbb9980ee2 branches: changeset: 7823:ffcbb9980ee2 user: Maxim Dounin date: Tue Apr 13 18:13:58 2021 +0300 description: nginx-1.19.10-RELEASE diffstat: docs/xml/nginx/changes.xml | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 43 insertions(+), 0 deletions(-) diffs (53 lines): diff -r 82e174e47663 -r ffcbb9980ee2 docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml Thu Apr 08 00:16:30 2021 +0300 +++ b/docs/xml/nginx/changes.xml Tue Apr 13 18:13:58 2021 +0300 @@ -5,6 +5,49 @@ + + + + +? ????????? keepalive_requests ???????? ?? ????????? ???????? ?? 1000. + + +the default value of the "keepalive_requests" directive was changed to 1000. + + + + + +????????? keepalive_time. + + +the "keepalive_time" directive. + + + + + +?????????? $connection_time. + + +the $connection_time variable. + + + + + +??? ????????????? zlib-ng +? ????? ?????????? ????????? "gzip filter failed to use preallocated memory". + + +"gzip filter failed to use preallocated memory" alerts appeared in logs +when using zlib-ng. + + + + + + From mdounin at mdounin.ru Tue Apr 13 15:34:28 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 13 Apr 2021 15:34:28 +0000 Subject: [nginx] release-1.19.10 tag Message-ID: details: https://hg.nginx.org/nginx/rev/b56c45e3bd50 branches: changeset: 7824:b56c45e3bd50 user: Maxim Dounin date: Tue Apr 13 18:13:59 2021 +0300 description: release-1.19.10 tag diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff -r ffcbb9980ee2 -r b56c45e3bd50 .hgtags --- a/.hgtags Tue Apr 13 18:13:58 2021 +0300 +++ b/.hgtags Tue Apr 13 18:13:59 2021 +0300 @@ -459,3 +459,4 @@ f618488eb769e0ed74ef0d93cd118d2ad79ef94d 3fa6e2095a7a51acc630517e1c27a7b7ac41f7b3 release-1.19.7 8c65d21464aaa5923775f80c32474adc7a320068 release-1.19.8 da571b8eaf8f30f36c43b3c9b25e01e31f47149c release-1.19.9 +ffcbb9980ee2bad27b4d7b1cd680b14ff47b29aa release-1.19.10 From anb at papla.net Thu Apr 15 04:41:42 2021 From: anb at papla.net (Anbang Wen) Date: Wed, 14 Apr 2021 21:41:42 -0700 Subject: [PATCH] Stream: added config option for TCP_FASTOPEN Message-ID: <8735vstb09.fsf@papla.net> # HG changeset patch # User Anbang Wen # Date 1618433103 25200 # Wed Apr 14 13:45:03 2021 -0700 # Node ID 495a4d1d58835f7a05b24fb1aad84027f43f90c9 # Parent b56c45e3bd5029f98e1e847eebad75430e2cca27 Stream: added config option for TCP_FASTOPEN This commit adds a "fastopen" option to stream module. The option behaves exactly the same as the one in HTTP core module. diff -r b56c45e3bd50 -r 495a4d1d5883 src/stream/ngx_stream.c --- a/src/stream/ngx_stream.c Tue Apr 13 18:13:59 2021 +0300 +++ b/src/stream/ngx_stream.c Wed Apr 14 13:45:03 2021 -0700 @@ -514,6 +514,10 @@ ls->reuseport = addr[i].opt.reuseport; #endif +#if (NGX_HAVE_TCP_FASTOPEN) + ls->fastopen = addr[i].opt.fastopen; +#endif + stport = ngx_palloc(cf->pool, sizeof(ngx_stream_port_t)); if (stport == NULL) { return NGX_CONF_ERROR; diff -r b56c45e3bd50 -r 495a4d1d5883 src/stream/ngx_stream.h --- a/src/stream/ngx_stream.h Tue Apr 13 18:13:59 2021 +0300 +++ b/src/stream/ngx_stream.h Wed Apr 14 13:45:03 2021 -0700 @@ -66,6 +66,9 @@ int rcvbuf; int sndbuf; int type; +#if (NGX_HAVE_TCP_FASTOPEN) + int fastopen; +#endif } ngx_stream_listen_t; diff -r b56c45e3bd50 -r 495a4d1d5883 src/stream/ngx_stream_core_module.c --- a/src/stream/ngx_stream_core_module.c Tue Apr 13 18:13:59 2021 +0300 +++ b/src/stream/ngx_stream_core_module.c Wed Apr 14 13:45:03 2021 -0700 @@ -619,6 +619,10 @@ ls->ipv6only = 1; #endif +#if (NGX_HAVE_TCP_FASTOPEN) + ls->fastopen = -1; +#endif + backlog = 0; for (i = 2; i < cf->args->nelts; i++) { @@ -836,6 +840,19 @@ continue; } +#if (NGX_HAVE_TCP_FASTOPEN) + if (ngx_strncmp(value[i].data, "fastopen=", 9) == 0) { + ls->fastopen = ngx_atoi(value[i].data + 9, value[i].len - 9); + if (ls->fastopen == NGX_ERROR) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid fastopen \"%V\"", &value[i]); + return NGX_CONF_ERROR; + } + + continue; + } +#endif + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "the invalid \"%V\" parameter", &value[i]); return NGX_CONF_ERROR; @@ -859,6 +876,10 @@ if (ls->proxy_protocol) { return "\"proxy_protocol\" parameter is incompatible with \"udp\""; } + + if (ls->fastopen >= 0) { + return "\"fastopen\" parameter is incompatible with \"udp\""; + } } als = cmcf->listen.elts; From johnnywang1991 at msn.com Thu Apr 15 08:59:16 2021 From: johnnywang1991 at msn.com (=?big5?B?yqggpac=?=) Date: Thu, 15 Apr 2021 08:59:16 +0000 Subject: PATCH: Initialize 'ch' with 'ngx_memzero' to prevent the use of uninitialized bytes. In-Reply-To: References: Message-ID: # HG changeset patch # User Johnny Wang # Date 1618472983 -28800 # Thu Apr 15 15:49:43 2021 +0800 # Node ID a5903fe55622e839906e9544c1a08a66c1e0cf91 # Parent b56c45e3bd5029f98e1e847eebad75430e2cca27 Initialize 'ch' with 'ngx_memzero' to prevent the use of uninitialized bytes. I found the following error when running nginx with valgrind: ==31539== Syscall param sendmsg(msg.msg_iov[0]) points to uninitialised byte(s) ==31539== at 0x5047C50: __sendmsg_nocancel (in /lib64/libpthread-2.17.so) ==31539== by 0x44A6F8: ngx_write_channel (ngx_channel.c:79) ==31539== by 0x44C19A: ngx_pass_open_channel (ngx_process_cycle.c:481) ==31539== by 0x44C21A: ngx_start_worker_processes (ngx_process_cycle.c:385) ==31539== by 0x44E0BA: ngx_master_process_cycle (ngx_process_cycle.c:241) ==31539== by 0x424719: main (nginx.c:386) ==31539== Address 0x1ffefffabc is on thread 1's stack ==31539== in frame #2, created by ngx_pass_open_channel (ngx_process_cycle.c:455) ==31539== { Memcheck:Param sendmsg(msg.msg_iov[0]) fun:__sendmsg_nocancel fun:ngx_write_channel fun:ngx_pass_open_channel fun:ngx_start_worker_processes fun:ngx_master_process_cycle fun:main } diff -r b56c45e3bd50 -r a5903fe55622 src/os/unix/ngx_process_cycle.c --- a/src/os/unix/ngx_process_cycle.c Tue Apr 13 18:13:59 2021 +0300 +++ b/src/os/unix/ngx_process_cycle.c Thu Apr 15 15:49:43 2021 +0800 @@ -398,6 +398,8 @@ ngx_int_t i; ngx_channel_t ch; + ngx_memzero(&ch, sizeof(ngx_channel_t)); + ch.command = NGX_CMD_OPEN_CHANNEL; ch.pid = ngx_processes[ngx_process_slot].pid; ch.slot = ngx_process_slot; -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Fri Apr 16 14:29:11 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 16 Apr 2021 14:29:11 +0000 Subject: [njs] Version bump. Message-ID: details: https://hg.nginx.org/njs/rev/4a7c18a7825a branches: changeset: 1625:4a7c18a7825a user: Dmitry Volyntsev date: Fri Apr 16 13:35:20 2021 +0000 description: Version bump. diffstat: src/njs.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r d25d92370bfd -r 4a7c18a7825a src/njs.h --- a/src/njs.h Tue Mar 30 13:58:27 2021 +0000 +++ b/src/njs.h Fri Apr 16 13:35:20 2021 +0000 @@ -11,7 +11,7 @@ #include -#define NJS_VERSION "0.5.3" +#define NJS_VERSION "0.6.0" #include /* STDOUT_FILENO, STDERR_FILENO */ From xeioex at nginx.com Fri Apr 16 14:29:13 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 16 Apr 2021 14:29:13 +0000 Subject: [njs] Fixed expect tests. Message-ID: details: https://hg.nginx.org/njs/rev/0746b2ed76cc branches: changeset: 1626:0746b2ed76cc user: Dmitry Volyntsev date: Fri Apr 16 13:46:46 2021 +0000 description: Fixed expect tests. diffstat: test/njs_expect_test.exp | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diffs (33 lines): diff -r 4a7c18a7825a -r 0746b2ed76cc test/njs_expect_test.exp --- a/test/njs_expect_test.exp Fri Apr 16 13:35:20 2021 +0000 +++ b/test/njs_expect_test.exp Fri Apr 16 13:46:46 2021 +0000 @@ -70,7 +70,7 @@ njs_test { # Global completions, yes njs_test { {"\t\ty" - "\a\r\nDisplay all*possibilities? (y or n)*await"} + "\a\r\nDisplay all*possibilities? (y or n)*Array"} } # Global completions, single partial match @@ -393,9 +393,9 @@ njs_test { # Non-ASCII characters njs_test { {"'???'\r\n" - "???"} + "'???'"} {"var v = '?????????????';v[10]\r\n" - "?"} + "'?'"} } # Immediate events @@ -805,7 +805,7 @@ njs_test { {"(new Function('return this'))() === globalThis\r\n" "true\r\n"} {"new Function('return this;')\r\n" - "TypeError: function constructor is disabled in \"safe\" mode\r\n"} + "[Function]"} {"new Function('return thi')\r\n" "TypeError: function constructor is disabled in \"safe\" mode\r\n"} } "-u" From mdounin at mdounin.ru Tue Apr 20 14:42:22 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 20 Apr 2021 14:42:22 +0000 Subject: [nginx] Stable branch. Message-ID: details: https://hg.nginx.org/nginx/rev/1231c9e3afb8 branches: stable-1.20 changeset: 7825:1231c9e3afb8 user: Maxim Dounin date: Tue Apr 20 16:06:58 2021 +0300 description: Stable branch. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r b56c45e3bd50 -r 1231c9e3afb8 src/core/nginx.h --- a/src/core/nginx.h Tue Apr 13 18:13:59 2021 +0300 +++ b/src/core/nginx.h Tue Apr 20 16:06:58 2021 +0300 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1019010 -#define NGINX_VERSION "1.19.10" +#define nginx_version 1020000 +#define NGINX_VERSION "1.20.0" #define NGINX_VER "nginx/" NGINX_VERSION #ifdef NGX_BUILD From mdounin at mdounin.ru Tue Apr 20 14:42:25 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 20 Apr 2021 14:42:25 +0000 Subject: [nginx] nginx-1.20.0-RELEASE Message-ID: details: https://hg.nginx.org/nginx/rev/1df854b66534 branches: stable-1.20 changeset: 7826:1df854b66534 user: Maxim Dounin date: Tue Apr 20 16:35:46 2021 +0300 description: nginx-1.20.0-RELEASE diffstat: docs/xml/nginx/changes.xml | 14 ++++++++++++++ 1 files changed, 14 insertions(+), 0 deletions(-) diffs (24 lines): diff -r 1231c9e3afb8 -r 1df854b66534 docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml Tue Apr 20 16:06:58 2021 +0300 +++ b/docs/xml/nginx/changes.xml Tue Apr 20 16:35:46 2021 +0300 @@ -5,6 +5,20 @@ + + + + +?????????? ????? 1.20.x. + + +1.20.x stable branch. + + + + + + From mdounin at mdounin.ru Tue Apr 20 14:42:28 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 20 Apr 2021 14:42:28 +0000 Subject: [nginx] release-1.20.0 tag Message-ID: details: https://hg.nginx.org/nginx/rev/3ebf8a5fb670 branches: stable-1.20 changeset: 7827:3ebf8a5fb670 user: Maxim Dounin date: Tue Apr 20 16:35:47 2021 +0300 description: release-1.20.0 tag diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff -r 1df854b66534 -r 3ebf8a5fb670 .hgtags --- a/.hgtags Tue Apr 20 16:35:46 2021 +0300 +++ b/.hgtags Tue Apr 20 16:35:47 2021 +0300 @@ -460,3 +460,4 @@ 3fa6e2095a7a51acc630517e1c27a7b7ac41f7b3 8c65d21464aaa5923775f80c32474adc7a320068 release-1.19.8 da571b8eaf8f30f36c43b3c9b25e01e31f47149c release-1.19.9 ffcbb9980ee2bad27b4d7b1cd680b14ff47b29aa release-1.19.10 +1df854b66534be699b0b9bbe337d4f799ebf5d13 release-1.20.0 From roy-orbison at devo.net.au Wed Apr 21 02:17:10 2021 From: roy-orbison at devo.net.au (roy-orbison at devo.net.au) Date: Wed, 21 Apr 2021 11:47:10 +0930 Subject: Literal dollar signs in values Message-ID: <64246d71-b444-b1a5-5af4-eea3fbf2f78c@devo.net.au> Hi, I can't find an existing bug on this, and don't have a recent version handy to test on, so I'm asking here first: Is it still impossible to insert a literal dollar symbol in a config value? I was debugging nested location config with a statement like so: return 200 'loc /foo/bar/ \.xyz$ $uri'; This caused an "[emerg] invalid variable name" error when testing the config. Apparently a known issue that's documented at https://github.com/openresty/nginx-tutorials/blob/9f7242b60f9f95bd840ec54388f83a17e3407ee3/en/01-NginxVariables01.tut#L93 but the workaround is very hacky. If this is not resolved already, could this be done in a backwards-compatible manner by nginx interpreting "$$" as a literal "$"? Regards, Roy. From mdounin at mdounin.ru Wed Apr 21 13:24:34 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 21 Apr 2021 16:24:34 +0300 Subject: Literal dollar signs in values In-Reply-To: <64246d71-b444-b1a5-5af4-eea3fbf2f78c@devo.net.au> References: <64246d71-b444-b1a5-5af4-eea3fbf2f78c@devo.net.au> Message-ID: Hello! On Wed, Apr 21, 2021 at 11:47:10AM +0930, roy-orbison at devo.net.au wrote: > I can't find an existing bug on this, and don't have a recent version > handy to test on, so I'm asking here first: Is it still impossible to > insert a literal dollar symbol in a config value? https://trac.nginx.org/nginx/ticket/640 -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Wed Apr 21 21:24:10 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 21 Apr 2021 21:24:10 +0000 Subject: [nginx] Version bump. Message-ID: details: https://hg.nginx.org/nginx/rev/c40f7a65c506 branches: changeset: 7828:c40f7a65c506 user: Maxim Dounin date: Wed Apr 21 23:24:48 2021 +0300 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r b56c45e3bd50 -r c40f7a65c506 src/core/nginx.h --- a/src/core/nginx.h Tue Apr 13 18:13:59 2021 +0300 +++ b/src/core/nginx.h Wed Apr 21 23:24:48 2021 +0300 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1019010 -#define NGINX_VERSION "1.19.10" +#define nginx_version 1021000 +#define NGINX_VERSION "1.21.0" #define NGINX_VER "nginx/" NGINX_VERSION #ifdef NGX_BUILD From mdounin at mdounin.ru Wed Apr 21 21:24:13 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 21 Apr 2021 21:24:13 +0000 Subject: [nginx] Mail: fixed reading with fully filled buffer (ticket #2159). Message-ID: details: https://hg.nginx.org/nginx/rev/2851e4c7de03 branches: changeset: 7829:2851e4c7de03 user: Maxim Dounin date: Wed Apr 21 23:24:59 2021 +0300 description: Mail: fixed reading with fully filled buffer (ticket #2159). With SMTP pipelining, ngx_mail_read_command() can be called with s->buffer without any space available, to parse additional commands received to the buffer on previous calls. Previously, this resulted in recv() being called with zero length, resulting in zero being returned, which was interpreted as a connection close by the client, so nginx silently closed connection. Fix is to avoid calling c->recv() if there is no free space in the buffer, but continue parsing of the already received commands. diffstat: src/mail/ngx_mail_handler.c | 25 ++++++++++++++----------- 1 files changed, 14 insertions(+), 11 deletions(-) diffs (38 lines): diff -r c40f7a65c506 -r 2851e4c7de03 src/mail/ngx_mail_handler.c --- a/src/mail/ngx_mail_handler.c Wed Apr 21 23:24:48 2021 +0300 +++ b/src/mail/ngx_mail_handler.c Wed Apr 21 23:24:59 2021 +0300 @@ -833,20 +833,23 @@ ngx_mail_read_command(ngx_mail_session_t ngx_str_t l; ngx_mail_core_srv_conf_t *cscf; - n = c->recv(c, s->buffer->last, s->buffer->end - s->buffer->last); + if (s->buffer->last < s->buffer->end) { + + n = c->recv(c, s->buffer->last, s->buffer->end - s->buffer->last); - if (n == NGX_ERROR || n == 0) { - ngx_mail_close_connection(c); - return NGX_ERROR; - } + if (n == NGX_ERROR || n == 0) { + ngx_mail_close_connection(c); + return NGX_ERROR; + } - if (n > 0) { - s->buffer->last += n; - } + if (n > 0) { + s->buffer->last += n; + } - if (n == NGX_AGAIN) { - if (s->buffer->pos == s->buffer->last) { - return NGX_AGAIN; + if (n == NGX_AGAIN) { + if (s->buffer->pos == s->buffer->last) { + return NGX_AGAIN; + } } } From ru at nginx.com Thu Apr 22 13:19:22 2021 From: ru at nginx.com (Ruslan Ermilov) Date: Thu, 22 Apr 2021 13:19:22 +0000 Subject: [nginx] Restored zeroing of ngx_channel_t in ngx_pass_open_channel(). Message-ID: details: https://hg.nginx.org/nginx/rev/f2ff291bbdac branches: changeset: 7830:f2ff291bbdac user: Ruslan Ermilov date: Thu Apr 22 16:12:52 2021 +0300 description: Restored zeroing of ngx_channel_t in ngx_pass_open_channel(). Due to structure's alignment, some uninitialized memory contents may have been passed between processes. Zeroing was removed in 0215ec9aaa8a. Reported by Johnny Wang. diffstat: src/os/unix/ngx_process_cycle.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diffs (12 lines): diff -r 2851e4c7de03 -r f2ff291bbdac src/os/unix/ngx_process_cycle.c --- a/src/os/unix/ngx_process_cycle.c Wed Apr 21 23:24:59 2021 +0300 +++ b/src/os/unix/ngx_process_cycle.c Thu Apr 22 16:12:52 2021 +0300 @@ -398,6 +398,8 @@ ngx_pass_open_channel(ngx_cycle_t *cycle ngx_int_t i; ngx_channel_t ch; + ngx_memzero(&ch, sizeof(ngx_channel_t)); + ch.command = NGX_CMD_OPEN_CHANNEL; ch.pid = ngx_processes[ngx_process_slot].pid; ch.slot = ngx_process_slot; From ru at nginx.com Thu Apr 22 13:22:11 2021 From: ru at nginx.com (Ruslan Ermilov) Date: Thu, 22 Apr 2021 16:22:11 +0300 Subject: PATCH: Initialize 'ch' with 'ngx_memzero' to prevent the use of uninitialized bytes. In-Reply-To: References: Message-ID: <20210422132211.GA37970@lo0.su> On Thu, Apr 15, 2021 at 08:59:16AM +0000, ? ? wrote: > # HG changeset patch > # User Johnny Wang > # Date 1618472983 -28800 > # Thu Apr 15 15:49:43 2021 +0800 > # Node ID a5903fe55622e839906e9544c1a08a66c1e0cf91 > # Parent b56c45e3bd5029f98e1e847eebad75430e2cca27 > Initialize 'ch' with 'ngx_memzero' to prevent the use of uninitialized bytes. > > I found the following error when running nginx with valgrind: > > ==31539== Syscall param sendmsg(msg.msg_iov[0]) points to uninitialised byte(s) > ==31539== at 0x5047C50: __sendmsg_nocancel (in /lib64/libpthread-2.17.so) > ==31539== by 0x44A6F8: ngx_write_channel (ngx_channel.c:79) > ==31539== by 0x44C19A: ngx_pass_open_channel (ngx_process_cycle.c:481) > ==31539== by 0x44C21A: ngx_start_worker_processes (ngx_process_cycle.c:385) > ==31539== by 0x44E0BA: ngx_master_process_cycle (ngx_process_cycle.c:241) > ==31539== by 0x424719: main (nginx.c:386) > ==31539== Address 0x1ffefffabc is on thread 1's stack > ==31539== in frame #2, created by ngx_pass_open_channel (ngx_process_cycle.c:455) > ==31539== > { > > Memcheck:Param > sendmsg(msg.msg_iov[0]) > fun:__sendmsg_nocancel > fun:ngx_write_channel > fun:ngx_pass_open_channel > fun:ngx_start_worker_processes > fun:ngx_master_process_cycle > fun:main > } > > diff -r b56c45e3bd50 -r a5903fe55622 src/os/unix/ngx_process_cycle.c > --- a/src/os/unix/ngx_process_cycle.c Tue Apr 13 18:13:59 2021 +0300 > +++ b/src/os/unix/ngx_process_cycle.c Thu Apr 15 15:49:43 2021 +0800 > @@ -398,6 +398,8 @@ > ngx_int_t i; > ngx_channel_t ch; > + ngx_memzero(&ch, sizeof(ngx_channel_t)); > + > ch.command = NGX_CMD_OPEN_CHANNEL; > ch.pid = ngx_processes[ngx_process_slot].pid; > ch.slot = ngx_process_slot; > http://hg.nginx.org/nginx/rev/f2ff291bbdac From gfrankliu at gmail.com Thu Apr 22 19:41:57 2021 From: gfrankliu at gmail.com (Frank Liu) Date: Thu, 22 Apr 2021 12:41:57 -0700 Subject: [nginx] Restored zeroing of ngx_channel_t in ngx_pass_open_channel(). In-Reply-To: References: Message-ID: How does this impact 1.20.0? Shall we wait for 1.20.1? On Thu, Apr 22, 2021 at 6:19 AM Ruslan Ermilov wrote: > details: https://hg.nginx.org/nginx/rev/f2ff291bbdac > branches: > changeset: 7830:f2ff291bbdac > user: Ruslan Ermilov > date: Thu Apr 22 16:12:52 2021 +0300 > description: > Restored zeroing of ngx_channel_t in ngx_pass_open_channel(). > > Due to structure's alignment, some uninitialized memory contents may have > been passed between processes. > > Zeroing was removed in 0215ec9aaa8a. > > Reported by Johnny Wang. > > diffstat: > > src/os/unix/ngx_process_cycle.c | 2 ++ > 1 files changed, 2 insertions(+), 0 deletions(-) > > diffs (12 lines): > > diff -r 2851e4c7de03 -r f2ff291bbdac src/os/unix/ngx_process_cycle.c > --- a/src/os/unix/ngx_process_cycle.c Wed Apr 21 23:24:59 2021 +0300 > +++ b/src/os/unix/ngx_process_cycle.c Thu Apr 22 16:12:52 2021 +0300 > @@ -398,6 +398,8 @@ ngx_pass_open_channel(ngx_cycle_t *cycle > ngx_int_t i; > ngx_channel_t ch; > > + ngx_memzero(&ch, sizeof(ngx_channel_t)); > + > ch.command = NGX_CMD_OPEN_CHANNEL; > ch.pid = ngx_processes[ngx_process_slot].pid; > ch.slot = ngx_process_slot; > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Thu Apr 22 20:34:28 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 22 Apr 2021 23:34:28 +0300 Subject: [nginx] Restored zeroing of ngx_channel_t in ngx_pass_open_channel(). In-Reply-To: References: Message-ID: Hello! On Thu, Apr 22, 2021 at 12:41:57PM -0700, Frank Liu wrote: > How does this impact 1.20.0? Shall we wait for 1.20.1? The only impact is that Valgrind complains. No plans to merge this into the 1.20.x branch. -- Maxim Dounin http://mdounin.ru/ From gunter at grodotzki.com Tue Apr 27 21:21:26 2021 From: gunter at grodotzki.com (Gunter Grodotzki) Date: Tue, 27 Apr 2021 23:21:26 +0200 Subject: njs: get raw requestURI or full querystring Message-ID: An HTML attachment was scrubbed... URL: From osa at freebsd.org.ru Wed Apr 28 02:50:38 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Wed, 28 Apr 2021 05:50:38 +0300 Subject: njs: get raw requestURI or full querystring In-Reply-To: References: Message-ID: Hi Gunter, hope you're doing well these days. On Tue, Apr 27, 2021 at 11:21:26PM +0200, Gunter Grodotzki wrote: > Hi all, > > I would love to try out nginx njs for https://url.rw > > Unfortunately I do not seem to be able to get the value of the querystring e.g. > in my case it does not contain any key-value pairs: > > https://url.rw/?https%3A%2F%2Flifeofguenter.de > > I am interested in the value ?https%3A%2F%2Flifeofguenter.de? (well ideally > urldecoded). According to https://nginx.org/en/docs/njs/reference.html#http, the HTTP request object has arguments: r.args{} request arguments object, read-only It's possible to get an argument with the following example: for (var arg in r.args) { if (arg != '') { str += arg; break; } } The str can be unescaped then with the following function, http://nginx.org/en/docs/njs/reference.html#querystring_unescape querystring.unescape(string) Performs decoding of URL percent-encoded characters of the string, returns an unescaped query string. Hope that helps. -- Sergey Osokin From gunter at grodotzki.com Wed Apr 28 05:45:37 2021 From: gunter at grodotzki.com (Gunter Grodotzki) Date: Wed, 28 Apr 2021 07:45:37 +0200 Subject: njs: get raw requestURI or full querystring In-Reply-To: References: Message-ID: Unfortunately r.args is empty if no key/value pair is given. > On 28 Apr 2021, at 04:50, Sergey A. Osokin wrote: > > ?Hi Gunter, > > hope you're doing well these days. > >> On Tue, Apr 27, 2021 at 11:21:26PM +0200, Gunter Grodotzki wrote: >> Hi all, >> >> I would love to try out nginx njs for https://url.rw >> >> Unfortunately I do not seem to be able to get the value of the querystring e.g. >> in my case it does not contain any key-value pairs: >> >> https://url.rw/?https%3A%2F%2Flifeofguenter.de >> >> I am interested in the value ?https%3A%2F%2Flifeofguenter.de? (well ideally >> urldecoded). > > According to https://nginx.org/en/docs/njs/reference.html#http, the HTTP > request object has arguments: > > r.args{} > request arguments object, read-only > > It's possible to get an argument with the following example: > > for (var arg in r.args) { > if (arg != '') { > str += arg; > break; > } > } > > The str can be unescaped then with the following function, > http://nginx.org/en/docs/njs/reference.html#querystring_unescape > > querystring.unescape(string) > > Performs decoding of URL percent-encoded characters of the string, > returns an unescaped query string. > > > Hope that helps. > > -- > Sergey Osokin > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From osa at freebsd.org.ru Wed Apr 28 12:17:09 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Wed, 28 Apr 2021 15:17:09 +0300 Subject: njs: get raw requestURI or full querystring In-Reply-To: References: Message-ID: On Wed, Apr 28, 2021 at 07:45:37AM +0200, Gunter Grodotzki wrote: > Unfortunately r.args is empty if no key/value pair is given. Well, the r.args is empty in case of an emptyness of a querystring. But we have a bit different case when URL contains a query string with a key, but without a value for that one, i.e. "https://url.rw/?https%3A%2F%2Flifeofguenter.de". So, the "https%3A%2F%2Flifeofguenter.de" is the key and its value is empty. -- Sergey Osokin From gunter at grodotzki.com Wed Apr 28 13:12:09 2021 From: gunter at grodotzki.com (Gunter Grodotzki) Date: Wed, 28 Apr 2021 15:12:09 +0200 Subject: njs: get raw requestURI or full querystring In-Reply-To: References: Message-ID: <1E337432-A8ED-4C12-9E6F-DE724BF5CBB8@grodotzki.com> sadly the whole object is empty. no key / no value. > On 28 Apr 2021, at 14:17, Sergey A. Osokin wrote: > > ?On Wed, Apr 28, 2021 at 07:45:37AM +0200, Gunter Grodotzki wrote: >> Unfortunately r.args is empty if no key/value pair is given. > > Well, the r.args is empty in case of an emptyness of a querystring. > > But we have a bit different case when URL contains a query > string with a key, but without a value for that one, > i.e. "https://url.rw/?https%3A%2F%2Flifeofguenter.de". > > So, the "https%3A%2F%2Flifeofguenter.de" is the key and its value > is empty. > > -- > Sergey Osokin > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From xeioex at nginx.com Wed Apr 28 13:17:24 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 28 Apr 2021 16:17:24 +0300 Subject: njs: get raw requestURI or full querystring In-Reply-To: <1E337432-A8ED-4C12-9E6F-DE724BF5CBB8@grodotzki.com> References: <1E337432-A8ED-4C12-9E6F-DE724BF5CBB8@grodotzki.com> Message-ID: <88381b64-b1e4-2038-d7f2-beb9face4b00@nginx.com> On 28.04.2021 16:12, Gunter Grodotzki wrote: > sadly the whole object is empty. no key / no value. > > Try the following variable http://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_uri decodeURIComponent(r.variables.request_uri.split('?')[1]) in njs CLI >> decodeURIComponent("https://url.rw/?https%3A%2F%2Flifeofguenter.de".split('?')[1]) 'https://lifeofguenter.de' >> On 28 Apr 2021, at 14:17, Sergey A. Osokin wrote: >> >> ?On Wed, Apr 28, 2021 at 07:45:37AM +0200, Gunter Grodotzki wrote: >>> Unfortunately r.args is empty if no key/value pair is given. >> Well, the r.args is empty in case of an emptyness of a querystring. >> >> But we have a bit different case when URL contains a query >> string with a key, but without a value for that one, >> i.e. "https://url.rw/?https%3A%2F%2Flifeofguenter.de". >> >> So, the "https%3A%2F%2Flifeofguenter.de" is the key and its value >> is empty. >> >> -- >> Sergey Osokin >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From gunter at grodotzki.com Wed Apr 28 14:43:26 2021 From: gunter at grodotzki.com (Gunter Grodotzki) Date: Wed, 28 Apr 2021 16:43:26 +0200 Subject: njs: get raw requestURI or full querystring In-Reply-To: <88381b64-b1e4-2038-d7f2-beb9face4b00@nginx.com> References: <1E337432-A8ED-4C12-9E6F-DE724BF5CBB8@grodotzki.com>, <88381b64-b1e4-2038-d7f2-beb9face4b00@nginx.com> Message-ID: <4AE19A3D-113A-4441-82DC-519B71A77329@hxcore.ol> An HTML attachment was scrubbed... URL: From osa at freebsd.org.ru Wed Apr 28 15:10:08 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Wed, 28 Apr 2021 18:10:08 +0300 Subject: njs: get raw requestURI or full querystring In-Reply-To: <4AE19A3D-113A-4441-82DC-519B71A77329@hxcore.ol> References: <1E337432-A8ED-4C12-9E6F-DE724BF5CBB8@grodotzki.com> <88381b64-b1e4-2038-d7f2-beb9face4b00@nginx.com> <4AE19A3D-113A-4441-82DC-519B71A77329@hxcore.ol> Message-ID: Hi Gunter, On Wed, Apr 28, 2021 at 04:43:26PM +0200, Gunter Grodotzki wrote: > Thanks so much Dmitry & Sergey, > > I did not catch this earlier since dumping r.variables was empty, only by > explicitly accessing the request_uri property I was now able to get what I > needed. > > Very excited to move forward with this ? thanks for the confirmation, I'm glad it works. -- Sergey Osokin From gunter at grodotzki.com Thu Apr 29 01:34:39 2021 From: gunter at grodotzki.com (Gunter Grodotzki) Date: Thu, 29 Apr 2021 03:34:39 +0200 Subject: njs: output compression? Message-ID: An HTML attachment was scrubbed... URL: From gunter at grodotzki.com Thu Apr 29 01:34:39 2021 From: gunter at grodotzki.com (Gunter Grodotzki) Date: Thu, 29 Apr 2021 03:34:39 +0200 Subject: njs: output compression? Message-ID: An HTML attachment was scrubbed... URL: From osa at freebsd.org.ru Thu Apr 29 02:23:28 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Thu, 29 Apr 2021 05:23:28 +0300 Subject: njs: output compression? In-Reply-To: References: Message-ID: Hi Gunter, On Thu, Apr 29, 2021 at 03:34:39AM +0200, Gunter Grodotzki wrote: > Hi all, > > Is there a simple way that I am missing to enable output compression for njs > scripts? I don't think it's good practice to use the njs module for a compression or any other heavy-math calculations. > It seems to be working for r.internalRedirect(?/foo.html?) > > But not for: > r.headersOut['Content-Type'] = 'text/html; charset=UTF-8' > r.return(200, someHtml) Could you share more light about goals you're doing to archive in this case. -- Sergey Osokin From gunter at grodotzki.com Thu Apr 29 06:22:23 2021 From: gunter at grodotzki.com (Gunter Grodotzki) Date: Thu, 29 Apr 2021 08:22:23 +0200 Subject: njs: output compression? In-Reply-To: References: Message-ID: <383BBEED-3152-48DD-95BD-DB71F3F0E181@grodotzki.com> Hi :) So my njs script is outputting html but I would like that output to be compressed for the client (if the client supports). So kind hoping that gzip or brotli module would be chained after njs, but seems that is only the case if you do an internalredirect. Cheers, Gunter > On 29 Apr 2021, at 04:23, Sergey A. Osokin wrote: > > ?Hi Gunter, > >> On Thu, Apr 29, 2021 at 03:34:39AM +0200, Gunter Grodotzki wrote: >> Hi all, >> >> Is there a simple way that I am missing to enable output compression for njs >> scripts? > > I don't think it's good practice to use the njs module for > a compression or any other heavy-math calculations. > >> It seems to be working for r.internalRedirect(?/foo.html?) >> >> But not for: >> r.headersOut['Content-Type'] = 'text/html; charset=UTF-8' >> r.return(200, someHtml) > > Could you share more light about goals you're doing to archive > in this case. > > -- > Sergey Osokin > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From xeioex at nginx.com Fri Apr 30 10:11:39 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 30 Apr 2021 10:11:39 +0000 Subject: [njs] Added function micro benchmarks. Message-ID: details: https://hg.nginx.org/njs/rev/282a13ecd193 branches: changeset: 1627:282a13ecd193 user: Dmitry Volyntsev date: Fri Apr 30 10:10:57 2021 +0000 description: Added function micro benchmarks. diffstat: src/test/njs_benchmark.c | 52 +++++++++++++++++++++++++++++++++++++++++------ 1 files changed, 45 insertions(+), 7 deletions(-) diffs (98 lines): diff -r 0746b2ed76cc -r 282a13ecd193 src/test/njs_benchmark.c --- a/src/test/njs_benchmark.c Fri Apr 16 13:46:46 2021 +0000 +++ b/src/test/njs_benchmark.c Fri Apr 30 10:10:57 2021 +0000 @@ -35,7 +35,7 @@ njs_benchmark_test(njs_vm_t *parent, njs { u_char *start; njs_vm_t *vm, *nvm; - uint64_t us; + uint64_t ns; njs_int_t ret, proto_id; njs_str_t s, *expected; njs_uint_t i, n; @@ -76,7 +76,7 @@ njs_benchmark_test(njs_vm_t *parent, njs expected = &test->result; ret = NJS_ERROR; - us = njs_time() / 1000; + ns = njs_time(); for (i = 0; i < n; i++) { @@ -105,18 +105,18 @@ njs_benchmark_test(njs_vm_t *parent, njs nvm = NULL; } - us = njs_time() / 1000 - us; + ns = njs_time() - ns; if (!opts->dump_report) { if (n == 1) { njs_printf("%s%s: %.3fs\n", opts->previous ? " " : "", - test->name, (double) us / 1000000); + test->name, (double) ns / 1000000000); } else { njs_printf("%s%s: %.3f?s, %d times/s\n", opts->previous ? " " : "", - test->name, (double) us / n, - (int) ((uint64_t) n * 1000000 / us)); + test->name, (double) ns / n / 1000, + (int) ((uint64_t) n * 1000000000 / ns)); } } @@ -133,7 +133,7 @@ njs_benchmark_test(njs_vm_t *parent, njs goto done; } - njs_value_number_set(&usec, us); + njs_value_number_set(&usec, 1000 * ns); njs_value_number_set(×, n); ret = njs_vm_object_alloc(parent, result, &name_key, &name, @@ -165,6 +165,44 @@ static njs_benchmark_test_t njs_test[] njs_str("null"), 1000000 }, + { "func call", + njs_str("function test(a) { return 1 }" + "" + "test(1);" + "test(1);" + "test(1);" + "test(1);"), + njs_str("1"), + 100000 }, + + { "func call (3 local functions)", + njs_str("function test(a) { " + " function g(x) {}" + " function h(x) {}" + " function f(x) {}" + " return 1;" + "}" + "" + "test(1);" + "test(1);" + "test(1);" + "test(1);"), + njs_str("1"), + 100000 }, + + { "closure var global", + njs_str("function test(a) { sum++ }" + "" + "var sum = 0;" + "" + "test(1);" + "test(1);" + "test(1);" + "test(1);" + "sum"), + njs_str("4"), + 100000 }, + { "JSON.parse", njs_str("JSON.parse('{\"a\":123, \"XXX\":[3,4,null]}').a"), njs_str("123"), From xeioex at nginx.com Fri Apr 30 10:11:41 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 30 Apr 2021 10:11:41 +0000 Subject: [njs] Introduced "undefined" as a token in lexer. Message-ID: details: https://hg.nginx.org/njs/rev/cb99db22e3e5 branches: changeset: 1628:cb99db22e3e5 user: Dmitry Volyntsev date: Fri Apr 16 15:25:22 2021 +0000 description: Introduced "undefined" as a token in lexer. diffstat: src/njs_lexer.h | 1 + src/njs_lexer_tables.h | 20 +++++++++++++------- utils/lexer_keyword.py | 1 + 3 files changed, 15 insertions(+), 7 deletions(-) diffs (87 lines): diff -r 282a13ecd193 -r cb99db22e3e5 src/njs_lexer.h --- a/src/njs_lexer.h Fri Apr 30 10:10:57 2021 +0000 +++ b/src/njs_lexer.h Fri Apr 16 15:25:22 2021 +0000 @@ -116,6 +116,7 @@ typedef enum { NJS_TOKEN_NULL, NJS_TOKEN_NUMBER, NJS_TOKEN_TRUE, + NJS_TOKEN_UNDEFINED, NJS_TOKEN_FALSE, NJS_TOKEN_STRING, diff -r 282a13ecd193 -r cb99db22e3e5 src/njs_lexer_tables.h --- a/src/njs_lexer_tables.h Fri Apr 30 10:10:57 2021 +0000 +++ b/src/njs_lexer_tables.h Fri Apr 16 15:25:22 2021 +0000 @@ -10,7 +10,7 @@ #define _NJS_LEXER_TABLES_H_INCLUDED_ -static const njs_keyword_t njs_lexer_kws[53] = +static const njs_keyword_t njs_lexer_kws[54] = { { .entry = { njs_str("arguments") }, @@ -301,6 +301,12 @@ static const njs_keyword_t njs_lexer_kws }, { + .entry = { njs_str("undefined") }, + .type = NJS_TOKEN_UNDEFINED, + .reserved = 0 + }, + + { .entry = { njs_str("var") }, .type = NJS_TOKEN_VAR, .reserved = 1 @@ -364,10 +370,10 @@ static const njs_lexer_keyword_entry_t n { "null", &njs_lexer_kws[32], 4, 0 }, { NULL, NULL, 0, 0 }, { "do", &njs_lexer_kws[12], 2, 0 }, - { "var", &njs_lexer_kws[48], 3, 0 }, + { "var", &njs_lexer_kws[49], 3, 0 }, { "if", &njs_lexer_kws[23], 2, 7 }, { "implements", &njs_lexer_kws[24], 10, 0 }, - { "with", &njs_lexer_kws[51], 4, 0 }, + { "with", &njs_lexer_kws[52], 4, 0 }, { NULL, NULL, 0, 0 }, { "eval", &njs_lexer_kws[15], 4, 9 }, { NULL, NULL, 0, 0 }, @@ -379,14 +385,14 @@ static const njs_lexer_keyword_entry_t n { NULL, NULL, 0, 0 }, { NULL, NULL, 0, 0 }, { "default", &njs_lexer_kws[10], 7, 0 }, - { "void", &njs_lexer_kws[49], 4, 0 }, + { "void", &njs_lexer_kws[50], 4, 0 }, { NULL, NULL, 0, 0 }, { NULL, NULL, 0, 0 }, - { NULL, NULL, 0, 0 }, + { "undefined", &njs_lexer_kws[48], 9, 0 }, { "from", &njs_lexer_kws[21], 4, 0 }, { "package", &njs_lexer_kws[34], 7, 15 }, { NULL, NULL, 0, 0 }, - { "yield", &njs_lexer_kws[52], 5, 0 }, + { "yield", &njs_lexer_kws[53], 5, 0 }, { NULL, NULL, 0, 0 }, { NULL, NULL, 0, 0 }, { "of", &njs_lexer_kws[33], 2, 0 }, @@ -403,7 +409,7 @@ static const njs_lexer_keyword_entry_t n { NULL, NULL, 0, 0 }, { NULL, NULL, 0, 0 }, { "for", &njs_lexer_kws[20], 3, 0 }, - { "while", &njs_lexer_kws[50], 5, 0 }, + { "while", &njs_lexer_kws[51], 5, 0 }, { NULL, NULL, 0, 0 }, { NULL, NULL, 0, 0 }, { NULL, NULL, 0, 0 }, diff -r 282a13ecd193 -r cb99db22e3e5 utils/lexer_keyword.py --- a/utils/lexer_keyword.py Fri Apr 30 10:10:57 2021 +0000 +++ b/utils/lexer_keyword.py Fri Apr 16 15:25:22 2021 +0000 @@ -6,6 +6,7 @@ global_keywords = { "null": 1, "false": 1, "true": 1, + "undefined": 0, # Operators. From alexander.borisov at nginx.com Fri Apr 30 13:04:40 2021 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Fri, 30 Apr 2021 13:04:40 +0000 Subject: [njs] Scopes refactoring. Message-ID: details: https://hg.nginx.org/njs/rev/0a2a0b5a74f4 branches: changeset: 1629:0a2a0b5a74f4 user: Alexander Borisov date: Fri Apr 30 16:02:08 2021 +0300 description: Scopes refactoring. diffstat: auto/sources | 1 + nginx/ngx_js.c | 3 +- src/njs.h | 7 +- src/njs_builtin.c | 20 +- src/njs_disassembler.c | 15 + src/njs_function.c | 456 +++++++++++++++++++----------- src/njs_function.h | 58 ++- src/njs_generator.c | 466 +++++++++++++++++-------------- src/njs_generator.h | 3 +- src/njs_lexer.h | 3 +- src/njs_main.h | 1 + src/njs_module.c | 12 +- src/njs_parser.c | 667 +++++++++++++++++++++++++-------------------- src/njs_parser.h | 37 +- src/njs_scope.c | 259 +++++++++++++++++ src/njs_scope.h | 122 ++++++++ src/njs_shell.c | 4 +- src/njs_string.c | 139 --------- src/njs_string.h | 3 - src/njs_value.h | 21 +- src/njs_variable.c | 689 ++++++++++++++++++++++++---------------------- src/njs_variable.h | 25 +- src/njs_vm.c | 182 +++++------ src/njs_vm.h | 88 +---- src/njs_vmcode.c | 160 +++++----- src/njs_vmcode.h | 182 ++++++----- src/test/njs_unit_test.c | 148 +++++++++- 27 files changed, 2194 insertions(+), 1577 deletions(-) diffs (truncated from 5843 to 1000 lines): diff -r cb99db22e3e5 -r 0a2a0b5a74f4 auto/sources --- a/auto/sources Fri Apr 16 15:25:22 2021 +0000 +++ b/auto/sources Fri Apr 30 16:02:08 2021 +0300 @@ -59,6 +59,7 @@ NJS_LIB_SRCS=" \ src/njs_encoding.c \ src/njs_buffer.c \ src/njs_iterator.c \ + src/njs_scope.c \ " NJS_LIB_TEST_SRCS=" \ diff -r cb99db22e3e5 -r 0a2a0b5a74f4 nginx/ngx_js.c --- a/nginx/ngx_js.c Fri Apr 16 15:25:22 2021 +0000 +++ b/nginx/ngx_js.c Fri Apr 30 16:02:08 2021 +0300 @@ -261,7 +261,8 @@ ngx_js_ext_log(njs_vm_t *vm, njs_value_t handler = c->log->handler; c->log->handler = NULL; - ngx_log_error(level, c->log, 0, "js: %*s", msg.length, msg.start); + ngx_log_error((ngx_uint_t) level, c->log, 0, "js: %*s", + msg.length, msg.start); c->log->handler = handler; diff -r cb99db22e3e5 -r 0a2a0b5a74f4 src/njs.h --- a/src/njs.h Fri Apr 16 15:25:22 2021 +0000 +++ b/src/njs.h Fri Apr 30 16:02:08 2021 +0300 @@ -210,7 +210,7 @@ typedef struct { #define NJS_VM_OPT_UNHANDLED_REJECTION_THROW 1 /* - * accumulative - enables "accumulative" mode to support incremental compiling. + * interactive - enables "interactive" mode. * (REPL). Allows starting parent VM without cloning. * disassemble - enables disassemble. * backtrace - enables backtraces. @@ -225,10 +225,9 @@ typedef struct { * - throwing inside a Promise without a catch block. * - throwing inside in a finally or catch block. */ - + uint8_t interactive; /* 1 bit */ uint8_t trailer; /* 1 bit */ uint8_t init; /* 1 bit */ - uint8_t accumulative; /* 1 bit */ uint8_t disassemble; /* 1 bit */ uint8_t backtrace; /* 1 bit */ uint8_t quiet; /* 1 bit */ @@ -277,7 +276,7 @@ NJS_EXPORT njs_int_t njs_vm_posted(njs_v NJS_EXPORT njs_int_t njs_vm_call(njs_vm_t *vm, njs_function_t *function, const njs_value_t *args, njs_uint_t nargs); NJS_EXPORT njs_int_t njs_vm_invoke(njs_vm_t *vm, njs_function_t *function, - const njs_value_t *args, njs_uint_t nargs, njs_index_t retval); + const njs_value_t *args, njs_uint_t nargs, njs_value_t *retval); /* * Runs posted events. diff -r cb99db22e3e5 -r 0a2a0b5a74f4 src/njs_builtin.c --- a/src/njs_builtin.c Fri Apr 16 15:25:22 2021 +0000 +++ b/src/njs_builtin.c Fri Apr 30 16:02:08 2021 +0300 @@ -320,6 +320,8 @@ njs_builtin_objects_create(njs_vm_t *vm) vm->global_object = shared->objects[0]; vm->global_object.shared = 0; + njs_set_object(&vm->global_value, &vm->global_object); + string_object = &shared->string_object; njs_lvlhsh_init(&string_object->hash); string_object->shared_hash = shared->string_instance_hash; @@ -637,7 +639,7 @@ njs_vm_expression_completions(njs_vm_t * } var = ((njs_variable_node_t *) node)->variable; - value = njs_vmcode_operand(vm, var->index); + value = njs_scope_valid_value(vm, var->index); if (!njs_is_object(value)) { return NULL; @@ -939,6 +941,8 @@ njs_global_this_prop_handler(njs_vm_t *v { njs_int_t ret; njs_value_t *value; + njs_variable_t *var; + njs_function_t *function; njs_rbtree_node_t *rb_node; njs_lvlhsh_query_t lhq; njs_variable_node_t *node, var_node; @@ -965,7 +969,19 @@ njs_global_this_prop_handler(njs_vm_t *v } node = (njs_variable_node_t *) rb_node; - value = njs_vmcode_operand(vm, node->variable->index); + + var = node->variable; + + value = njs_scope_valid_value(vm, var->index); + + if (var->type == NJS_VARIABLE_FUNCTION && njs_is_undefined(value)) { + *value = var->value; + + function = njs_function_value_copy(vm, value); + if (njs_slow_path(function == NULL)) { + return NJS_ERROR; + } + } if (setval != NULL) { *value = *setval; diff -r cb99db22e3e5 -r 0a2a0b5a74f4 src/njs_disassembler.c --- a/src/njs_disassembler.c Fri Apr 16 15:25:22 2021 +0000 +++ b/src/njs_disassembler.c Fri Apr 30 16:02:08 2021 +0300 @@ -35,6 +35,9 @@ static njs_code_name_t code_names[] = { { NJS_VMCODE_OBJECT_COPY, sizeof(njs_vmcode_object_copy_t), njs_str("OBJECT COPY ") }, + { NJS_VMCODE_FUNCTION_COPY, sizeof(njs_vmcode_function_copy_t), + njs_str("FUNCTION COPY ") }, + { NJS_VMCODE_PROPERTY_GET, sizeof(njs_vmcode_prop_get_t), njs_str("PROP GET ") }, { NJS_VMCODE_GLOBAL_GET, sizeof(njs_vmcode_prop_get_t), @@ -186,6 +189,7 @@ njs_disassemble(njs_vm_code_t *code) njs_vmcode_catch_t *catch; njs_vmcode_finally_t *finally; njs_vmcode_try_end_t *try_end; + njs_vmcode_move_arg_t *move_arg; njs_vmcode_try_start_t *try_start; njs_vmcode_operation_t operation; njs_vmcode_cond_jump_t *cond_jump; @@ -485,6 +489,17 @@ njs_disassemble(njs_vm_code_t *code) continue; } + if (operation == NJS_VMCODE_MOVE_ARG) { + move_arg = (njs_vmcode_move_arg_t *) p; + + njs_printf("%5uD | %05uz MOVE ARGUMENT %uD %04Xz\n", + line, p - start, move_arg->dst, (size_t) move_arg->src); + + p += sizeof(njs_vmcode_move_arg_t); + + continue; + } + code_name = code_names; n = njs_nitems(code_names); diff -r cb99db22e3e5 -r 0a2a0b5a74f4 src/njs_function.c --- a/src/njs_function.c Fri Apr 16 15:25:22 2021 +0000 +++ b/src/njs_function.c Fri Apr 30 16:02:08 2021 +0300 @@ -8,21 +8,13 @@ #include -static njs_function_t *njs_function_copy(njs_vm_t *vm, - njs_function_t *function); -static njs_native_frame_t *njs_function_frame_alloc(njs_vm_t *vm, size_t size); - - njs_function_t * -njs_function_alloc(njs_vm_t *vm, njs_function_lambda_t *lambda, - njs_closure_t *closures[], njs_bool_t shared) +njs_function_alloc(njs_vm_t *vm, njs_function_lambda_t *lambda) { size_t size; - njs_uint_t n, nesting; njs_function_t *function; - nesting = lambda->nesting; - size = sizeof(njs_function_t) + nesting * sizeof(njs_closure_t *); + size = sizeof(njs_function_t) + lambda->nclosures * sizeof(njs_value_t *); function = njs_mp_zalloc(vm->mem_pool, size); if (njs_slow_path(function == NULL)) { @@ -48,21 +40,8 @@ njs_function_alloc(njs_vm_t *vm, njs_fun function->object.__proto__ = &vm->prototypes[NJS_OBJ_TYPE_FUNCTION].object; function->object.type = NJS_FUNCTION; - function->object.shared = shared; function->object.extensible = 1; - if (nesting != 0 && closures != NULL) { - function->closure = 1; - - n = 0; - - do { - /* GC: retain closure. */ - njs_function_closures(function)[n] = closures[n]; - n++; - } while (n < nesting); - } - return function; fail: @@ -121,14 +100,6 @@ njs_function_value_copy(njs_vm_t *vm, nj } -njs_inline njs_closure_t ** -njs_function_active_closures(njs_vm_t *vm, njs_function_t *function) -{ - return (function->closure) ? njs_function_closures(function) - : njs_frame_closures(vm->active_frame); -} - - njs_int_t njs_function_name_set(njs_vm_t *vm, njs_function_t *function, njs_value_t *name, const char *prefix) @@ -199,17 +170,16 @@ njs_function_name_set(njs_vm_t *vm, njs_ } -static njs_function_t * +njs_function_t * njs_function_copy(njs_vm_t *vm, njs_function_t *function) { - size_t size; - njs_uint_t n, nesting; - njs_closure_t **closures; + size_t size, n; + njs_value_t **from, **to; njs_function_t *copy; - nesting = (function->native) ? 0 : function->u.lambda->nesting; + n = (function->native) ? 0 : function->u.lambda->nclosures; - size = sizeof(njs_function_t) + nesting * sizeof(njs_closure_t *); + size = sizeof(njs_function_t) + n * sizeof(njs_value_t *); copy = njs_mp_alloc(vm->mem_pool, size); if (njs_slow_path(copy == NULL)) { @@ -220,21 +190,19 @@ njs_function_copy(njs_vm_t *vm, njs_func copy->object.__proto__ = &vm->prototypes[NJS_OBJ_TYPE_FUNCTION].object; copy->object.shared = 0; - if (nesting == 0) { + if (n == 0) { return copy; } - copy->closure = 1; - - closures = njs_function_active_closures(vm, function); - - n = 0; + from = njs_function_closures(function); + to = njs_function_closures(copy); do { - /* GC: retain closure. */ - njs_function_closures(copy)[n] = closures[n]; - n++; - } while (n < nesting); + n--; + + to[n] = from[n]; + + } while (n != 0); return copy; } @@ -287,7 +255,7 @@ njs_function_arguments_object_init(njs_v for (n = 0; n < nargs; n++) { njs_uint32_to_string(&value, n); - prop = njs_object_prop_alloc(vm, &value, &frame->arguments[n + 1], 1); + prop = njs_object_prop_alloc(vm, &value, &frame->arguments[n], 1); if (njs_slow_path(prop == NULL)) { return NJS_ERROR; } @@ -326,19 +294,20 @@ njs_function_rest_parameters_init(njs_vm return NJS_ERROR; } - if (n <= nargs) { - i = 0; - do { - /* GC: retain. */ - array->start[i++] = frame->arguments[n++]; - } while (n <= nargs); + for (i = 0; i < length; i++) { + array->start[i] = frame->arguments[i + n - 1]; } - rest_arguments = &frame->arguments[frame->function->u.lambda->nargs]; + rest_arguments = njs_mp_alloc(vm->mem_pool, sizeof(njs_value_t)); + if (njs_slow_path(rest_arguments == NULL)) { + return NJS_ERROR; + } /* GC: retain. */ njs_set_array(rest_arguments, array); + vm->top_frame->local[n] = rest_arguments; + return NJS_OK; } @@ -397,7 +366,9 @@ njs_function_native_frame(njs_vm_t *vm, frame->pc = NULL; value = (njs_value_t *) ((u_char *) frame + NJS_NATIVE_FRAME_SIZE); + frame->arguments = value; + frame->arguments_offset = value + function->args_offset; bound = function->bound; @@ -415,8 +386,6 @@ njs_function_native_frame(njs_vm_t *vm, } while (n != 0); } - vm->scopes[NJS_SCOPE_CALLEE_ARGUMENTS] = value; - if (args != NULL) { memcpy(value, args, nargs * sizeof(njs_value_t)); } @@ -430,9 +399,9 @@ njs_function_lambda_frame(njs_vm_t *vm, const njs_value_t *this, const njs_value_t *args, njs_uint_t nargs, njs_bool_t ctor) { - size_t size; - njs_uint_t n, max_args, closures; - njs_value_t *value, *bound; + size_t n, frame_size; + uint32_t args_count, value_count, value_size, temp_size; + njs_value_t *value, *bound, **new, **temp; njs_frame_t *frame; njs_function_t *target; njs_native_frame_t *native_frame; @@ -461,76 +430,86 @@ njs_function_lambda_frame(njs_vm_t *vm, lambda = target->u.lambda; } - max_args = njs_max(nargs, lambda->nargs); - - closures = lambda->nesting + lambda->block_closures; + args_count = function->args_offset + njs_max(nargs, lambda->nargs); + value_count = args_count + njs_max(args_count, lambda->nlocal); - size = njs_frame_size(closures) - + (function->args_offset + max_args) * sizeof(njs_value_t) - + lambda->local_size; + value_size = value_count * sizeof(njs_value_t *); + temp_size = lambda->temp * sizeof(njs_value_t *); - native_frame = njs_function_frame_alloc(vm, size); + frame_size = value_size + temp_size + + ((value_count + lambda->temp) * sizeof(njs_value_t)); + + native_frame = njs_function_frame_alloc(vm, NJS_FRAME_SIZE + frame_size); if (njs_slow_path(native_frame == NULL)) { return NJS_ERROR; } + /* Local */ + + new = (njs_value_t **) ((u_char *) native_frame + NJS_FRAME_SIZE); + value = (njs_value_t *) ((u_char *) new + value_size + temp_size); + + n = value_count + lambda->temp; + + while (n != 0) { + n--; + new[n] = &value[n]; + njs_set_invalid(new[n]); + } + + /* Temp */ + + temp = (njs_value_t **) ((u_char *) native_frame + NJS_FRAME_SIZE + + value_size); + + native_frame->arguments = value; + native_frame->arguments_offset = value + (function->args_offset - 1); + native_frame->local = new + args_count; + native_frame->temp = temp; native_frame->function = target; native_frame->nargs = nargs; native_frame->ctor = ctor; native_frame->native = 0; native_frame->pc = NULL; - /* Function arguments. */ - - value = (njs_value_t *) ((u_char *) native_frame + - njs_frame_size(closures)); - native_frame->arguments = value; - - if (bound == NULL) { - *value = *this; + /* Set this and bound arguments. */ + *native_frame->local[0] = *this; - if (njs_slow_path(function->global_this - && njs_is_null_or_undefined(this))) { - njs_set_object(value, &vm->global_object); - } + if (njs_slow_path(function->global_this + && njs_is_null_or_undefined(this))) + { + njs_set_object(native_frame->local[0], &vm->global_object); + } - value++; - - } else { + if (bound != NULL) { n = function->args_offset; native_frame->nargs += n - 1; - if (ctor) { - *value++ = *this; - bound++; - n--; + if (!ctor) { + *native_frame->local[0] = *bound; } + bound++; + n--; + while (n != 0) { *value++ = *bound++; n--; }; } - vm->scopes[NJS_SCOPE_CALLEE_ARGUMENTS] = value; + /* Copy arguments. */ if (args != NULL) { while (nargs != 0) { *value++ = *args++; - max_args--; nargs--; } } - while (max_args != 0) { - njs_set_undefined(value++); - max_args--; - } - frame = (njs_frame_t *) native_frame; frame->exception.catch = NULL; frame->exception.next = NULL; - frame->local = value; frame->previous_active_frame = vm->active_frame; return NJS_OK; @@ -543,15 +522,7 @@ njs_function_frame_alloc(njs_vm_t *vm, s size_t spare_size, chunk_size; njs_native_frame_t *frame; - /* - * The size value must be aligned to njs_value_t because vm->top_frame - * may point to frame->free and vm->top_frame is used as a base pointer - * in njs_vm_continuation() which is expected to return pointers aligned - * to njs_value_t. - */ - size = njs_align_size(size, sizeof(njs_value_t)); - - spare_size = vm->top_frame->free_size; + spare_size = vm->top_frame ? vm->top_frame->free_size : 0; if (njs_fast_path(size <= spare_size)) { frame = (njs_native_frame_t *) vm->top_frame->free; @@ -602,7 +573,7 @@ njs_function_call2(njs_vm_t *vm, njs_fun return ret; } - ret = njs_function_frame_invoke(vm, (njs_index_t) &dst); + ret = njs_function_frame_invoke(vm, &dst); if (ret == NJS_OK) { *retval = dst; @@ -615,78 +586,50 @@ njs_function_call2(njs_vm_t *vm, njs_fun njs_int_t njs_function_lambda_call(njs_vm_t *vm) { - size_t size; + uint32_t n; njs_int_t ret; - njs_uint_t n, nesting; njs_frame_t *frame; - njs_value_t *dst, *src; - njs_closure_t *closure, **closures; + njs_value_t *args, **local, *value; + njs_value_t **cur_local, **cur_closures, **cur_temp; njs_function_t *function; njs_function_lambda_t *lambda; frame = (njs_frame_t *) vm->top_frame; function = frame->native.function; + if (function->global && !function->closure_copied) { + ret = njs_function_capture_global_closures(vm, function); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + } + lambda = function->u.lambda; -#if (NJS_DEBUG) - vm->scopes[NJS_SCOPE_CALLEE_ARGUMENTS] = NULL; -#endif - - vm->scopes[NJS_SCOPE_ARGUMENTS] = frame->native.arguments; + args = vm->top_frame->arguments; + local = vm->top_frame->local + function->args_offset; - /* Function local variables and temporary values. */ - - vm->scopes[NJS_SCOPE_LOCAL] = frame->local; - - memcpy(frame->local, lambda->local_scope, lambda->local_size); - - /* Parent closures values. */ + /* Move all arguments. */ - n = 0; - nesting = lambda->nesting; + for (n = 0; n < function->args_count; n++) { + if (!njs_is_valid(args)) { + njs_set_undefined(args); + } - if (nesting != 0) { - closures = njs_function_active_closures(vm, function); - do { - closure = *closures++; - - njs_frame_closures(frame)[n] = closure; - vm->scopes[NJS_SCOPE_CLOSURE + n] = &closure->u.values; - - n++; - } while (n < nesting); + *local++ = args++; } - /* Function closure values. */ - - if (lambda->block_closures > 0) { - closure = NULL; - - size = lambda->closure_size; + /* Store current level. */ - if (size != 0) { - closure = njs_mp_align(vm->mem_pool, sizeof(njs_value_t), size); - if (njs_slow_path(closure == NULL)) { - njs_memory_error(vm); - return NJS_ERROR; - } + cur_local = vm->levels[NJS_LEVEL_LOCAL]; + cur_closures = vm->levels[NJS_LEVEL_CLOSURE]; + cur_temp = vm->levels[NJS_LEVEL_TEMP]; - size -= sizeof(njs_value_t); - closure->u.count = 0; - dst = closure->values; - - src = lambda->closure_scope; + /* Replace current level. */ - do { - *dst++ = *src++; - size -= sizeof(njs_value_t); - } while (size != 0); - } - - njs_frame_closures(frame)[n] = closure; - vm->scopes[NJS_SCOPE_CLOSURE + n] = &closure->u.values; - } + vm->levels[NJS_LEVEL_LOCAL] = vm->top_frame->local; + vm->levels[NJS_LEVEL_CLOSURE] = njs_function_closures(function); + vm->levels[NJS_LEVEL_TEMP] = frame->native.temp; if (lambda->rest_parameters) { ret = njs_function_rest_parameters_init(vm, &frame->native); @@ -695,9 +638,41 @@ njs_function_lambda_call(njs_vm_t *vm) } } + /* Self */ + + if (lambda->self != NJS_INDEX_NONE) { + value = njs_scope_value(vm, lambda->self); + + if (!njs_is_valid(value)) { + njs_set_function(value, function); + } + } + vm->active_frame = frame; - return njs_vmcode_interpreter(vm, lambda->start); + /* Closures */ + + n = lambda->ndeclarations; + + while (n != 0) { + n--; + + function = njs_function(lambda->declarations[n]); + + ret = njs_function_capture_closure(vm, function, function->u.lambda); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + } + + ret = njs_vmcode_interpreter(vm, lambda->start); + + /* Restore current level. */ + vm->levels[NJS_LEVEL_LOCAL] = cur_local; + vm->levels[NJS_LEVEL_CLOSURE] = cur_closures; + vm->levels[NJS_LEVEL_TEMP] = cur_temp; + + return ret; } @@ -705,7 +680,6 @@ njs_int_t njs_function_native_call(njs_vm_t *vm) { njs_int_t ret; - njs_value_t *value; njs_function_t *function, *target; njs_native_frame_t *native, *previous; njs_function_native_t call; @@ -741,12 +715,7 @@ njs_function_native_call(njs_vm_t *vm) njs_vm_scopes_restore(vm, native, previous); if (!native->skip) { - value = njs_vmcode_operand(vm, native->retval); - /* - * GC: value external/internal++ depending - * on vm->retval and retval type - */ - *value = vm->retval; + *native->retval = vm->retval; } njs_function_frame_free(vm, native); @@ -775,6 +744,142 @@ njs_function_frame_free(njs_vm_t *vm, nj } +njs_int_t +njs_function_capture_closure(njs_vm_t *vm, njs_function_t *function, + njs_function_lambda_t *lambda) +{ + void *start, *end; + uint32_t n; + njs_value_t *value, **closure; + njs_native_frame_t *frame; + + if (lambda->nclosures == 0) { + return NJS_OK; + } + + frame = &vm->active_frame->native; + + while (frame->native) { + frame = frame->previous; + } + + start = frame; + end = frame->free; + + closure = njs_function_closures(function); + n = lambda->nclosures; + + do { + n--; + + value = njs_scope_value(vm, lambda->closures[n]); + + if (start <= (void *) value && (void *) value < end) { + value = njs_scope_value_clone(vm, lambda->closures[n], value); + if (njs_slow_path(value == NULL)) { + return NJS_ERROR; + } + } + + closure[n] = value; + + } while (n != 0); + + return NJS_OK; +} + + +njs_inline njs_value_t * +njs_function_closure_value(njs_vm_t *vm, njs_value_t **scope, njs_index_t index, + void *start, void *end) +{ + njs_value_t *value, *newval; + + value = scope[njs_scope_index_value(index)]; + + if (start <= (void *) value && end > (void *) value) { + newval = njs_mp_alloc(vm->mem_pool, sizeof(njs_value_t)); + if (njs_slow_path(newval == NULL)) { + njs_memory_error(vm); + return NULL; + } + + *newval = *value; + value = newval; + } + + scope[njs_scope_index_value(index)] = value; + + return value; +} + + +njs_int_t +njs_function_capture_global_closures(njs_vm_t *vm, njs_function_t *function) +{ + void *start, *end; + uint32_t n; + njs_value_t *value, **refs, **global; + njs_index_t *indexes, index; + njs_native_frame_t *native; + njs_function_lambda_t *lambda; + + lambda = function->u.lambda; + + if (lambda->nclosures == 0) { + return NJS_OK; + } + + native = vm->top_frame; + + while (native->previous->function != NULL) { + native = native->previous; + } + + start = native; + end = native->free; + + indexes = lambda->closures; + refs = njs_function_closures(function); + + global = vm->levels[NJS_LEVEL_GLOBAL]; + + n = lambda->nclosures; + + while (n > 0) { + n--; + + index = indexes[n]; + + switch (njs_scope_index_type(index)) { + case NJS_LEVEL_LOCAL: + value = njs_function_closure_value(vm, native->local, index, + start, end); + break; + + case NJS_LEVEL_GLOBAL: + value = njs_function_closure_value(vm, global, index, start, end); + break; + + default: + njs_type_error(vm, "unexpected value type for closure \"%uD\"", + njs_scope_index_type(index)); + return NJS_ERROR; + } + + if (njs_slow_path(value == NULL)) { + return NJS_ERROR; + } + + refs[n] = value; + } + + function->closure_copied = 1; + + return NJS_OK; +} + + static njs_value_t * njs_function_property_prototype_set(njs_vm_t *vm, njs_lvlhsh_t *hash, njs_value_t *prototype) @@ -937,7 +1042,7 @@ njs_function_constructor(njs_vm_t *vm, n parser.lexer = &lexer; - ret = njs_parser(vm, &parser, NULL); + ret = njs_parser(vm, &parser); if (njs_slow_path(ret != NJS_OK)) { return ret; } @@ -953,7 +1058,14 @@ njs_function_constructor(njs_vm_t *vm, n type = &safe_ast[0]; for (; *type != NJS_TOKEN_ILLEGAL; type++, node = node->right) { - if (node == NULL || node->left != NULL) { + if (node == NULL) { + goto fail; + } + + if (node->left != NULL + && node->token_type != NJS_TOKEN_FUNCTION_EXPRESSION + && node->left->token_type != NJS_TOKEN_NAME) + { goto fail; } @@ -970,11 +1082,6 @@ njs_function_constructor(njs_vm_t *vm, n return ret; } - ret = njs_variables_scope_reference(vm, scope); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - njs_memzero(&generator, sizeof(njs_generator_t)); code = njs_generate_scope(vm, &generator, scope, &njs_entry_anonymous); @@ -990,11 +1097,12 @@ njs_function_constructor(njs_vm_t *vm, n lambda = ((njs_vmcode_function_t *) generator.code_start)->lambda; - function = njs_function_alloc(vm, lambda, NULL, 0); + function = njs_function_alloc(vm, lambda); if (njs_slow_path(function == NULL)) { return NJS_ERROR; } + function->global = 1; function->global_this = 1; function->args_count = lambda->nargs - lambda->rest_parameters; diff -r cb99db22e3e5 -r 0a2a0b5a74f4 src/njs_function.h --- a/src/njs_function.h Fri Apr 16 15:25:22 2021 +0000 +++ b/src/njs_function.h Fri Apr 30 16:02:08 2021 +0300 @@ -9,23 +9,21 @@ struct njs_function_lambda_s { - uint32_t nargs; - uint32_t local_size; - uint32_t closure_size; + njs_index_t *closures; + uint32_t nclosures; + uint32_t nlocal; + uint32_t temp; - /* Function nesting level. */ - uint8_t nesting; /* 4 bits */ + njs_value_t **declarations; + uint32_t ndeclarations; - /* Function internal block closures levels. */ - uint8_t block_closures; /* 4 bits */ + njs_index_t self; + + uint32_t nargs; uint8_t ctor; /* 1 bit */ uint8_t rest_parameters; /* 1 bit */ - /* Initial values of local scope. */ - njs_value_t *local_scope; - njs_value_t *closure_scope; - u_char *start; }; @@ -35,11 +33,10 @@ struct njs_function_lambda_s { njs_align_size(sizeof(njs_native_frame_t), sizeof(njs_value_t)) /* The frame size must be aligned to njs_value_t. */ -#define njs_frame_size(closures) \ - njs_align_size(sizeof(njs_frame_t) + closures * sizeof(njs_closure_t *), \ - sizeof(njs_value_t)) +#define NJS_FRAME_SIZE \ + njs_align_size(sizeof(njs_frame_t), sizeof(njs_value_t)) -#define NJS_FRAME_SPARE_SIZE 512 +#define NJS_FRAME_SPARE_SIZE (4 * 1024) struct njs_native_frame_s { @@ -51,11 +48,15 @@ struct njs_native_frame_s { njs_value_t *arguments; njs_object_t *arguments_object; - - njs_index_t retval; + njs_value_t *arguments_offset; + njs_value_t **local; + njs_value_t **temp; uint32_t size; uint32_t free_size; + + njs_value_t *retval; + uint32_t nargs; uint8_t native; /* 1 bit */ @@ -81,19 +82,14 @@ struct njs_frame_s { njs_exception_t exception; njs_frame_t *previous_active_frame; - - njs_value_t *local; - -#define njs_frame_closures(frame) \ - ((njs_closure_t **) ((u_char *) frame + sizeof(njs_frame_t))) }; -njs_function_t *njs_function_alloc(njs_vm_t *vm, njs_function_lambda_t *lambda, - njs_closure_t *closures[], njs_bool_t shared); +njs_function_t *njs_function_alloc(njs_vm_t *vm, njs_function_lambda_t *lambda); njs_function_t *njs_function_value_copy(njs_vm_t *vm, njs_value_t *value); njs_int_t njs_function_name_set(njs_vm_t *vm, njs_function_t *function, njs_value_t *name, const char *prefix); +njs_function_t *njs_function_copy(njs_vm_t *vm, njs_function_t *function); njs_int_t njs_function_arguments_object_init(njs_vm_t *vm, njs_native_frame_t *frame); njs_int_t njs_function_rest_parameters_init(njs_vm_t *vm, @@ -113,7 +109,12 @@ njs_int_t njs_function_call2(njs_vm_t *v njs_uint_t nargs, njs_value_t *retval, njs_bool_t ctor); njs_int_t njs_function_lambda_call(njs_vm_t *vm); njs_int_t njs_function_native_call(njs_vm_t *vm); +njs_native_frame_t *njs_function_frame_alloc(njs_vm_t *vm, size_t size); void njs_function_frame_free(njs_vm_t *vm, njs_native_frame_t *frame); +njs_int_t njs_function_capture_closure(njs_vm_t *vm, njs_function_t *function, + njs_function_lambda_t *lambda); +njs_int_t njs_function_capture_global_closures(njs_vm_t *vm, + njs_function_t *function); njs_inline njs_function_lambda_t * @@ -161,7 +162,7 @@ njs_function_previous_frame(njs_native_f njs_inline njs_int_t -njs_function_frame_invoke(njs_vm_t *vm, njs_index_t retval) +njs_function_frame_invoke(njs_vm_t *vm, njs_value_t *retval) { njs_native_frame_t *frame; @@ -202,6 +203,13 @@ njs_native_function_same(const njs_funct } +njs_inline njs_value_t ** +njs_function_closures(const njs_function_t *func) +{ + return (njs_value_t **) ((u_char *) func + sizeof(njs_function_t)); +} + + extern const njs_object_type_init_t njs_function_type_init; extern const njs_object_init_t njs_function_instance_init; extern const njs_object_init_t njs_arrow_instance_init; diff -r cb99db22e3e5 -r 0a2a0b5a74f4 src/njs_generator.c --- a/src/njs_generator.c Fri Apr 16 15:25:22 2021 +0000 +++ b/src/njs_generator.c Fri Apr 30 16:02:08 2021 +0300 @@ -2,6 +2,7 @@ /* * Copyright (C) Igor Sysoev * Copyright (C) Dmitry Volyntsev + * Copyright (C) Alexander Borisov * Copyright (C) NGINX, Inc. */ @@ -18,7 +19,7 @@ struct njs_generator_patch_s { * because pointer to u_char accesses only one byte so this does not * work on big endian platforms. */ - njs_jump_off_t jump_offset; + njs_jump_off_t jump_offset; njs_generator_patch_t *next; njs_str_t label; @@ -62,7 +63,8 @@ static njs_int_t njs_generate_code_map(n static njs_int_t njs_generate_name(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node); static njs_int_t njs_generate_variable(njs_vm_t *vm, njs_generator_t *generator, - njs_parser_node_t *node, njs_reference_type_t type); + njs_parser_node_t *node, njs_reference_type_t type, + njs_variable_t **retvar); static njs_int_t njs_generate_var_statement(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node); From alexander.borisov at nginx.com Fri Apr 30 13:04:42 2021 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Fri, 30 Apr 2021 13:04:42 +0000 Subject: [njs] Introduced reserved words in strict mode. Message-ID: details: https://hg.nginx.org/njs/rev/28f33155ad3f branches: changeset: 1630:28f33155ad3f user: Alexander Borisov date: Fri Apr 30 16:02:20 2021 +0300 description: Introduced reserved words in strict mode. diffstat: src/njs_lexer_tables.h | 16 ++++++++-------- src/njs_parser.c | 6 ++++++ src/test/njs_unit_test.c | 8 +++++++- utils/lexer_keyword.py | 16 ++++++++-------- 4 files changed, 29 insertions(+), 17 deletions(-) diffs (134 lines): diff -r 0a2a0b5a74f4 -r 28f33155ad3f src/njs_lexer_tables.h --- a/src/njs_lexer_tables.h Fri Apr 30 16:02:08 2021 +0300 +++ b/src/njs_lexer_tables.h Fri Apr 30 16:02:20 2021 +0300 @@ -159,7 +159,7 @@ static const njs_keyword_t njs_lexer_kws { .entry = { njs_str("implements") }, .type = NJS_TOKEN_IMPLEMENTS, - .reserved = 0 + .reserved = 1 }, { @@ -183,13 +183,13 @@ static const njs_keyword_t njs_lexer_kws { .entry = { njs_str("interface") }, .type = NJS_TOKEN_INTERFACE, - .reserved = 0 + .reserved = 1 }, { .entry = { njs_str("let") }, .type = NJS_TOKEN_LET, - .reserved = 0 + .reserved = 1 }, { @@ -219,25 +219,25 @@ static const njs_keyword_t njs_lexer_kws { .entry = { njs_str("package") }, .type = NJS_TOKEN_PACKAGE, - .reserved = 0 + .reserved = 1 }, { .entry = { njs_str("private") }, .type = NJS_TOKEN_PRIVATE, - .reserved = 0 + .reserved = 1 }, { .entry = { njs_str("protected") }, .type = NJS_TOKEN_PROTECTED, - .reserved = 0 + .reserved = 1 }, { .entry = { njs_str("public") }, .type = NJS_TOKEN_PUBLIC, - .reserved = 0 + .reserved = 1 }, { @@ -249,7 +249,7 @@ static const njs_keyword_t njs_lexer_kws { .entry = { njs_str("static") }, .type = NJS_TOKEN_STATIC, - .reserved = 0 + .reserved = 1 }, { diff -r 0a2a0b5a74f4 -r 28f33155ad3f src/njs_parser.c --- a/src/njs_parser.c Fri Apr 30 16:02:08 2021 +0300 +++ b/src/njs_parser.c Fri Apr 30 16:02:20 2021 +0300 @@ -4550,6 +4550,12 @@ njs_parser_declaration(njs_parser_t *par break; default: + if (njs_lexer_token_is_reserved(token)) { + njs_lexer_consume_token(parser->lexer, 1); + njs_parser_next(parser, njs_parser_failed_state); + return NJS_OK; + } + if (njs_lexer_token_is_binding_identifier(token)) { njs_parser_next(parser, njs_parser_lexical_declaration); break; diff -r 0a2a0b5a74f4 -r 28f33155ad3f src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Fri Apr 30 16:02:08 2021 +0300 +++ b/src/test/njs_unit_test.c Fri Apr 30 16:02:20 2021 +0300 @@ -9475,6 +9475,12 @@ static njs_unit_test_t njs_test[] = { njs_str("function f(){} typeof(f)"), njs_str("function") }, + { njs_str("function let() {}"), + njs_str("SyntaxError: Unexpected token \"let\" in 1") }, + + { njs_str("function static() {}"), + njs_str("SyntaxError: Unexpected token \"static\" in 1") }, + /* Recursive factorial. */ { njs_str("function f(a) {" @@ -18101,7 +18107,7 @@ static njs_unit_test_t njs_test[] = { njs_str("var let = 123;" "let"), - njs_str("123") }, + njs_str("SyntaxError: Unexpected token \"let\" in 1") }, { njs_str("var const = 123"), njs_str("SyntaxError: Unexpected token \"const\" in 1") }, diff -r 0a2a0b5a74f4 -r 28f33155ad3f utils/lexer_keyword.py --- a/utils/lexer_keyword.py Fri Apr 30 16:02:08 2021 +0300 +++ b/utils/lexer_keyword.py Fri Apr 30 16:02:20 2021 +0300 @@ -64,14 +64,14 @@ global_keywords = { "debugger": 1, "enum": 1, "extends": 1, - "implements": 0, - "interface": 0, - "let": 0, - "package": 0, - "private": 0, - "protected": 0, - "public": 0, - "static": 0, + "implements": 1, + "interface": 1, + "let": 1, + "package": 1, + "private": 1, + "protected": 1, + "public": 1, + "static": 1, "super": 1 } From alexander.borisov at nginx.com Fri Apr 30 13:04:45 2021 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Fri, 30 Apr 2021 13:04:45 +0000 Subject: [njs] Introduced let implementation. Message-ID: details: https://hg.nginx.org/njs/rev/27a4a52621d5 branches: changeset: 1631:27a4a52621d5 user: Alexander Borisov date: Fri Apr 30 16:02:28 2021 +0300 description: Introduced let implementation. This closes #105 issue on GitHub. diffstat: src/njs_builtin.c | 4 + src/njs_disassembler.c | 11 + src/njs_generator.c | 336 +++++++++++++++++++++++++++++++++++++++++++--- src/njs_module.c | 3 +- src/njs_parser.c | 224 +++++++++++++++++++++++++----- src/njs_parser.h | 5 + src/njs_scope.c | 6 +- src/njs_scope.h | 32 +++- src/njs_variable.c | 56 ++++++- src/njs_variable.h | 3 +- src/njs_vmcode.c | 117 ++++++++++++--- src/njs_vmcode.h | 12 + src/test/njs_benchmark.c | 11 + src/test/njs_unit_test.c | 237 +++++++++++++++++++++++++++++++++ 14 files changed, 942 insertions(+), 115 deletions(-) diffs (truncated from 1982 to 1000 lines): diff -r 28f33155ad3f -r 27a4a52621d5 src/njs_builtin.c --- a/src/njs_builtin.c Fri Apr 30 16:02:20 2021 +0300 +++ b/src/njs_builtin.c Fri Apr 30 16:02:28 2021 +0300 @@ -972,6 +972,10 @@ njs_global_this_prop_handler(njs_vm_t *v var = node->variable; + if (var->type == NJS_VARIABLE_LET) { + return NJS_DECLINED; + } + value = njs_scope_valid_value(vm, var->index); if (var->type == NJS_VARIABLE_FUNCTION && njs_is_undefined(value)) { diff -r 28f33155ad3f -r 27a4a52621d5 src/njs_disassembler.c --- a/src/njs_disassembler.c Fri Apr 30 16:02:20 2021 +0300 +++ b/src/njs_disassembler.c Fri Apr 30 16:02:28 2021 +0300 @@ -139,6 +139,17 @@ static njs_code_name_t code_names[] = { { NJS_VMCODE_THROW, sizeof(njs_vmcode_throw_t), njs_str("THROW ") }, + { NJS_VMCODE_LET, sizeof(njs_vmcode_variable_t), + njs_str("LET ") }, + + { NJS_VMCODE_LET_UPDATE, sizeof(njs_vmcode_variable_t), + njs_str("LET UPDATE ") }, + + { NJS_VMCODE_INITIALIZATION_TEST, sizeof(njs_vmcode_variable_t), + njs_str("INIT TEST ") }, + + { NJS_VMCODE_NOT_INITIALIZED, sizeof(njs_vmcode_variable_t), + njs_str("NOT INIT ") }, }; diff -r 28f33155ad3f -r 27a4a52621d5 src/njs_generator.c --- a/src/njs_generator.c Fri Apr 30 16:02:20 2021 +0300 +++ b/src/njs_generator.c Fri Apr 30 16:02:28 2021 +0300 @@ -67,6 +67,8 @@ static njs_int_t njs_generate_variable(n njs_variable_t **retvar); static njs_int_t njs_generate_var_statement(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node); +static njs_int_t njs_generate_let(njs_vm_t *vm, njs_generator_t *generator, + njs_parser_node_t *node, njs_variable_t *var); static njs_int_t njs_generate_if_statement(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node); static njs_int_t njs_generate_cond_expression(njs_vm_t *vm, @@ -79,6 +81,10 @@ static njs_int_t njs_generate_do_while_s njs_generator_t *generator, njs_parser_node_t *node); static njs_int_t njs_generate_for_statement(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node); +static njs_int_t njs_generate_for_let_update(njs_vm_t *vm, + njs_generator_t *generator, njs_parser_node_t *node, size_t depth); +static njs_int_t njs_generate_for_resolve_closure(njs_vm_t *vm, + njs_parser_node_t *node, size_t depth); static njs_int_t njs_generate_for_in_statement(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node); static njs_int_t njs_generate_start_block(njs_vm_t *vm, @@ -261,6 +267,9 @@ static njs_int_t njs_generate_reference_ ##__VA_ARGS__) +#define NJS_GENERATE_MAX_DEPTH 4096 + + static const njs_str_t no_label = njs_str(""); static const njs_str_t return_label = njs_str("@return"); /* GCC and Clang complain about NULL argument passed to memcmp(). */ @@ -277,6 +286,7 @@ njs_generate(njs_vm_t *vm, njs_generator switch (node->token_type) { case NJS_TOKEN_VAR: + case NJS_TOKEN_LET: return njs_generate_var_statement(vm, generator, node); case NJS_TOKEN_IF: @@ -479,7 +489,7 @@ njs_generator(njs_vm_t *vm, njs_generato { njs_int_t ret; - if (njs_slow_path(generator->count++ > 4096)) { + if (njs_slow_path(generator->count++ > NJS_GENERATE_MAX_DEPTH)) { njs_range_error(vm, "Maximum call stack size exceeded"); return NJS_ERROR; } @@ -492,6 +502,25 @@ njs_generator(njs_vm_t *vm, njs_generato } +static njs_int_t +njs_generate_wo_dest(njs_vm_t *vm, njs_generator_t *generator, + njs_parser_node_t *node) +{ + njs_int_t ret; + njs_parser_scope_t *scope; + + scope = njs_function_scope(node->scope); + + scope->dest_disable = 1; + + ret = njs_generator(vm, generator, node); + + scope->dest_disable = 0; + + return ret; +} + + static u_char * njs_generate_reserve(njs_vm_t *vm, njs_generator_t *generator, size_t size) { @@ -591,6 +620,8 @@ njs_generate_name(njs_vm_t *vm, njs_gene njs_parser_node_t *node) { njs_variable_t *var; + njs_parser_scope_t *scope; + njs_vmcode_variable_t *variable; njs_vmcode_function_copy_t *copy; var = njs_variable_reference(vm, node); @@ -605,6 +636,20 @@ njs_generate_name(njs_vm_t *vm, njs_gene copy->retval = node->index; } + if (var->init) { + return NJS_OK; + } + + if (var->type == NJS_VARIABLE_LET) { + scope = njs_function_scope(node->scope); + + if (scope->dest_disable) { + njs_generate_code(generator, njs_vmcode_variable_t, variable, + NJS_VMCODE_NOT_INITIALIZED, 1, node); + variable->dst = node->index; + } + } + return NJS_OK; } @@ -614,6 +659,8 @@ njs_generate_variable(njs_vm_t *vm, njs_ njs_parser_node_t *node, njs_reference_type_t type, njs_variable_t **retvar) { njs_variable_t *var; + njs_parser_scope_t *scope; + njs_vmcode_variable_t *variable; njs_vmcode_function_copy_t *copy; var = njs_variable_reference(vm, node); @@ -641,11 +688,44 @@ njs_generate_variable(njs_vm_t *vm, njs_ copy->retval = node->index; } + if (var->init) { + return NJS_OK; + } + + if (var->type == NJS_VARIABLE_LET) { + scope = njs_function_scope(node->scope); + + if ((!scope->dest_disable && njs_function_scope(var->scope) == scope)) { + njs_generate_code(generator, njs_vmcode_variable_t, variable, + NJS_VMCODE_NOT_INITIALIZED, 1, node); + variable->dst = node->index; + } + } + return NJS_OK; } static njs_int_t +njs_generate_variable_wo_dest(njs_vm_t *vm, njs_generator_t *generator, + njs_parser_node_t *node, njs_reference_type_t type, njs_variable_t **retvar) +{ + njs_int_t ret; + njs_parser_scope_t *scope; + + scope = njs_function_scope(node->scope); + + scope->dest_disable = 1; + + ret = njs_generate_variable(vm, generator, node, type, retvar); + + scope->dest_disable = 0; + + return ret; +} + + +static njs_int_t njs_generate_var_statement(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node) { @@ -656,26 +736,50 @@ njs_generate_var_statement(njs_vm_t *vm, lvalue = node->left; - ret = njs_generate_variable(vm, generator, lvalue, NJS_DECLARATION, &var); + ret = njs_generate_variable_wo_dest(vm, generator, lvalue, + NJS_DECLARATION, &var); if (njs_slow_path(ret != NJS_OK)) { return NJS_ERROR; } - lvalue->index = var->index; expr = node->right; if (expr == NULL) { /* Variable is only declared. */ + if (var->type == NJS_VARIABLE_LET) { + ret = njs_generate_let(vm, generator, node, var); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + } + + var->init = 1; + return NJS_OK; } - expr->dest = lvalue; - - ret = njs_generator(vm, generator, expr); - if (njs_slow_path(ret != NJS_OK)) { - return ret; + if (var->type == NJS_VARIABLE_LET) { + ret = njs_generate_wo_dest(vm, generator, expr); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + ret = njs_generate_let(vm, generator, node, var); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + } else { + expr->dest = lvalue; + + ret = njs_generator(vm, generator, expr); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } } + var->init = 1; + /* * lvalue and expression indexes are equal if the expression is an * empty object or expression result is stored directly in variable. @@ -693,6 +797,20 @@ njs_generate_var_statement(njs_vm_t *vm, static njs_int_t +njs_generate_let(njs_vm_t *vm, njs_generator_t *generator, + njs_parser_node_t *node, njs_variable_t *var) +{ + njs_vmcode_variable_t *code; + + njs_generate_code(generator, njs_vmcode_variable_t, code, + NJS_VMCODE_LET, 0, node); + code->dst = var->index; + + return NJS_OK; +} + + +static njs_int_t njs_generate_if_statement(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node) { @@ -1089,7 +1207,7 @@ njs_generate_for_statement(njs_vm_t *vm, { njs_int_t ret; njs_jump_off_t jump_offset, loop_offset; - njs_parser_node_t *condition, *update; + njs_parser_node_t *condition, *update, *init; njs_vmcode_jump_t *jump; njs_vmcode_cond_jump_t *cond_jump; @@ -1113,9 +1231,20 @@ njs_generate_for_statement(njs_vm_t *vm, return ret; } + init = node->left; node = node->right; condition = node->left; + /* + * Closures can occur in conditional and loop updates. This must be + * foreseen in order to generate optimized code for let updates. + */ + + ret = njs_generate_for_resolve_closure(vm, condition, generator->count); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + /* GCC complains about uninitialized jump_offset. */ jump_offset = 0; @@ -1142,10 +1271,20 @@ njs_generate_for_statement(njs_vm_t *vm, /* The loop update. */ + update = node->right; + + ret = njs_generate_for_resolve_closure(vm, update, generator->count); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + ret = njs_generate_for_let_update(vm, generator, init, generator->count); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + njs_generate_patch_block(vm, generator, generator->block->continuation); - update = node->right; - ret = njs_generator(vm, generator, update); if (njs_slow_path(ret != NJS_OK)) { return ret; @@ -1186,13 +1325,95 @@ njs_generate_for_statement(njs_vm_t *vm, static njs_int_t +njs_generate_for_let_update(njs_vm_t *vm, njs_generator_t *generator, + njs_parser_node_t *node, size_t depth) +{ + njs_parser_node_t *let; + njs_vmcode_variable_t *code_var; + njs_variable_reference_t *ref; + + if (node == NULL) { + return NJS_OK; + } + + if (depth >= NJS_GENERATE_MAX_DEPTH) { + return NJS_ERROR; + } + + if (node->token_type != NJS_TOKEN_STATEMENT) { + return NJS_OK; + } + + let = node->right; + + if (let->token_type != NJS_TOKEN_LET) { + return NJS_OK; + } + + ref = &let->left->u.reference; + + if (ref->variable->closure) { + njs_generate_code(generator, njs_vmcode_variable_t, code_var, + NJS_VMCODE_LET_UPDATE, 0, let); + code_var->dst = let->left->index; + } + + return njs_generate_for_let_update(vm, generator, node->left, depth + 1); +} + + +static njs_int_t +njs_generate_for_resolve_closure(njs_vm_t *vm, njs_parser_node_t *node, + size_t depth) +{ + njs_int_t ret; + njs_bool_t closure; + njs_variable_t *var; + + if (node == NULL) { + return NJS_OK; + } + + if (node->token_type == NJS_TOKEN_NAME) { + var = njs_variable_resolve(vm, node); + + if (njs_fast_path(var != NULL)) { + closure = njs_variable_closure_test(node->scope, var->scope); + + if (closure) { + var->closure = 1; + } + } + } + + if (depth >= NJS_GENERATE_MAX_DEPTH) { + njs_range_error(vm, "Maximum call stack size exceeded"); + return NJS_ERROR; + } + + ret = njs_generate_for_resolve_closure(vm, node->left, depth + 1); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + ret = njs_generate_for_resolve_closure(vm, node->right, depth + 1); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + return NJS_OK; +} + + +static njs_int_t njs_generate_for_in_statement(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node) { njs_int_t ret; njs_index_t index; + njs_variable_t *var; njs_jump_off_t loop_offset, prop_offset; - njs_parser_node_t *foreach; + njs_parser_node_t *foreach, *name; njs_vmcode_prop_next_t *prop_next; njs_vmcode_prop_foreach_t *prop_foreach; @@ -1205,15 +1426,36 @@ njs_generate_for_in_statement(njs_vm_t * /* The object. */ foreach = node->left; - - ret = njs_generator(vm, generator, foreach->left); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - ret = njs_generator(vm, generator, foreach->right); - if (njs_slow_path(ret != NJS_OK)) { - return ret; + name = foreach->left->right; + + if (name != NULL) { + name = name->left; + + ret = njs_generate_variable_wo_dest(vm, generator, name, + NJS_DECLARATION, &var); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + + foreach->left->index = name->index; + + ret = njs_generator(vm, generator, foreach->right); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + var->init = 1; + + } else { + ret = njs_generator(vm, generator, foreach->left); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + ret = njs_generator(vm, generator, foreach->right); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } } njs_generate_code(generator, njs_vmcode_prop_foreach_t, prop_foreach, @@ -1239,6 +1481,14 @@ njs_generate_for_in_statement(njs_vm_t * /* The loop iterator. */ + if (name != NULL) { + ret = njs_generate_for_let_update(vm, generator, foreach->left, + generator->count); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + } + njs_generate_patch_block(vm, generator, generator->block->continuation); njs_code_set_jump_offset(generator, njs_vmcode_prop_foreach_t, prop_offset); @@ -1564,12 +1814,38 @@ static njs_int_t njs_generate_statement(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node) { - njs_int_t ret; + njs_int_t ret; + njs_variable_t *var; + njs_parser_node_t *right; + njs_vmcode_variable_t *code; + + right = node->right; + + if (right != NULL && right->token_type == NJS_TOKEN_NAME) { + var = njs_variable_reference(vm, right); + if (njs_slow_path(var == NULL)) { + goto statement; + } + + if (!var->init && var->type == NJS_VARIABLE_LET) { + njs_generate_code(generator, njs_vmcode_variable_t, code, + NJS_VMCODE_INITIALIZATION_TEST, 0, right); + code->dst = right->index; + } + + if (node->left == NULL) { + return NJS_OK; + } + + node = node->left; + } + +statement: ret = njs_generate_children(vm, generator, node); if (njs_fast_path(ret == NJS_OK)) { - return njs_generate_node_index_release(vm, generator, node->right); + return njs_generate_node_index_release(vm, generator, right); } return ret; @@ -3311,8 +3587,9 @@ static njs_index_t njs_generate_dest_index(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node) { - njs_index_t ret; - njs_parser_node_t *dest; + njs_index_t ret; + njs_parser_node_t *dest; + njs_parser_scope_t *scope; ret = njs_generate_children_indexes_release(vm, generator, node); if (njs_slow_path(ret != NJS_OK)) { @@ -3322,7 +3599,11 @@ njs_generate_dest_index(njs_vm_t *vm, nj dest = node->dest; if (dest != NULL && dest->index != NJS_INDEX_NONE) { - return dest->index; + scope = njs_function_scope(node->scope); + + if (!scope->dest_disable) { + return dest->index; + } } return njs_generate_node_temp_index_get(vm, generator, node); @@ -3386,7 +3667,8 @@ njs_generate_temp_index_get(njs_vm_t *vm return NJS_ERROR; } - return njs_scope_index(scope->type, scope->temp++, NJS_LEVEL_TEMP); + return njs_scope_index(scope->type, scope->temp++, NJS_LEVEL_TEMP, + NJS_VARIABLE_VAR); } diff -r 28f33155ad3f -r 27a4a52621d5 src/njs_module.c --- a/src/njs_module.c Fri Apr 30 16:02:20 2021 +0300 +++ b/src/njs_module.c Fri Apr 30 16:02:28 2021 +0300 @@ -574,7 +574,8 @@ njs_module_insert(njs_parser_t *parser, scope = njs_parser_global_scope(parser); vm = parser->vm; - module->index = njs_scope_index(scope->type, scope->items, NJS_LEVEL_LOCAL); + module->index = njs_scope_index(scope->type, scope->items, NJS_LEVEL_LOCAL, + NJS_VARIABLE_VAR); scope->items++; if (vm->modules == NULL) { diff -r 28f33155ad3f -r 27a4a52621d5 src/njs_parser.c --- a/src/njs_parser.c Fri Apr 30 16:02:20 2021 +0300 +++ b/src/njs_parser.c Fri Apr 30 16:02:28 2021 +0300 @@ -238,6 +238,8 @@ static njs_int_t njs_parser_statement_li static njs_int_t njs_parser_statement_list_item(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current); +static njs_int_t njs_parser_lexical_declaration(njs_parser_t *parser, + njs_lexer_token_t *token, njs_queue_link_t *current); static njs_int_t njs_parser_variable_statement(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current); static njs_int_t njs_parser_variable_declaration_list(njs_parser_t *parser, @@ -287,7 +289,8 @@ static njs_int_t njs_parser_iteration_st static njs_int_t njs_parser_iteration_statement_for_map(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current); static njs_int_t njs_parser_for_var_binding_or_var_list(njs_parser_t *parser, - njs_lexer_token_t *token, njs_queue_link_t *current); + njs_lexer_token_t *token, njs_queue_link_t *current, + njs_token_type_t token_type); static njs_int_t njs_parser_for_var_in_statement(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current); static njs_int_t njs_parser_for_var_in_statement_after(njs_parser_t *parser, @@ -311,6 +314,8 @@ static njs_int_t njs_parser_switch_state njs_lexer_token_t *token, njs_queue_link_t *current); static njs_int_t njs_parser_switch_block(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current); +static njs_int_t njs_parser_switch_block_after(njs_parser_t *parser, + njs_lexer_token_t *token, njs_queue_link_t *current); static njs_int_t njs_parser_switch_case(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current); static njs_int_t njs_parser_switch_case_wo_def(njs_parser_t *parser, @@ -362,6 +367,8 @@ static njs_int_t njs_parser_catch_after( njs_lexer_token_t *token, njs_queue_link_t *current); static njs_int_t njs_parser_catch_parenthesis(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current); +static njs_int_t njs_parser_catch_statement_open_brace(njs_parser_t *parser, + njs_lexer_token_t *token, njs_queue_link_t *current); static njs_int_t njs_parser_catch_finally(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current); @@ -506,8 +513,10 @@ njs_parser_reject(njs_parser_t *parser) njs_int_t njs_parser(njs_vm_t *vm, njs_parser_t *parser) { - njs_int_t ret; - njs_lexer_token_t *token; + njs_int_t ret; + njs_str_t str; + njs_lexer_token_t *token; + const njs_lexer_keyword_entry_t *keyword; parser->vm = vm; @@ -526,6 +535,16 @@ njs_parser(njs_vm_t *vm, njs_parser_t *p parser->ret = NJS_OK; } + /* Add this as first variable. */ + njs_string_get(&njs_string_undefined, &str); + + keyword = njs_lexer_keyword(str.start, str.length); + if (njs_slow_path(keyword == NULL)) { + return NJS_ERROR; + } + + parser->undefined_id = (uintptr_t) keyword->value; + njs_queue_init(&parser->stack); parser->target = NULL; @@ -645,7 +664,8 @@ njs_parser_scope_begin(njs_parser_t *par return NJS_ERROR; } - var->index = njs_scope_index(type, 0, NJS_LEVEL_LOCAL); + var->index = njs_scope_index(type, 0, NJS_LEVEL_LOCAL, + NJS_VARIABLE_VAR); } } @@ -737,14 +757,6 @@ njs_parser_class_declaration(njs_parser_ static njs_int_t -njs_parser_lexical_declaration(njs_parser_t *parser, njs_lexer_token_t *token, - njs_queue_link_t *current) -{ - return njs_parser_not_supported(parser, token); -} - - -static njs_int_t njs_parser_function_or_generator(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current) { @@ -890,12 +902,14 @@ njs_parser_expression_parenthesis(njs_pa static njs_int_t -njs_parser_set_line_state(njs_parser_t *parser, +njs_parser_iteration_statement_for_end(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current) { parser->node->token_line = (uint32_t) (uintptr_t) parser->target; parser->target = NULL; + njs_parser_scope_end(parser); + return njs_parser_stack_pop(parser); } @@ -4534,7 +4548,7 @@ njs_parser_declaration(njs_parser_t *par switch (token->type) { case NJS_TOKEN_CLASS: njs_parser_next(parser, njs_parser_class_declaration); - break; + return NJS_OK; case NJS_TOKEN_LET: case NJS_TOKEN_CONST: @@ -4570,7 +4584,8 @@ njs_parser_declaration(njs_parser_t *par return NJS_DECLINED; } - return NJS_OK; + return njs_parser_after(parser, current, parser->node, 1, + njs_parser_statement_after); } @@ -4733,12 +4748,32 @@ njs_parser_statement_list_item(njs_parse /* + * 13.3.1 Let and Const Declarations + */ +static njs_int_t +njs_parser_lexical_declaration(njs_parser_t *parser, njs_lexer_token_t *token, + njs_queue_link_t *current) +{ + parser->var_type = (token->type == NJS_TOKEN_LET) ? NJS_VARIABLE_LET + : NJS_VARIABLE_CONST; + + njs_lexer_consume_token(parser->lexer, 1); + + njs_parser_next(parser, njs_parser_variable_declaration_list); + + return njs_parser_after(parser, current, NULL, 1, njs_parser_semicolon); +} + + +/* * 13.3.2 Variable Statement */ static njs_int_t njs_parser_variable_statement(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current) { + parser->var_type = NJS_VARIABLE_VAR; + njs_parser_next(parser, njs_parser_variable_declaration_list); return njs_parser_after(parser, current, NULL, 1, njs_parser_semicolon); @@ -4789,6 +4824,7 @@ njs_parser_variable_declaration(njs_pars { njs_int_t ret; njs_variable_t *var; + njs_token_type_t type; njs_parser_node_t *name; ret = njs_parser_binding_pattern(parser, token, current); @@ -4807,14 +4843,14 @@ njs_parser_variable_declaration(njs_pars return NJS_DONE; } - name = njs_parser_variable_node(parser, token->unique_id, NJS_VARIABLE_VAR, + name = njs_parser_variable_node(parser, token->unique_id, parser->var_type, &var); if (name == NULL) { return NJS_ERROR; } if (var->self) { - var->type = NJS_VARIABLE_VAR; + var->type = parser->var_type; var->self = 0; } @@ -4829,7 +4865,21 @@ njs_parser_variable_declaration(njs_pars return NJS_ERROR; } - ret = njs_parser_initializer_assign(parser, NJS_TOKEN_VAR); + switch (parser->var_type) { + case NJS_VARIABLE_LET: + type = NJS_TOKEN_LET; + break; + + case NJS_VARIABLE_CONST: + type = NJS_TOKEN_CONST; + break; + + default: + type = NJS_TOKEN_VAR; + break; + } + + ret = njs_parser_initializer_assign(parser, type); if (ret != NJS_OK) { return ret; } @@ -4930,6 +4980,12 @@ njs_parser_expression_statement(njs_pars return NJS_ERROR; } + if (token->type == NJS_TOKEN_NAME) { + njs_parser_syntax_error(parser, "let declaration cannot appear " + "in a single-statement context"); + return NJS_DONE; + } + if (token->type == NJS_TOKEN_OPEN_BRACKET) { return njs_parser_failed(parser); } @@ -5172,14 +5228,21 @@ static njs_int_t njs_parser_iteration_statement_for(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current) { + njs_int_t ret; + if (token->type == NJS_TOKEN_OPEN_PARENTHESIS) { njs_lexer_consume_token(parser->lexer, 1); + ret = njs_parser_scope_begin(parser, NJS_SCOPE_BLOCK, 0); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + njs_parser_next(parser, njs_parser_iteration_statement_for_map); return njs_parser_after(parser, current, (void *) (uintptr_t) parser->line, 1, - njs_parser_set_line_state); + njs_parser_iteration_statement_for_end); } if (token->type == NJS_TOKEN_AWAIT) { @@ -5194,8 +5257,9 @@ static njs_int_t njs_parser_iteration_statement_for_map(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current) { - njs_int_t ret; - njs_str_t *text; + njs_int_t ret; + njs_str_t *text; + njs_token_type_t token_type; /* * "var" ";" ? ";" ? ")" @@ -5242,6 +5306,9 @@ njs_parser_iteration_statement_for_map(n return NJS_OK; case NJS_TOKEN_VAR: + case NJS_TOKEN_LET: + token_type = token->type; + token = njs_lexer_peek_token(parser->lexer, token, 0); if (token == NULL) { return NJS_ERROR; @@ -5249,7 +5316,8 @@ njs_parser_iteration_statement_for_map(n njs_lexer_consume_token(parser->lexer, 1); - ret = njs_parser_for_var_binding_or_var_list(parser, token, current); + ret = njs_parser_for_var_binding_or_var_list(parser, token, + current, token_type); if (ret != NJS_OK) { if (ret == NJS_DONE) { return NJS_OK; @@ -5260,7 +5328,6 @@ njs_parser_iteration_statement_for_map(n break; - case NJS_TOKEN_LET: case NJS_TOKEN_CONST: return njs_parser_not_supported(parser, token); @@ -5288,11 +5355,23 @@ njs_parser_iteration_statement_for_map(n static njs_int_t njs_parser_for_var_binding_or_var_list(njs_parser_t *parser, - njs_lexer_token_t *token, njs_queue_link_t *current) -{ - njs_int_t ret; - njs_lexer_token_t *next; - njs_parser_node_t *node, *var; + njs_lexer_token_t *token, njs_queue_link_t *current, + njs_token_type_t token_type) +{ + njs_int_t ret; + njs_lexer_token_t *next; + njs_parser_node_t *node, *var, *node_type, *statement; + njs_variable_type_t type; + + switch (token_type) { + case NJS_TOKEN_LET: + type = NJS_VARIABLE_LET; + break; + + default: + type = NJS_VARIABLE_VAR; + break; + } switch (token->type) { /* BindingPattern */ @@ -5318,18 +5397,33 @@ njs_parser_for_var_binding_or_var_list(n } if (next->type != NJS_TOKEN_IN) { + parser->var_type = type; + njs_parser_next(parser, njs_parser_variable_declaration_list); return NJS_OK; } + statement = njs_parser_node_new(parser, NJS_TOKEN_STATEMENT); + if (njs_slow_path(statement == NULL)) { + return NJS_ERROR; + } + + node_type = njs_parser_node_new(parser, token_type); + if (njs_slow_path(node_type == NULL)) { + return NJS_ERROR; + } + var = njs_parser_variable_node(parser, token->unique_id, - NJS_VARIABLE_VAR, NULL); + type, NULL); if (var == NULL) { return NJS_ERROR; } + node_type->token_line = token->line; var->token_line = token->line; + statement->right = node_type; + node_type->left = var; parser->node = NULL; node = njs_parser_node_new(parser, NJS_TOKEN_IN); @@ -5338,7 +5432,7 @@ njs_parser_for_var_binding_or_var_list(n } node->token_line = next->line; - node->left = var; + node->left = statement; njs_parser_next(parser, njs_parser_expression); @@ -5829,6 +5923,8 @@ static njs_int_t njs_parser_switch_block(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current) { + njs_int_t ret; + if (token->type != NJS_TOKEN_OPEN_BRACE) { return njs_parser_failed(parser); } @@ -5837,9 +5933,24 @@ njs_parser_switch_block(njs_parser_t *pa parser->target->left = parser->node; + ret = njs_parser_scope_begin(parser, NJS_SCOPE_BLOCK, 0); + if (ret != NJS_OK) { + return NJS_ERROR; + } + njs_parser_next(parser, njs_parser_switch_case); - return NJS_OK; + return njs_parser_after(parser, current, NULL, 1, + njs_parser_switch_block_after); +} + +static njs_int_t +njs_parser_switch_block_after(njs_parser_t *parser, njs_lexer_token_t *token, + njs_queue_link_t *current) +{ + njs_parser_scope_end(parser); + + return njs_parser_stack_pop(parser); } @@ -6310,8 +6421,6 @@ njs_parser_catch_after(njs_parser_t *par { njs_parser_node_t *node; - njs_parser_scope_end(parser); - parser->target->right->right = parser->node; if (token->type == NJS_TOKEN_FINALLY) { @@ -6355,7 +6464,7 @@ njs_parser_catch_parenthesis(njs_parser_ parser->target->right->right = parser->node; parser->node = NULL; - njs_parser_next(parser, njs_parser_block_statement_open_brace); + njs_parser_next(parser, njs_parser_catch_statement_open_brace); return njs_parser_after(parser, current, parser->target, 1, njs_parser_catch_after); @@ -6363,6 +6472,42 @@ njs_parser_catch_parenthesis(njs_parser_ static njs_int_t +njs_parser_catch_statement_open_brace(njs_parser_t *parser, + njs_lexer_token_t *token, njs_queue_link_t *current) +{ + void *target; + + if (token->type != NJS_TOKEN_OPEN_BRACE) { + return njs_parser_failed(parser); + } + From alexander.borisov at nginx.com Fri Apr 30 13:04:47 2021 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Fri, 30 Apr 2021 13:04:47 +0000 Subject: [njs] Introduced const implementation. Message-ID: details: https://hg.nginx.org/njs/rev/f58a06be33dc branches: changeset: 1632:f58a06be33dc user: Alexander Borisov date: Fri Apr 30 16:02:34 2021 +0300 description: Introduced const implementation. diffstat: src/njs_builtin.c | 2 +- src/njs_disassembler.c | 3 + src/njs_generator.c | 57 +++++++++++- src/njs_parser.c | 18 +++- src/njs_scope.h | 2 +- src/njs_variable.c | 3 +- src/njs_vmcode.c | 4 + src/njs_vmcode.h | 1 + src/test/njs_unit_test.c | 192 +++++++++++++++++++++++++++++++++++++++++++++++ 9 files changed, 266 insertions(+), 16 deletions(-) diffs (524 lines): diff -r 27a4a52621d5 -r f58a06be33dc src/njs_builtin.c --- a/src/njs_builtin.c Fri Apr 30 16:02:28 2021 +0300 +++ b/src/njs_builtin.c Fri Apr 30 16:02:34 2021 +0300 @@ -972,7 +972,7 @@ njs_global_this_prop_handler(njs_vm_t *v var = node->variable; - if (var->type == NJS_VARIABLE_LET) { + if (var->type == NJS_VARIABLE_LET || var->type == NJS_VARIABLE_CONST) { return NJS_DECLINED; } diff -r 27a4a52621d5 -r f58a06be33dc src/njs_disassembler.c --- a/src/njs_disassembler.c Fri Apr 30 16:02:28 2021 +0300 +++ b/src/njs_disassembler.c Fri Apr 30 16:02:34 2021 +0300 @@ -150,6 +150,9 @@ static njs_code_name_t code_names[] = { { NJS_VMCODE_NOT_INITIALIZED, sizeof(njs_vmcode_variable_t), njs_str("NOT INIT ") }, + + { NJS_VMCODE_ASSIGNMENT_ERROR, sizeof(njs_vmcode_variable_t), + njs_str("ASSIGNMENT ERROR") }, }; diff -r 27a4a52621d5 -r f58a06be33dc src/njs_generator.c --- a/src/njs_generator.c Fri Apr 30 16:02:28 2021 +0300 +++ b/src/njs_generator.c Fri Apr 30 16:02:34 2021 +0300 @@ -287,6 +287,7 @@ njs_generate(njs_vm_t *vm, njs_generator case NJS_TOKEN_VAR: case NJS_TOKEN_LET: + case NJS_TOKEN_CONST: return njs_generate_var_statement(vm, generator, node); case NJS_TOKEN_IF: @@ -640,7 +641,7 @@ njs_generate_name(njs_vm_t *vm, njs_gene return NJS_OK; } - if (var->type == NJS_VARIABLE_LET) { + if (var->type == NJS_VARIABLE_LET || var->type == NJS_VARIABLE_CONST) { scope = njs_function_scope(node->scope); if (scope->dest_disable) { @@ -692,7 +693,7 @@ njs_generate_variable(njs_vm_t *vm, njs_ return NJS_OK; } - if (var->type == NJS_VARIABLE_LET) { + if (var->type == NJS_VARIABLE_LET || var->type == NJS_VARIABLE_CONST) { scope = njs_function_scope(node->scope); if ((!scope->dest_disable && njs_function_scope(var->scope) == scope)) { @@ -746,6 +747,12 @@ njs_generate_var_statement(njs_vm_t *vm, if (expr == NULL) { /* Variable is only declared. */ + + if (var->type == NJS_VARIABLE_CONST) { + njs_syntax_error(vm, "missing initializer in const declaration"); + return NJS_ERROR; + } + if (var->type == NJS_VARIABLE_LET) { ret = njs_generate_let(vm, generator, node, var); if (njs_slow_path(ret != NJS_OK)) { @@ -758,7 +765,7 @@ njs_generate_var_statement(njs_vm_t *vm, return NJS_OK; } - if (var->type == NJS_VARIABLE_LET) { + if (var->type == NJS_VARIABLE_LET || var->type == NJS_VARIABLE_CONST) { ret = njs_generate_wo_dest(vm, generator, expr); if (njs_slow_path(ret != NJS_OK)) { return ret; @@ -1346,7 +1353,9 @@ njs_generate_for_let_update(njs_vm_t *vm let = node->right; - if (let->token_type != NJS_TOKEN_LET) { + if (let->token_type != NJS_TOKEN_LET + && let->token_type != NJS_TOKEN_CONST) + { return NJS_OK; } @@ -1827,7 +1836,9 @@ njs_generate_statement(njs_vm_t *vm, njs goto statement; } - if (!var->init && var->type == NJS_VARIABLE_LET) { + if (!var->init && (var->type == NJS_VARIABLE_LET + || var->type == NJS_VARIABLE_CONST)) + { njs_generate_code(generator, njs_vmcode_variable_t, code, NJS_VMCODE_INITIALIZATION_TEST, 0, right); code->dst = right->index; @@ -1950,8 +1961,10 @@ njs_generate_assignment(njs_vm_t *vm, nj { njs_int_t ret; njs_index_t index, src; + njs_variable_t *var; njs_parser_node_t *lvalue, *expr, *object, *property; njs_vmcode_move_t *move; + njs_vmcode_variable_t *var_code; njs_vmcode_prop_set_t *prop_set; lvalue = node->left; @@ -1961,11 +1974,19 @@ njs_generate_assignment(njs_vm_t *vm, nj if (lvalue->token_type == NJS_TOKEN_NAME) { ret = njs_generate_variable(vm, generator, lvalue, NJS_DECLARATION, - NULL); + &var); if (njs_slow_path(ret != NJS_OK)) { return ret; } + if (var != NULL && var->type == NJS_VARIABLE_CONST) { + njs_generate_code(generator, njs_vmcode_variable_t, var_code, + NJS_VMCODE_ASSIGNMENT_ERROR, 0, node); + var_code->dst = var->index; + + return NJS_OK; + } + expr->dest = lvalue; ret = njs_generator(vm, generator, expr); @@ -2075,9 +2096,11 @@ njs_generate_operation_assignment(njs_vm { njs_int_t ret; njs_index_t index, src; + njs_variable_t *var; njs_parser_node_t *lvalue, *expr, *object, *property; njs_vmcode_move_t *move; njs_vmcode_3addr_t *code; + njs_vmcode_variable_t *var_code; njs_vmcode_prop_get_t *prop_get; njs_vmcode_prop_set_t *prop_set; @@ -2086,11 +2109,19 @@ njs_generate_operation_assignment(njs_vm if (lvalue->token_type == NJS_TOKEN_NAME) { ret = njs_generate_variable(vm, generator, lvalue, NJS_DECLARATION, - NULL); + &var); if (njs_slow_path(ret != NJS_OK)) { return ret; } + if (var != NULL && var->type == NJS_VARIABLE_CONST) { + njs_generate_code(generator, njs_vmcode_variable_t, var_code, + NJS_VMCODE_ASSIGNMENT_ERROR, 0, node); + var_code->dst = var->index; + + return NJS_OK; + } + index = lvalue->index; expr = node->right; @@ -2616,8 +2647,10 @@ njs_generate_inc_dec_operation(njs_vm_t { njs_int_t ret; njs_index_t index, dest_index; + njs_variable_t *var; njs_parser_node_t *lvalue; njs_vmcode_3addr_t *code; + njs_vmcode_variable_t *var_code; njs_vmcode_prop_get_t *prop_get; njs_vmcode_prop_set_t *prop_set; @@ -2626,11 +2659,19 @@ njs_generate_inc_dec_operation(njs_vm_t if (lvalue->token_type == NJS_TOKEN_NAME) { ret = njs_generate_variable(vm, generator, lvalue, NJS_DECLARATION, - NULL); + &var); if (njs_slow_path(ret != NJS_OK)) { return ret; } + if (var != NULL && var->type == NJS_VARIABLE_CONST) { + njs_generate_code(generator, njs_vmcode_variable_t, var_code, + NJS_VMCODE_ASSIGNMENT_ERROR, 0, node); + var_code->dst = var->index; + + return NJS_OK; + } + index = njs_generate_dest_index(vm, generator, node); if (njs_slow_path(index == NJS_INDEX_ERROR)) { return index; diff -r 27a4a52621d5 -r f58a06be33dc src/njs_parser.c --- a/src/njs_parser.c Fri Apr 30 16:02:28 2021 +0300 +++ b/src/njs_parser.c Fri Apr 30 16:02:34 2021 +0300 @@ -4946,6 +4946,7 @@ static njs_int_t njs_parser_expression_statement(njs_parser_t *parser, njs_lexer_token_t *token, njs_queue_link_t *current) { + njs_token_type_t type; njs_lexer_token_t *next; switch (token->type) { @@ -4974,15 +4975,20 @@ njs_parser_expression_statement(njs_pars break; + case NJS_TOKEN_CONST: case NJS_TOKEN_LET: + type = token->type; + token = njs_lexer_peek_token(parser->lexer, token, 0); if (token == NULL) { return NJS_ERROR; } if (token->type == NJS_TOKEN_NAME) { - njs_parser_syntax_error(parser, "let declaration cannot appear " - "in a single-statement context"); + njs_parser_syntax_error(parser, "%s declaration cannot appear " + "in a single-statement context", + (type == NJS_TOKEN_CONST ? "const" + : "let" )); return NJS_DONE; } @@ -5307,6 +5313,7 @@ njs_parser_iteration_statement_for_map(n case NJS_TOKEN_VAR: case NJS_TOKEN_LET: + case NJS_TOKEN_CONST: token_type = token->type; token = njs_lexer_peek_token(parser->lexer, token, 0); @@ -5328,9 +5335,6 @@ njs_parser_iteration_statement_for_map(n break; - case NJS_TOKEN_CONST: - return njs_parser_not_supported(parser, token); - default: njs_parser_next(parser, njs_parser_expression); break; @@ -5368,6 +5372,10 @@ njs_parser_for_var_binding_or_var_list(n type = NJS_VARIABLE_LET; break; + case NJS_TOKEN_CONST: + type = NJS_VARIABLE_CONST; + break; + default: type = NJS_VARIABLE_VAR; break; diff -r 27a4a52621d5 -r f58a06be33dc src/njs_scope.h --- a/src/njs_scope.h Fri Apr 30 16:02:28 2021 +0300 +++ b/src/njs_scope.h Fri Apr 30 16:02:34 2021 +0300 @@ -82,7 +82,7 @@ njs_scope_valid_value(njs_vm_t *vm, njs_ value = njs_scope_value(vm, index); if (!njs_is_valid(value)) { - if (njs_scope_index_var(index) == NJS_VARIABLE_LET) { + if (njs_scope_index_var(index) <= NJS_VARIABLE_LET) { njs_reference_error(vm, "cannot access to variable " "before initialization"); return NULL; diff -r 27a4a52621d5 -r f58a06be33dc src/njs_variable.c --- a/src/njs_variable.c Fri Apr 30 16:02:28 2021 +0300 +++ b/src/njs_variable.c Fri Apr 30 16:02:34 2021 +0300 @@ -184,6 +184,7 @@ njs_variable_scope_find(njs_parser_t *pa } switch (type) { + case NJS_VARIABLE_CONST: case NJS_VARIABLE_LET: if (scope->type == NJS_SCOPE_GLOBAL && parser->undefined_id == unique_id) @@ -222,7 +223,7 @@ njs_variable_scope_find(njs_parser_t *pa return root; } - if (var->type == NJS_VARIABLE_LET) { + if (var->type == NJS_VARIABLE_LET || var->type == NJS_VARIABLE_CONST) { goto failed; } diff -r 27a4a52621d5 -r f58a06be33dc src/njs_vmcode.c --- a/src/njs_vmcode.c Fri Apr 30 16:02:28 2021 +0300 +++ b/src/njs_vmcode.c Fri Apr 30 16:02:34 2021 +0300 @@ -938,6 +938,10 @@ next: njs_vmcode_error(vm, pc); goto error; + case NJS_VMCODE_ASSIGNMENT_ERROR: + njs_type_error(vm, "assignment to constant variable"); + goto error; + default: njs_internal_error(vm, "%d has NO retval", op); goto error; diff -r 27a4a52621d5 -r f58a06be33dc src/njs_vmcode.h --- a/src/njs_vmcode.h Fri Apr 30 16:02:28 2021 +0300 +++ b/src/njs_vmcode.h Fri Apr 30 16:02:34 2021 +0300 @@ -62,6 +62,7 @@ enum { NJS_VMCODE_LET_UPDATE, NJS_VMCODE_INITIALIZATION_TEST, NJS_VMCODE_NOT_INITIALIZED, + NJS_VMCODE_ASSIGNMENT_ERROR, NJS_VMCODE_ERROR, diff -r 27a4a52621d5 -r f58a06be33dc src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Fri Apr 30 16:02:28 2021 +0300 +++ b/src/test/njs_unit_test.c Fri Apr 30 16:02:34 2021 +0300 @@ -9481,6 +9481,27 @@ static njs_unit_test_t njs_test[] = { njs_str("function static() {}"), njs_str("SyntaxError: Unexpected token \"static\" in 1") }, + { njs_str("var arr = [];" + "function fn(one) {" + " var x = one + 1;" + " let y = one + 2;" + " const u = one + 4;" + " {" + " {" + " let z = one + 3;" + " const v = one + 5;" + " function f() {" + " arr.push(one); arr.push(x);" + " arr.push(y); arr.push(z);" + " arr.push(u); arr.push(v);" + " }" + " f();" + " }" + " }" + "}" + "fn(1); arr"), + njs_str("1,2,3,4,5,6") }, + /* Recursive factorial. */ { njs_str("function f(a) {" @@ -20009,6 +20030,177 @@ static njs_unit_test_t njs_test[] = { njs_str("function static() {}"), njs_str("SyntaxError: Unexpected token \"static\" in 1") }, + + /* const */ + + { njs_str("const x"), + njs_str("SyntaxError: missing initializer in const declaration") }, + + { njs_str("const x = 1; x"), + njs_str("1") }, + + { njs_str("const x = 1; x = 1"), + njs_str("TypeError: assignment to constant variable") }, + + { njs_str("function abc() {const x}"), + njs_str("SyntaxError: missing initializer in const declaration") }, + + { njs_str("const x = [123]; x"), + njs_str("123") }, + + { njs_str("const x = () => x; x()"), + njs_str("[object Function]") }, + + { njs_str("const x = (() => x)()"), + njs_str("ReferenceError: cannot access to variable before initialization") }, + + { njs_str("x; const x = 123"), + njs_str("ReferenceError: cannot access to variable before initialization") }, + + { njs_str("const x = x + 123"), + njs_str("ReferenceError: cannot access to variable before initialization") }, + + { njs_str("const x; var x"), + njs_str("SyntaxError: \"x\" has already been declared in 1") }, + + { njs_str("const x; let x"), + njs_str("SyntaxError: \"x\" has already been declared in 1") }, + + { njs_str("let x; const x"), + njs_str("SyntaxError: \"x\" has already been declared in 1") }, + + { njs_str("const x = 1; function x() {}"), + njs_str("SyntaxError: \"x\" has already been declared in 1") }, + + { njs_str("function x() {} const x = 1"), + njs_str("SyntaxError: \"x\" has already been declared in 1") }, + + { njs_str("function x() {const x; var x}"), + njs_str("SyntaxError: \"x\" has already been declared in 1") }, + + { njs_str("function x() {var x; const x}"), + njs_str("SyntaxError: \"x\" has already been declared in 1") }, + + { njs_str("const x = function f() {const f = 1}"), + njs_str("undefined") }, + + { njs_str("let res; const x = 1;" + "{const x = 2; res = x}" + "[x, res]"), + njs_str("1,2") }, + + { njs_str("let res; const x = 1;" + "if (true) {const x = 2; res = x}" + "[x, res]"), + njs_str("1,2") }, + + { njs_str("function func() {return x}" + "const x = 123;" + "func()"), + njs_str("123") }, + + { njs_str("function func() {return x}" + "func();" + "const x = 123"), + njs_str("ReferenceError: cannot access to variable before initialization") }, + + { njs_str("function func() {return () => x}" + "const x = 123;" + "func()()"), + njs_str("123") }, + + { njs_str("function func() {return () => x++}" + "const x = 123;" + "func()()"), + njs_str("TypeError: assignment to constant variable") }, + + { njs_str("for (const i = 0; i < 1; i++) {}"), + njs_str("TypeError: assignment to constant variable") }, + + { njs_str("let res = [];" + "for (const n in [1,2,3]) {res.push(n)}" + "res"), + njs_str("0,1,2") }, + + { njs_str("let arr = [], res = [];" + "" + "for (const n in [1,2,3]) {" + " arr.push(() => n);" + "}" + "" + "for (let n in arr) {" + " res.push(arr[n]());" + "}" + "res"), + njs_str("0,1,2") }, + + { njs_str("let arr = [];" + "" + "for (const n in [1,2,3]) {" + " let n = 1;" + " arr.push(n);" + "}" + "arr"), + njs_str("1,1,1") }, + + { njs_str("for (const n in [1,2,3]) {" + " let n = n + 1;" + "}"), + njs_str("ReferenceError: cannot access to variable before initialization") }, + + { njs_str("for (const n in [1,2,3]) {}" + "n"), + njs_str("ReferenceError: \"n\" is not defined") }, + + { njs_str("for (const n in [1,n,3]) {}"), + njs_str("ReferenceError: cannot access to variable before initialization") }, + + { njs_str("(function() {" + "function f() {return x + 1}" + "function abc() {f()};" + "abc();" + "const x = 1;" + "}())"), + njs_str("ReferenceError: cannot access to variable before initialization") }, + + { njs_str("if (false) const x = 1"), + njs_str("SyntaxError: const declaration cannot appear in a single-statement context in 1") }, + + { njs_str("while (false) const x = 1"), + njs_str("SyntaxError: const declaration cannot appear in a single-statement context in 1") }, + + { njs_str("for (;;) const x = 1"), + njs_str("SyntaxError: const declaration cannot appear in a single-statement context in 1") }, + + { njs_str("try {} catch (e) {const e = 1}"), + njs_str("SyntaxError: \"e\" has already been declared in 1") }, + + { njs_str("let arr = []; const x = 2;" + "switch(true) {default: const x = 1; arr.push(x)}" + "arr.push(x); arr"), + njs_str("1,2") }, + + { njs_str("let res;" + "switch(true) {case true: const x = 1; default: x = 2; res = x} res"), + njs_str("TypeError: assignment to constant variable") }, + + { njs_str("const null"), + njs_str("SyntaxError: Unexpected token \"null\" in 1") }, + + { njs_str("const continue"), + njs_str("SyntaxError: Unexpected token \"continue\" in 1") }, + + { njs_str("const undefined"), + njs_str("SyntaxError: \"undefined\" has already been declared in 1") }, + + { njs_str("const a = 1; globalThis.a"), + njs_str("undefined") }, + + { njs_str("if (false) {x = 2} else {x = 1} const x = 0"), + njs_str("ReferenceError: cannot access to variable before initialization") }, + + { njs_str("const const"), + njs_str("SyntaxError: Unexpected token \"const\" in 1") }, };