From gaoyan09 at baidu.com Mon Aug 3 05:18:38 2020 From: gaoyan09 at baidu.com (Gao,Yan(ACG VCP)) Date: Mon, 3 Aug 2020 05:18:38 +0000 Subject: [quic] How nginx-quic migrate quic connection when reload/restart Message-ID: <734E2259-D445-4E8B-B907-6E97AF361F07@baidu.com> Hi, all Quic connection can be migrated by the ID. And nginx-quic load balancing by reuseport, which using a hash based on 4-tuple of the connection. How to migrate quic connection when reload/restart? We can change reuseport behavior by bpf which using a hash based on the quic ID, but it seems useless, because listen processes/pids doubles when nginx reload/restart, and hash result would change -------------- next part -------------- An HTML attachment was scrubbed... URL: From sangdeuk.kwon at quantil.com Tue Aug 4 04:43:40 2020 From: sangdeuk.kwon at quantil.com (Sangdeuk Kwon) Date: Tue, 4 Aug 2020 13:43:40 +0900 Subject: [PATCH] slice and gzip problem Message-ID: # HG changeset patch # User Sangdeuk Kwon # Date 1596515879 -32400 # Tue Aug 04 13:37:59 2020 +0900 # Node ID 6ed0cfe89f288587b6815333bab798d6aedc6137 # Parent da8d758aabebbba833405463aa5a785b4c324495 slice and gzip problem When slice is enabled, client request content with "Accept-Encoding: gzip" and "Range: bytes=2048-3071", nginx serves only partial gzipped content with "200 OK" instead of "206 Parital Content". This response is wrong because nginx reply "200 OK" even if it serves only partial content. Client thinks he receives whole content because of "200 OK". ngx_http_gzip_filter_module is higher priority than ngx_http_slice_filter_module. gzip_filter_module remove slice_filter_module's info when gzip is invoked. So, nginx reply "200 OK" with gzipped whole content. But, what if user request has "Range" header (only one slice case), nginx can't access the previous slice of user request's. It means nginx can't serve correct content if gzip_filter_module remove slice's info. nginx can serve correct content with "206 Partial Content" without gzip because nginx can't access whole content. If user reuqest has "Range" header (only one slice case), gzip_filter should be disabled instead of slice_filter. diff -r da8d758aabeb -r 6ed0cfe89f28 src/http/modules/ngx_http_gzip_filter_module.c --- a/src/http/modules/ngx_http_gzip_filter_module.c Mon Jul 27 16:02:15 2020 +0300 +++ b/src/http/modules/ngx_http_gzip_filter_module.c Tue Aug 04 13:37:59 2020 +0900 @@ -233,6 +233,8 @@ && r->headers_out.content_encoding->value.len) || (r->headers_out.content_length_n != -1 && r->headers_out.content_length_n < conf->min_length) + || (r->allow_ranges && r == r->main + && r->headers_out.content_offset > 0) || ngx_http_test_content_type(r, &conf->types) == NULL || r->header_only) { -------------- next part -------------- An HTML attachment was scrubbed... URL: From rohitm at chelsio.com Tue Aug 4 11:16:00 2020 From: rohitm at chelsio.com (rohit maheshwari) Date: Tue, 4 Aug 2020 16:46:00 +0530 Subject: [PATCH] Enable TCP offload support on tls connecitons In-Reply-To: <20200729141815.GF12747@mdounin.ru> References: <32c7a0088f6d259163bb.1596014046@redhouse-blr-asicdesigners-com> <20200729141815.GF12747@mdounin.ru> Message-ID: <7685dbd1-e5d1-f5b1-a0cb-2d8d41f52c8d@chelsio.com> On 29/07/20 7:48 PM, Maxim Dounin wrote: > Hello! > > On Wed, Jul 29, 2020 at 02:44:06PM +0530, Rohit Maheshwari wrote: > >> # HG changeset patch >> # User Rohit Maheshwari >> # Date 1595354862 -19800 >> # Tue Jul 21 23:37:42 2020 +0530 >> # Node ID 32c7a0088f6d259163bb2820db0b44d36659b333 >> # Parent 32a343635b50662979975e1204417bb1fc7e1b1f >> Enable TCP offload support on tls connecitons >> >> Linux provides feasibility to enable TOE BYPASS iff setsockopt >> of type TCP_ULP is called just after socket creation. After that >> only, driver can register its TCP callbacks and move to TCP >> listen. > For TLS connections, setsockopt(TCP_ULP, "tls") is expected to be > called by the SSL layer. You may want to elaborate more on why > you are trying to call it on all connections instead. The main reason of calling it here is, to override stack's TCP listen with TCP offloaded listen, and so that TOE supported HW will get aware of the incoming TCP connection open request, and establishes and maintains that connection. ??I agree, it should have been called by SSL layer, but, since the socket is opened here, in my understanding SSL layer won't be aware of this server socket ever. One other way is open socket in SSL layer, and it will take care of calling setsockopt internally. #if (NGX_LINUX && NGX_TOE) ???????????? s = BIO_socket(ls[i].sockaddr->sa_family, ls[i].type, ??????????????????????????? 0, 0); #else ???????????? s = ngx_socket(ls[i].sockaddr->sa_family, ls[i].type, 0); #endif But this increases nginx code complexity, and that is the reason I avoided above change. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Thu Aug 6 03:42:46 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 06 Aug 2020 03:42:46 +0000 Subject: [nginx] Added size check to ngx_http_alloc_large_header_buffer(). Message-ID: details: https://hg.nginx.org/nginx/rev/8253424d1aff branches: changeset: 7690:8253424d1aff user: Maxim Dounin date: Thu Aug 06 05:02:22 2020 +0300 description: Added size check to ngx_http_alloc_large_header_buffer(). This ensures that copying won't write more than the buffer size even if the buffer comes from hc->free and it is smaller than the large client header buffer size in the virtual host configuration. This might happen if size of large client header buffers is different in name-based virtual hosts, similarly to the problem with number of buffers fixed in 6926:e662cbf1b932. diffstat: src/http/ngx_http_request.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diffs (16 lines): diff -r da8d758aabeb -r 8253424d1aff src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c Mon Jul 27 16:02:15 2020 +0300 +++ b/src/http/ngx_http_request.c Thu Aug 06 05:02:22 2020 +0300 @@ -1647,6 +1647,12 @@ ngx_http_alloc_large_header_buffer(ngx_h ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http large header copy: %uz", r->header_in->pos - old); + if (r->header_in->pos - old > b->end - b->start) { + ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, + "too large header to copy"); + return NGX_ERROR; + } + new = b->start; ngx_memcpy(new, old, r->header_in->pos - old); From mdounin at mdounin.ru Thu Aug 6 03:42:49 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 06 Aug 2020 03:42:49 +0000 Subject: [nginx] Request body: all read data are now sent to filters. Message-ID: details: https://hg.nginx.org/nginx/rev/08ff2e10ae92 branches: changeset: 7691:08ff2e10ae92 user: Maxim Dounin date: Thu Aug 06 05:02:44 2020 +0300 description: Request body: all read data are now sent to filters. This is a prerequisite for the next change to allow large reads on chunk boundaries. diffstat: src/http/ngx_http_request_body.c | 53 +++++++-------------------------------- 1 files changed, 10 insertions(+), 43 deletions(-) diffs (82 lines): diff -r 8253424d1aff -r 08ff2e10ae92 src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c Thu Aug 06 05:02:22 2020 +0300 +++ b/src/http/ngx_http_request_body.c Thu Aug 06 05:02:44 2020 +0300 @@ -282,28 +282,12 @@ ngx_http_do_read_client_request_body(ngx for ( ;; ) { if (rb->buf->last == rb->buf->end) { - if (rb->buf->pos != rb->buf->last) { - - /* pass buffer to request body filter chain */ - - out.buf = rb->buf; - out.next = NULL; - - rc = ngx_http_request_body_filter(r, &out); + /* update chains */ - if (rc != NGX_OK) { - return rc; - } - - } else { + rc = ngx_http_request_body_filter(r, NULL); - /* update chains */ - - rc = ngx_http_request_body_filter(r, NULL); - - if (rc != NGX_OK) { - return rc; - } + if (rc != NGX_OK) { + return rc; } if (rb->busy != NULL) { @@ -355,17 +339,15 @@ ngx_http_do_read_client_request_body(ngx rb->buf->last += n; r->request_length += n; - if (n == rest) { - /* pass buffer to request body filter chain */ + /* pass buffer to request body filter chain */ - out.buf = rb->buf; - out.next = NULL; + out.buf = rb->buf; + out.next = NULL; - rc = ngx_http_request_body_filter(r, &out); + rc = ngx_http_request_body_filter(r, &out); - if (rc != NGX_OK) { - return rc; - } + if (rc != NGX_OK) { + return rc; } if (rb->rest == 0) { @@ -386,21 +368,6 @@ ngx_http_do_read_client_request_body(ngx if (!c->read->ready) { - if (r->request_body_no_buffering - && rb->buf->pos != rb->buf->last) - { - /* pass buffer to request body filter chain */ - - out.buf = rb->buf; - out.next = NULL; - - rc = ngx_http_request_body_filter(r, &out); - - if (rc != NGX_OK) { - return rc; - } - } - clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); ngx_add_timer(c->read, clcf->client_body_timeout); From mdounin at mdounin.ru Thu Aug 6 03:42:52 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 06 Aug 2020 03:42:52 +0000 Subject: [nginx] Request body: allowed large reads on chunk boundaries. Message-ID: details: https://hg.nginx.org/nginx/rev/0f7f1a509113 branches: changeset: 7692:0f7f1a509113 user: Maxim Dounin date: Thu Aug 06 05:02:55 2020 +0300 description: Request body: allowed large reads on chunk boundaries. If some additional data from a pipelined request happens to be read into the body buffer, we copy it to r->header_in or allocate an additional large client header buffer for it. diffstat: src/http/ngx_http_request_body.c | 121 ++++++++++++++++++++++++++++++++++++-- 1 files changed, 113 insertions(+), 8 deletions(-) diffs (194 lines): diff -r 08ff2e10ae92 -r 0f7f1a509113 src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c Thu Aug 06 05:02:44 2020 +0300 +++ b/src/http/ngx_http_request_body.c Thu Aug 06 05:02:55 2020 +0300 @@ -12,6 +12,8 @@ static void ngx_http_read_client_request_body_handler(ngx_http_request_t *r); static ngx_int_t ngx_http_do_read_client_request_body(ngx_http_request_t *r); +static ngx_int_t ngx_http_copy_pipelined_header(ngx_http_request_t *r, + ngx_buf_t *buf); static ngx_int_t ngx_http_write_request_body(ngx_http_request_t *r); static ngx_int_t ngx_http_read_discarded_request_body(ngx_http_request_t *r); static ngx_int_t ngx_http_discard_request_body_filter(ngx_http_request_t *r, @@ -379,6 +381,10 @@ ngx_http_do_read_client_request_body(ngx } } + if (ngx_http_copy_pipelined_header(r, rb->buf) != NGX_OK) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + if (c->read->timer_set) { ngx_del_timer(c->read); } @@ -393,6 +399,88 @@ ngx_http_do_read_client_request_body(ngx static ngx_int_t +ngx_http_copy_pipelined_header(ngx_http_request_t *r, ngx_buf_t *buf) +{ + size_t n; + ngx_buf_t *b; + ngx_chain_t *cl; + ngx_http_connection_t *hc; + ngx_http_core_srv_conf_t *cscf; + + b = r->header_in; + n = buf->last - buf->pos; + + if (buf == b || n == 0) { + return NGX_OK; + } + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http body pipelined header: %uz", n); + + /* + * if there is a pipelined request in the client body buffer, + * copy it to the r->header_in buffer if there is enough room, + * or allocate a large client header buffer + */ + + if (n > (size_t) (b->end - b->last)) { + + hc = r->http_connection; + + if (hc->free) { + cl = hc->free; + hc->free = cl->next; + + b = cl->buf; + + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http large header free: %p %uz", + b->pos, b->end - b->last); + + } else { + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + + b = ngx_create_temp_buf(r->connection->pool, + cscf->large_client_header_buffers.size); + if (b == NULL) { + return NGX_ERROR; + } + + cl = ngx_alloc_chain_link(r->connection->pool); + if (cl == NULL) { + return NGX_ERROR; + } + + cl->buf = b; + + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http large header alloc: %p %uz", + b->pos, b->end - b->last); + } + + cl->next = hc->busy; + hc->busy = cl; + hc->nbusy++; + + r->header_in = b; + + if (n > (size_t) (b->end - b->last)) { + ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, + "too large pipelined header after reading body"); + return NGX_ERROR; + } + } + + ngx_memcpy(b->last, buf->pos, n); + + b->last += n; + r->request_length -= n; + + return NGX_OK; +} + + +static ngx_int_t ngx_http_write_request_body(ngx_http_request_t *r) { ssize_t n; @@ -637,8 +725,7 @@ ngx_http_read_discarded_request_body(ngx for ( ;; ) { if (r->headers_in.content_length_n == 0) { - r->read_event_handler = ngx_http_block_reading; - return NGX_OK; + break; } if (!r->connection->read->ready) { @@ -672,15 +759,24 @@ ngx_http_read_discarded_request_body(ngx return rc; } } + + if (ngx_http_copy_pipelined_header(r, &b) != NGX_OK) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + r->read_event_handler = ngx_http_block_reading; + + return NGX_OK; } static ngx_int_t ngx_http_discard_request_body_filter(ngx_http_request_t *r, ngx_buf_t *b) { - size_t size; - ngx_int_t rc; - ngx_http_request_body_t *rb; + size_t size; + ngx_int_t rc; + ngx_http_request_body_t *rb; + ngx_http_core_srv_conf_t *cscf; if (r->headers_in.chunked) { @@ -735,7 +831,10 @@ ngx_http_discard_request_body_filter(ngx /* set amount of data we want to see next time */ - r->headers_in.content_length_n = rb->chunked->length; + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + + r->headers_in.content_length_n = ngx_max(rb->chunked->length, + (off_t) cscf->large_client_header_buffers.size); break; } @@ -903,6 +1002,7 @@ ngx_http_request_body_chunked_filter(ngx ngx_chain_t *cl, *out, *tl, **ll; ngx_http_request_body_t *rb; ngx_http_core_loc_conf_t *clcf; + ngx_http_core_srv_conf_t *cscf; rb = r->request_body; @@ -916,8 +1016,10 @@ ngx_http_request_body_chunked_filter(ngx return NGX_HTTP_INTERNAL_SERVER_ERROR; } + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + r->headers_in.content_length_n = 0; - rb->rest = 3; + rb->rest = cscf->large_client_header_buffers.size; } out = NULL; @@ -1024,7 +1126,10 @@ ngx_http_request_body_chunked_filter(ngx /* set rb->rest, amount of data we want to see next time */ - rb->rest = rb->chunked->length; + cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); + + rb->rest = ngx_max(rb->chunked->length, + (off_t) cscf->large_client_header_buffers.size); break; } From mdounin at mdounin.ru Thu Aug 6 03:42:55 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 06 Aug 2020 03:42:55 +0000 Subject: [nginx] Request body: optimized handling of small chunks. Message-ID: details: https://hg.nginx.org/nginx/rev/f5a2af0e7079 branches: changeset: 7693:f5a2af0e7079 user: Maxim Dounin date: Thu Aug 06 05:02:57 2020 +0300 description: Request body: optimized handling of small chunks. If there is a previous buffer, copy small chunks into it instead of allocating additional buffer. diffstat: src/http/ngx_http_request_body.c | 25 +++++++++++++++++++++++++ 1 files changed, 25 insertions(+), 0 deletions(-) diffs (42 lines): diff -r 0f7f1a509113 -r f5a2af0e7079 src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c Thu Aug 06 05:02:55 2020 +0300 +++ b/src/http/ngx_http_request_body.c Thu Aug 06 05:02:57 2020 +0300 @@ -1027,6 +1027,8 @@ ngx_http_request_body_chunked_filter(ngx for (cl = in; cl; cl = cl->next) { + b = NULL; + for ( ;; ) { ngx_log_debug7(NGX_LOG_DEBUG_EVENT, r->connection->log, 0, @@ -1061,6 +1063,29 @@ ngx_http_request_body_chunked_filter(ngx return NGX_HTTP_REQUEST_ENTITY_TOO_LARGE; } + if (b + && rb->chunked->size <= 128 + && cl->buf->last - cl->buf->pos >= rb->chunked->size) + { + r->headers_in.content_length_n += rb->chunked->size; + + if (rb->chunked->size < 8) { + + while (rb->chunked->size) { + *b->last++ = *cl->buf->pos++; + rb->chunked->size--; + } + + } else { + ngx_memmove(b->last, cl->buf->pos, rb->chunked->size); + b->last += rb->chunked->size; + cl->buf->pos += rb->chunked->size; + rb->chunked->size = 0; + } + + continue; + } + tl = ngx_chain_get_free_buf(r->pool, &rb->free); if (tl == NULL) { return NGX_HTTP_INTERNAL_SERVER_ERROR; From mdounin at mdounin.ru Thu Aug 6 13:25:53 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 6 Aug 2020 16:25:53 +0300 Subject: [PATCH] Enable TCP offload support on tls connecitons In-Reply-To: <7685dbd1-e5d1-f5b1-a0cb-2d8d41f52c8d@chelsio.com> References: <32c7a0088f6d259163bb.1596014046@redhouse-blr-asicdesigners-com> <20200729141815.GF12747@mdounin.ru> <7685dbd1-e5d1-f5b1-a0cb-2d8d41f52c8d@chelsio.com> Message-ID: <20200806132553.GA12747@mdounin.ru> Hello! On Tue, Aug 04, 2020 at 04:46:00PM +0530, rohit maheshwari wrote: > On 29/07/20 7:48 PM, Maxim Dounin wrote: > > Hello! > > > > On Wed, Jul 29, 2020 at 02:44:06PM +0530, Rohit Maheshwari wrote: > > > > > # HG changeset patch > > > # User Rohit Maheshwari > > > # Date 1595354862 -19800 > > > # Tue Jul 21 23:37:42 2020 +0530 > > > # Node ID 32c7a0088f6d259163bb2820db0b44d36659b333 > > > # Parent 32a343635b50662979975e1204417bb1fc7e1b1f > > > Enable TCP offload support on tls connecitons > > > > > > Linux provides feasibility to enable TOE BYPASS iff setsockopt > > > of type TCP_ULP is called just after socket creation. After that > > > only, driver can register its TCP callbacks and move to TCP > > > listen. > > For TLS connections, setsockopt(TCP_ULP, "tls") is expected to be > > called by the SSL layer. You may want to elaborate more on why > > you are trying to call it on all connections instead. > > The main reason of calling it here is, to override stack's TCP > listen with TCP offloaded listen, and so that TOE supported > HW will get aware of the incoming TCP connection open > request, and establishes and maintains that connection. > ??I agree, it should have been called by SSL layer, but, since > the socket is opened here, in my understanding SSL layer > won't be aware of this server socket ever. Shouldn't it be enough for the HW that the socket of a particular connection is properly marked with setsockopt() by the SSL layer? Either way, if the intention is to only touch listening sockets with SSL enabled, the patch is clearly wrong: it touches all listening sockets, not just SSL ones, as well as DNS client sockets which aren't listening and never use SSL. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Mon Aug 10 15:55:53 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 10 Aug 2020 15:55:53 +0000 Subject: [nginx] SSL: fixed shutdown handling. Message-ID: details: https://hg.nginx.org/nginx/rev/09fb2135a589 branches: changeset: 7694:09fb2135a589 user: Maxim Dounin date: Mon Aug 10 18:52:09 2020 +0300 description: SSL: fixed shutdown handling. Previously, bidirectional shutdown never worked, due to two issues in the code: 1. The code only tested SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE when there was an error in the error queue, which cannot happen. The bug was introduced in an attempt to fix unexpected error logging as reported with OpenSSL 0.9.8g (http://mailman.nginx.org/pipermail/nginx/2008-January/003084.html). 2. The code never called SSL_shutdown() for the second time to wait for the peer's close_notify alert. This change fixes both issues. Note that after this change bidirectional shutdown is expected to work for the first time, so c->ssl->no_wait_shutdown now makes a difference. This is not a problem for HTTP code which always uses c->ssl->no_wait_shutdown, but might be a problem for stream and mail code, as well as 3rd party modules. To minimize the effect of the change, the timeout, which was used to be 30 seconds and not configurable, though never actually used, is now set to 3 seconds. It is also expanded to apply to both SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE, so timeout is properly set if writing to the socket buffer is not possible. diffstat: src/event/ngx_event_openssl.c | 105 ++++++++++++++++++++++++----------------- 1 files changed, 61 insertions(+), 44 deletions(-) diffs (129 lines): diff -r f5a2af0e7079 -r 09fb2135a589 src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c Thu Aug 06 05:02:57 2020 +0300 +++ b/src/event/ngx_event_openssl.c Mon Aug 10 18:52:09 2020 +0300 @@ -2774,8 +2774,9 @@ ngx_ssl_free_buffer(ngx_connection_t *c) ngx_int_t ngx_ssl_shutdown(ngx_connection_t *c) { - int n, sslerr, mode; - ngx_err_t err; + int n, sslerr, mode; + ngx_err_t err; + ngx_uint_t tries; ngx_ssl_ocsp_cleanup(c); @@ -2816,55 +2817,71 @@ ngx_ssl_shutdown(ngx_connection_t *c) ngx_ssl_clear_error(c->log); - n = SSL_shutdown(c->ssl->connection); - - ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_shutdown: %d", n); - - sslerr = 0; - - /* before 0.9.8m SSL_shutdown() returned 0 instead of -1 on errors */ - - if (n != 1 && ERR_peek_error()) { + tries = 2; + + for ( ;; ) { + + /* + * For bidirectional shutdown, SSL_shutdown() needs to be called + * twice: first call sends the "close notify" alert and returns 0, + * second call waits for the peer's "close notify" alert. + */ + + n = SSL_shutdown(c->ssl->connection); + + ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_shutdown: %d", n); + + if (n == 1) { + SSL_free(c->ssl->connection); + c->ssl = NULL; + + return NGX_OK; + } + + if (n == 0 && tries-- > 1) { + continue; + } + + /* before 0.9.8m SSL_shutdown() returned 0 instead of -1 on errors */ + sslerr = SSL_get_error(c->ssl->connection, n); ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_get_error: %d", sslerr); - } - - if (n == 1 || sslerr == 0 || sslerr == SSL_ERROR_ZERO_RETURN) { + + if (sslerr == SSL_ERROR_WANT_READ || sslerr == SSL_ERROR_WANT_WRITE) { + c->read->handler = ngx_ssl_shutdown_handler; + c->write->handler = ngx_ssl_shutdown_handler; + + if (ngx_handle_read_event(c->read, 0) != NGX_OK) { + return NGX_ERROR; + } + + if (ngx_handle_write_event(c->write, 0) != NGX_OK) { + return NGX_ERROR; + } + + ngx_add_timer(c->read, 3000); + + return NGX_AGAIN; + } + + if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0) { + SSL_free(c->ssl->connection); + c->ssl = NULL; + + return NGX_OK; + } + + err = (sslerr == SSL_ERROR_SYSCALL) ? ngx_errno : 0; + + ngx_ssl_connection_error(c, sslerr, err, "SSL_shutdown() failed"); + SSL_free(c->ssl->connection); c->ssl = NULL; - return NGX_OK; - } - - if (sslerr == SSL_ERROR_WANT_READ || sslerr == SSL_ERROR_WANT_WRITE) { - c->read->handler = ngx_ssl_shutdown_handler; - c->write->handler = ngx_ssl_shutdown_handler; - - if (ngx_handle_read_event(c->read, 0) != NGX_OK) { - return NGX_ERROR; - } - - if (ngx_handle_write_event(c->write, 0) != NGX_OK) { - return NGX_ERROR; - } - - if (sslerr == SSL_ERROR_WANT_READ) { - ngx_add_timer(c->read, 30000); - } - - return NGX_AGAIN; - } - - err = (sslerr == SSL_ERROR_SYSCALL) ? ngx_errno : 0; - - ngx_ssl_connection_error(c, sslerr, err, "SSL_shutdown() failed"); - - SSL_free(c->ssl->connection); - c->ssl = NULL; - - return NGX_ERROR; + return NGX_ERROR; + } } From mdounin at mdounin.ru Mon Aug 10 15:55:56 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 10 Aug 2020 15:55:56 +0000 Subject: [nginx] HTTP/2: fixed c->timedout flag on timed out connections. Message-ID: details: https://hg.nginx.org/nginx/rev/d57f15922ca3 branches: changeset: 7695:d57f15922ca3 user: Maxim Dounin date: Mon Aug 10 18:52:20 2020 +0300 description: HTTP/2: fixed c->timedout flag on timed out connections. Without the flag, SSL shutdown is attempted on such connections, resulting in useless work and/or bogus "SSL_shutdown() failed (SSL: ... bad write retry)" critical log messages if there are blocked writes. diffstat: src/http/v2/ngx_http_v2.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 09fb2135a589 -r d57f15922ca3 src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c Mon Aug 10 18:52:09 2020 +0300 +++ b/src/http/v2/ngx_http_v2.c Mon Aug 10 18:52:20 2020 +0300 @@ -475,6 +475,7 @@ ngx_http_v2_write_handler(ngx_event_t *w ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http2 write event timed out"); c->error = 1; + c->timedout = 1; ngx_http_v2_finalize_connection(h2c, 0); return; } From mdounin at mdounin.ru Mon Aug 10 15:55:59 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 10 Aug 2020 15:55:59 +0000 Subject: [nginx] SSL: disabled sending shutdown after ngx_http_test_reading(). Message-ID: details: https://hg.nginx.org/nginx/rev/45764bca69b0 branches: changeset: 7696:45764bca69b0 user: Maxim Dounin date: Mon Aug 10 18:52:34 2020 +0300 description: SSL: disabled sending shutdown after ngx_http_test_reading(). Sending shutdown when ngx_http_test_reading() detects the connection is closed can result in "SSL_shutdown() failed (SSL: ... bad write retry)" critical log messages if there are blocked writes. Fix is to avoid sending shutdown via the c->ssl->no_send_shutdown flag, similarly to how it is done in ngx_http_keepalive_handler() for kqueue when pending EOF is detected. Reported by Jan Pracha? (http://mailman.nginx.org/pipermail/nginx-devel/2018-December/011702.html). diffstat: src/http/ngx_http_request.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diffs (16 lines): diff -r d57f15922ca3 -r 45764bca69b0 src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c Mon Aug 10 18:52:20 2020 +0300 +++ b/src/http/ngx_http_request.c Mon Aug 10 18:52:34 2020 +0300 @@ -2992,6 +2992,12 @@ closed: rev->error = 1; } +#if (NGX_HTTP_SSL) + if (c->ssl) { + c->ssl->no_send_shutdown = 1; + } +#endif + ngx_log_error(NGX_LOG_INFO, c->log, err, "client prematurely closed connection"); From mdounin at mdounin.ru Mon Aug 10 15:56:02 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 10 Aug 2020 15:56:02 +0000 Subject: [nginx] Core: added a warning about reusing connections. Message-ID: details: https://hg.nginx.org/nginx/rev/b9071b875194 branches: changeset: 7697:b9071b875194 user: Maxim Dounin date: Mon Aug 10 18:52:59 2020 +0300 description: Core: added a warning about reusing connections. Previously, reusing connections happened silently and was only visible in monitoring systems. This was shown to be not very user-friendly, and administrators often didn't realize there were too few connections available to withstand the load, and configured timeouts (keepalive_timeout and http2_idle_timeout) were effectively reduced to keep things running. To provide at least some information about this, a warning is now logged (at most once per second, to avoid flooding the logs). diffstat: src/core/ngx_connection.c | 13 +++++++++++++ src/core/ngx_cycle.h | 1 + 2 files changed, 14 insertions(+), 0 deletions(-) diffs (34 lines): diff -r 45764bca69b0 -r b9071b875194 src/core/ngx_connection.c --- a/src/core/ngx_connection.c Mon Aug 10 18:52:34 2020 +0300 +++ b/src/core/ngx_connection.c Mon Aug 10 18:52:59 2020 +0300 @@ -1298,6 +1298,19 @@ ngx_drain_connections(ngx_cycle_t *cycle ngx_queue_t *q; ngx_connection_t *c; + if (cycle->reusable_connections_n == 0) { + return; + } + + if (cycle->connections_reuse_time != ngx_time()) { + cycle->connections_reuse_time = ngx_time(); + + ngx_log_error(NGX_LOG_WARN, cycle->log, 0, + "%ui worker_connections are not enough, " + "reusing connections", + cycle->connection_n); + } + n = ngx_max(ngx_min(32, cycle->reusable_connections_n / 8), 1); for (i = 0; i < n; i++) { diff -r 45764bca69b0 -r b9071b875194 src/core/ngx_cycle.h --- a/src/core/ngx_cycle.h Mon Aug 10 18:52:34 2020 +0300 +++ b/src/core/ngx_cycle.h Mon Aug 10 18:52:59 2020 +0300 @@ -55,6 +55,7 @@ struct ngx_cycle_s { ngx_queue_t reusable_connections_queue; ngx_uint_t reusable_connections_n; + time_t connections_reuse_time; ngx_array_t listening; ngx_array_t paths; From mdounin at mdounin.ru Mon Aug 10 15:56:05 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 10 Aug 2020 15:56:05 +0000 Subject: [nginx] Core: reusing connections in advance. Message-ID: details: https://hg.nginx.org/nginx/rev/5440ff4ac6fc branches: changeset: 7698:5440ff4ac6fc user: Maxim Dounin date: Mon Aug 10 18:53:07 2020 +0300 description: Core: reusing connections in advance. Reworked connections reuse, so closing connections is attempted in advance, as long as number of free connections is less than 1/16 of worker connections configured. This ensures that new connections can be handled even if closing a reusable connection requires some time, for example, for a lingering close (ticket #2017). The 1/16 ratio is selected to be smaller than 1/8 used for disabling accept when working with accept mutex, so nginx will try to balance new connections to different workers first, and will start reusing connections only if this won't help. diffstat: src/core/ngx_connection.c | 11 +++++------ 1 files changed, 5 insertions(+), 6 deletions(-) diffs (29 lines): diff -r b9071b875194 -r 5440ff4ac6fc src/core/ngx_connection.c --- a/src/core/ngx_connection.c Mon Aug 10 18:52:59 2020 +0300 +++ b/src/core/ngx_connection.c Mon Aug 10 18:53:07 2020 +0300 @@ -1107,12 +1107,9 @@ ngx_get_connection(ngx_socket_t s, ngx_l return NULL; } - c = ngx_cycle->free_connections; + ngx_drain_connections((ngx_cycle_t *) ngx_cycle); - if (c == NULL) { - ngx_drain_connections((ngx_cycle_t *) ngx_cycle); - c = ngx_cycle->free_connections; - } + c = ngx_cycle->free_connections; if (c == NULL) { ngx_log_error(NGX_LOG_ALERT, log, 0, @@ -1298,7 +1295,9 @@ ngx_drain_connections(ngx_cycle_t *cycle ngx_queue_t *q; ngx_connection_t *c; - if (cycle->reusable_connections_n == 0) { + if (cycle->free_connection_n > cycle->connection_n / 16 + || cycle->reusable_connections_n == 0) + { return; } From xeioex at nginx.com Tue Aug 11 14:32:18 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 11 Aug 2020 14:32:18 +0000 Subject: [njs] Getting rid of non-portable _GNU_SOURCE define. Message-ID: details: https://hg.nginx.org/njs/rev/a0f2c61c1c83 branches: changeset: 1486:a0f2c61c1c83 user: Dmitry Volyntsev date: Tue Aug 11 14:31:04 2020 +0000 description: Getting rid of non-portable _GNU_SOURCE define. diffstat: auto/os | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diffs (13 lines): diff -r bcd1a41c6a67 -r a0f2c61c1c83 auto/os --- a/auto/os Tue Jul 28 16:58:59 2020 +0300 +++ b/auto/os Tue Aug 11 14:31:04 2020 +0000 @@ -14,9 +14,6 @@ case "$NJS_SYSTEM" in # Linux uname -p can return "unknown". NJS_SYSTEM_PLATFORM=`uname -m 2>/dev/null` CC=${CC:-cc} - - # NAN and INFINITY require _GNU_SOURCE on old Linux. - NJS_CFLAGS="$NJS_CFLAGS -D_GNU_SOURCE" ;; FreeBSD | NetBSD | OpenBSD) From xeioex at nginx.com Tue Aug 11 14:32:20 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 11 Aug 2020 14:32:20 +0000 Subject: [njs] Improved fs.rmdir() to support recursive directory removal. Message-ID: details: https://hg.nginx.org/njs/rev/24de499877ca branches: changeset: 1487:24de499877ca user: Artem S. Povalyukhin date: Sat Jul 25 15:49:03 2020 +0300 description: Improved fs.rmdir() to support recursive directory removal. diffstat: src/njs_fs.c | 262 ++++++++++++++++++++++++++++++++++++++++++-- test/js/fs_promises_009.js | 106 ++++++++++++++++++ test/njs_expect_test.exp | 3 + 3 files changed, 359 insertions(+), 12 deletions(-) diffs (415 lines): diff -r a0f2c61c1c83 -r 24de499877ca src/njs_fs.c --- a/src/njs_fs.c Tue Aug 11 14:31:04 2020 +0000 +++ b/src/njs_fs.c Sat Jul 25 15:49:03 2020 +0300 @@ -63,6 +63,28 @@ typedef struct { } njs_fs_entry_t; +typedef enum { + NJS_FTW_PHYS = 1, + NJS_FTW_MOUNT = 2, + NJS_FTW_DEPTH = 8, +} njs_ftw_flags_t; + + +typedef enum { + NJS_FTW_F, + NJS_FTW_D, + NJS_FTW_DNR, + NJS_FTW_NS, + NJS_FTW_SL, + NJS_FTW_DP, + NJS_FTW_SLN, +} njs_ftw_type_t; + + +typedef int (*njs_file_tree_walk_cb_t)(const char *, const struct stat *, + njs_ftw_type_t); + + static njs_int_t njs_fs_fd_read(njs_vm_t *vm, int fd, njs_str_t *data); static njs_int_t njs_fs_error(njs_vm_t *vm, const char *syscall, @@ -70,8 +92,14 @@ static njs_int_t njs_fs_error(njs_vm_t * static njs_int_t njs_fs_result(njs_vm_t *vm, njs_value_t *result, njs_index_t calltype, const njs_value_t* callback, njs_uint_t nargs); +static njs_int_t +njs_file_tree_walk(const char *path, njs_file_tree_walk_cb_t cb, int fd_limit, + njs_ftw_flags_t flags); + static njs_int_t njs_fs_make_path(njs_vm_t *vm, const char *path, mode_t md, njs_bool_t recursive, njs_value_t *retval); +static njs_int_t njs_fs_rmtree(njs_vm_t *vm, const char *path, + njs_bool_t recursive, njs_value_t *retval); static int njs_fs_flags(njs_vm_t *vm, njs_value_t *value, int default_flags); static mode_t njs_fs_mode(njs_vm_t *vm, njs_value_t *value, @@ -893,18 +921,7 @@ njs_fs_rmdir(njs_vm_t *vm, njs_value_t * } } - if (njs_is_true(&recursive)) { - njs_type_error(vm, "\"options.recursive\" is not supported"); - return NJS_ERROR; - } - - njs_set_undefined(&retval); - - ret = rmdir(file_path); - if (njs_slow_path(ret != 0)) { - ret = njs_fs_error(vm, "rmdir", strerror(errno), path, errno, - &retval); - } + ret = njs_fs_rmtree(vm, file_path, njs_is_true(&recursive), &retval); if (ret == NJS_OK) { return njs_fs_result(vm, &retval, calltype, callback, 1); @@ -1227,6 +1244,227 @@ failed: } +typedef struct njs_ftw_trace_s njs_ftw_trace_t; + +struct njs_ftw_trace_s { + struct njs_ftw_trace_s *chain; + dev_t dev; + ino_t ino; +}; + + +static int +njs_ftw(char *path, njs_file_tree_walk_cb_t cb, int fd_limit, + njs_ftw_flags_t flags, njs_ftw_trace_t *parent) +{ + int type, ret, dfd, err; + DIR *d; + size_t base, len, length; + const char *d_name; + struct stat st; + struct dirent *entry; + njs_ftw_trace_t trace, *h; + + ret = (flags & NJS_FTW_PHYS) ? lstat(path, &st) : stat(path, &st); + + if (ret < 0) { + if (!(flags & NJS_FTW_PHYS) && errno == ENOENT && !lstat(path, &st)) { + type = NJS_FTW_SLN; + + } else if (errno != EACCES) { + return -1; + + } else { + type = NJS_FTW_NS; + } + + } else if (S_ISDIR(st.st_mode)) { + type = (flags & NJS_FTW_DEPTH) ? NJS_FTW_DP : NJS_FTW_D; + + } else if (S_ISLNK(st.st_mode)) { + type = (flags & NJS_FTW_PHYS) ? NJS_FTW_SL : NJS_FTW_SLN; + + } else { + type = NJS_FTW_F; + } + + if ((flags & NJS_FTW_MOUNT) && parent != NULL && st.st_dev != parent->dev) { + return 0; + } + + len = njs_strlen(path); + base = len && (path[len - 1] == '/') ? len - 1 : len; + + trace.chain = parent; + trace.dev = st.st_dev; + trace.ino = st.st_ino; + + dfd = 0; + err = 0; + + if (type == NJS_FTW_D || type == NJS_FTW_DP) { + dfd = open(path, O_RDONLY); + err = errno; + if (dfd < 0 && err == EACCES) { + type = NJS_FTW_DNR; + } + + if (fd_limit == 0) { + close(dfd); + } + } + + if (!(flags & NJS_FTW_DEPTH)) { + ret = cb(path, &st, type); + if (njs_slow_path(ret != 0)) { + return ret; + } + } + + for (h = parent; h != NULL; h = h->chain) { + if (h->dev == st.st_dev && h->ino == st.st_ino) { + return 0; + } + } + + if ((type == NJS_FTW_D || type == NJS_FTW_DP) && fd_limit != 0) { + if (dfd < 0) { + errno = err; + return -1; + } + + d = fdopendir(dfd); + if (njs_slow_path(d == NULL)) { + close(dfd); + return -1; + } + + for ( ;; ) { + entry = readdir(d); + + if (entry == NULL) { + break; + } + + d_name = entry->d_name; + length = njs_strlen(d_name); + + if ((length == 1 && d_name[0] == '.') + || (length == 2 && (d_name[0] == '.' && d_name[1] == '.'))) + { + continue; + } + + if (njs_slow_path(length >= (PATH_MAX - len))) { + errno = ENAMETOOLONG; + closedir(d); + return -1; + } + + path[base] = '/'; + strcpy(path + base + 1, d_name); + + ret = njs_ftw(path, cb, fd_limit - 1, flags, &trace); + if (njs_slow_path(ret != 0)) { + closedir(d); + return ret; + } + } + + closedir(d); + } + + path[len] = '\0'; + + if (flags & NJS_FTW_DEPTH) { + ret = cb(path, &st, type); + if (njs_slow_path(ret != 0)) { + return ret; + } + } + + return 0; +} + + +static njs_int_t +njs_file_tree_walk(const char *path, njs_file_tree_walk_cb_t cb, int fd_limit, + njs_ftw_flags_t flags) +{ + size_t len; + char pathbuf[PATH_MAX + 1]; + + len = njs_strlen(path); + if (njs_slow_path(len > PATH_MAX)) { + errno = ENAMETOOLONG; + return -1; + } + + memcpy(pathbuf, path, len + 1); + + return njs_ftw(pathbuf, cb, fd_limit, flags, NULL); +} + + +static int +njs_fs_rmtree_cb(const char *path, const struct stat *sb, njs_ftw_type_t type) +{ + njs_int_t ret; + + ret = remove(path); + if (ret != 0) { + return -1; + } + + return 0; +} + + +static njs_int_t +njs_fs_rmtree(njs_vm_t *vm, const char *path, njs_bool_t recursive, + njs_value_t *retval) +{ + size_t size; + ssize_t length; + njs_int_t ret; + const char *description; + njs_value_t value; + + njs_set_undefined(retval); + + ret = rmdir(path); + if (ret == 0) { + return NJS_OK; + } + + description = strerror(errno); + + if (recursive && (errno == ENOTEMPTY || errno == EEXIST)) { + ret = njs_file_tree_walk(path, njs_fs_rmtree_cb, 16, + NJS_FTW_PHYS | NJS_FTW_MOUNT | NJS_FTW_DEPTH); + + if (ret == 0) { + return NJS_OK; + } + + description = strerror(errno); + } + + size = njs_strlen(path); + length = njs_utf8_length((u_char *) path, size); + if (njs_slow_path(length < 0)) { + length = 0; + } + + ret = njs_string_new(vm, &value, (u_char *) path, size, length); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + + return njs_fs_error(vm, "rmdir", description, &value, errno, retval); +} + + static int njs_fs_flags(njs_vm_t *vm, njs_value_t *value, int default_flags) { diff -r a0f2c61c1c83 -r 24de499877ca test/js/fs_promises_009.js --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test/js/fs_promises_009.js Sat Jul 25 15:49:03 2020 +0300 @@ -0,0 +1,106 @@ +var fs = require('fs'); +var fsp = fs.promises; +var root = './build/test/'; +var dname = 'fs_promises_???_009/'; +var lname = 'fs_promises_???_009_lnk'; +var path = 'one/two/three/???'; + + +var setContent = (root, path) => { + fs.mkdirSync(root + path, { recursive: true }); + path + .split('/') + .forEach((x, i, a) => { + for (var j = 1; j < 10; ++j) { + var path = root + a.slice(0, i + 1).join('/') + '_file' + j; + fs.writeFileSync(path, path); + } + }); +}; + + +var isNode = () => process.argv[0].includes('node'); + + +var testSync = () => new Promise((resolve, reject) => { + try { + fs.unlinkSync(root + lname); + } catch (e) { + } + try { + fs.rmdirSync(root + dname, { recursive: true }); + } catch (e) { + } + + try { + + fs.mkdirSync(root + dname); + fs.symlinkSync(dname, root + lname); + try { + fs.rmdirSync(root + lname); + throw new Error('fs.rmdirSync() - error 0'); + } catch (e) { + if (e.code != "ENOTDIR") { + throw e; + } + } + fs.rmdirSync(root + dname); + fs.unlinkSync(root + lname); + + if (!isNode()) { + fs.mkdirSync(root + dname); + fs.symlinkSync(dname, root + lname); + try { + fs.rmdirSync(root + lname, { recursive: true }); + throw new Error('fs.rmdirSync() - error 1'); + } catch (e) { + if (e.code != "ENOTDIR") { + throw e; + } + } + fs.rmdirSync(root + dname); + fs.unlinkSync(root + lname); + } + + fs.mkdirSync(root + dname, { mode: 0 }); + fs.rmdirSync(root + dname, { recursive: true }); + + setContent(root + dname, path); + fs.rmdirSync(root + dname, { recursive: true }); + + try { + fs.accessSync(root + dname); + throw new Error('fs.rmdirSync() - error 2'); + } catch (e) { + if (e.code != "ENOENT") { + throw e; + } + } + + if (!isNode()) { + try { + fs.rmdirSync(root + dname, { recursive: true }); + throw new Error('fs.rmdirSync() - error 3'); + } catch (e) { + if (e.code != "ENOENT") { + throw e; + } + } + } + + resolve(); + + } catch (e) { + reject(e); + } +}); + + +Promise.resolve() +.then(testSync) +.then(() => { + console.log('test recursive fs.rmdirSync()'); +}) +.catch((e) => { + console.log('test failed recursive fs.rmdirSync()', e.message, JSON.stringify(e)); +}); diff -r a0f2c61c1c83 -r 24de499877ca test/njs_expect_test.exp --- a/test/njs_expect_test.exp Tue Aug 11 14:31:04 2020 +0000 +++ b/test/njs_expect_test.exp Sat Jul 25 15:49:03 2020 +0300 @@ -1143,3 +1143,6 @@ test fsp.readdir" njs_run {"./test/js/fs_promises_008.js"} \ "test recursive fs.mkdirSync" + +njs_run {"./test/js/fs_promises_009.js"} \ +"test recursive fs.rmdirSync" From xeioex at nginx.com Tue Aug 11 14:32:22 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 11 Aug 2020 14:32:22 +0000 Subject: [njs] Version 0.4.3. Message-ID: details: https://hg.nginx.org/njs/rev/1ada1061a040 branches: changeset: 1488:1ada1061a040 user: Dmitry Volyntsev date: Tue Aug 11 14:31:07 2020 +0000 description: Version 0.4.3. diffstat: CHANGES | 26 ++++++++++++++++++++++++++ 1 files changed, 26 insertions(+), 0 deletions(-) diffs (33 lines): diff -r 24de499877ca -r 1ada1061a040 CHANGES --- a/CHANGES Sat Jul 25 15:49:03 2020 +0300 +++ b/CHANGES Tue Aug 11 14:31:07 2020 +0000 @@ -1,3 +1,29 @@ + +Changes with njs 0.4.3 11 Aug 2020 + + Core: + + *) Feature: added Query String module. + + *) Feature: improved fs.mkdir() to support recursive directory creation. + Thanks to Artem S. Povalyukhin. + + *) Feature: improved fs.rmdir() to support recursive directory removal. + Thanks to Artem S. Povalyukhin. + + *) Feature: introduced UTF-8 decoder according to WHATWG encoding spec. + + *) Feature: added TextEncoder/TextDecoder implementation. + + *) Bugfix: fixed parsing return statement without semicolon. + + *) Bugfix: fixed njs_number_to_int32() for big-endian platforms. + + *) Bugfix: fixed unit test on big-endian platforms. + + *) Bugfix: fixed regexp-literals parsing with '=' characters. + + *) Bugfix: fixed pre/post increment/decrement in assignment operations. Changes with njs 0.4.2 07 Jul 2020 From xeioex at nginx.com Tue Aug 11 14:32:24 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 11 Aug 2020 14:32:24 +0000 Subject: [njs] Added tag 0.4.3 for changeset 1ada1061a040 Message-ID: details: https://hg.nginx.org/njs/rev/8e92a5b4b951 branches: changeset: 1489:8e92a5b4b951 user: Dmitry Volyntsev date: Tue Aug 11 14:31:50 2020 +0000 description: Added tag 0.4.3 for changeset 1ada1061a040 diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff -r 1ada1061a040 -r 8e92a5b4b951 .hgtags --- a/.hgtags Tue Aug 11 14:31:07 2020 +0000 +++ b/.hgtags Tue Aug 11 14:31:50 2020 +0000 @@ -36,3 +36,4 @@ fc4eeaaf0dfe7dc7d41232f1b643bd364f1efe82 6144aafa1472fbdf79bc9d1f858555938ee08452 0.4.0 9400790bf53843001f94c77b47bc99b05518af78 0.4.1 b409e86fd02a6f2cb3d741a41b6562471e1b66ef 0.4.2 +1ada1061a040e5cd5ec55744bfa916dfc6744e4c 0.4.3 From mdounin at mdounin.ru Tue Aug 11 15:03:25 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 11 Aug 2020 15:03:25 +0000 Subject: [nginx] nginx-1.19.2-RELEASE Message-ID: details: https://hg.nginx.org/nginx/rev/a7b46539f507 branches: changeset: 7699:a7b46539f507 user: Maxim Dounin date: Tue Aug 11 17:52:30 2020 +0300 description: nginx-1.19.2-RELEASE diffstat: docs/xml/nginx/changes.xml | 102 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 102 insertions(+), 0 deletions(-) diffs (112 lines): diff -r 5440ff4ac6fc -r a7b46539f507 docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml Mon Aug 10 18:53:07 2020 +0300 +++ b/docs/xml/nginx/changes.xml Tue Aug 11 17:52:30 2020 +0300 @@ -5,6 +5,108 @@ + + + + +?????? nginx ???????? ????????? keepalive-??????????, +?? ????????? ?????????? ???? ????????? ??????????, +? ????? ????? ?? ???? ?????????????? ? ??? ??????. + + +now nginx starts closing keepalive connections +before all free worker connections are exhausted, +and logs a warning about this to the error log. + + + + + +??????????? ?????? ???? ??????? +??? ????????????? chunked transfer encoding. + + +optimization of client request body reading +when using chunked transfer encoding. + + + + + +?????? ?????? ??? ????????????? ????????? ssl_ocsp. + + +memory leak if the "ssl_ocsp" directive was used. + + + + + +? ????? ????? ?????????? ????????? "zero size buf in output", +???? FastCGI-?????? ????????? ???????????? ?????; +?????? ????????? ? 1.19.1. + + +"zero size buf in output" alerts might appear in logs +if a FastCGI server returned an incorrect response; +the bug had appeared in 1.19.1. + + + + + +? ??????? ???????? ??? ????????? segmentation fault, +???? ??????? large_client_header_buffers ?????????? +? ?????? ??????????? ????????. + + +a segmentation fault might occur in a worker process +if different large_client_header_buffers sizes were used +in different virtual servers. + + + + + +SSL shutdown ??? ?? ????????. + + +SSL shutdown might not work. + + + + + +? ????? ????? ?????????? ????????? +"SSL_shutdown() failed (SSL: ... bad write retry)". + + +"SSL_shutdown() failed (SSL: ... bad write retry)" +messages might appear in logs. + + + + + +? ?????? ngx_http_slice_module. + + +in the ngx_http_slice_module. + + + + + +? ?????? ngx_http_xslt_filter_module. + + +in the ngx_http_xslt_filter_module. + + + + + + From mdounin at mdounin.ru Tue Aug 11 15:03:28 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 11 Aug 2020 15:03:28 +0000 Subject: [nginx] release-1.19.2 tag Message-ID: details: https://hg.nginx.org/nginx/rev/7d46c9f56c9a branches: changeset: 7700:7d46c9f56c9a user: Maxim Dounin date: Tue Aug 11 17:52:30 2020 +0300 description: release-1.19.2 tag diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff -r a7b46539f507 -r 7d46c9f56c9a .hgtags --- a/.hgtags Tue Aug 11 17:52:30 2020 +0300 +++ b/.hgtags Tue Aug 11 17:52:30 2020 +0300 @@ -451,3 +451,4 @@ 5e8d52bca714d4b85284ddb649d1ba4a3ca978a8 c44970de01474f6f3e01b0adea85ec1d03e3a5f2 release-1.17.10 cbe6ba650211541310618849168631ce0b788f35 release-1.19.0 062920e2f3bf871ef7a3d8496edec1b3065faf80 release-1.19.1 +a7b46539f507e6c64efa0efda69ad60b6f4ffbce release-1.19.2 From xeioex at nginx.com Wed Aug 12 15:02:06 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 12 Aug 2020 15:02:06 +0000 Subject: [njs] Version bump. Message-ID: details: https://hg.nginx.org/njs/rev/7711232339a1 branches: changeset: 1490:7711232339a1 user: Dmitry Volyntsev date: Wed Aug 12 12:56:35 2020 +0000 description: Version bump. diffstat: src/njs.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 8e92a5b4b951 -r 7711232339a1 src/njs.h --- a/src/njs.h Tue Aug 11 14:31:50 2020 +0000 +++ b/src/njs.h Wed Aug 12 12:56:35 2020 +0000 @@ -11,7 +11,7 @@ #include -#define NJS_VERSION "0.4.3" +#define NJS_VERSION "0.4.4" #include /* STDOUT_FILENO, STDERR_FILENO */ From xeioex at nginx.com Wed Aug 12 15:02:08 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 12 Aug 2020 15:02:08 +0000 Subject: [njs] Fixing Coverity warnings introduced in 24de499877ca. Message-ID: details: https://hg.nginx.org/njs/rev/5cce5069440e branches: changeset: 1491:5cce5069440e user: Dmitry Volyntsev date: Wed Aug 12 14:57:31 2020 +0000 description: Fixing Coverity warnings introduced in 24de499877ca. CIDs: 1465870, 1465871, 1465872. diffstat: src/njs_fs.c | 31 ++++++++++++++++++------------- 1 files changed, 18 insertions(+), 13 deletions(-) diffs (65 lines): diff -r 7711232339a1 -r 5cce5069440e src/njs_fs.c --- a/src/njs_fs.c Wed Aug 12 12:56:35 2020 +0000 +++ b/src/njs_fs.c Wed Aug 12 14:57:31 2020 +0000 @@ -1292,6 +1292,12 @@ njs_ftw(char *path, njs_file_tree_walk_c return 0; } + for (h = parent; h != NULL; h = h->chain) { + if (h->dev == st.st_dev && h->ino == st.st_ino) { + return 0; + } + } + len = njs_strlen(path); base = len && (path[len - 1] == '/') ? len - 1 : len; @@ -1305,28 +1311,27 @@ njs_ftw(char *path, njs_file_tree_walk_c if (type == NJS_FTW_D || type == NJS_FTW_DP) { dfd = open(path, O_RDONLY); err = errno; - if (dfd < 0 && err == EACCES) { - type = NJS_FTW_DNR; - } - - if (fd_limit == 0) { - close(dfd); + if (dfd < 0) { + if (err == EACCES) { + type = NJS_FTW_DNR; + } + + } else if (fd_limit == 0) { + (void) close(dfd); } } if (!(flags & NJS_FTW_DEPTH)) { ret = cb(path, &st, type); if (njs_slow_path(ret != 0)) { + if (dfd >= 0) { + (void) close(dfd); + } + return ret; } } - for (h = parent; h != NULL; h = h->chain) { - if (h->dev == st.st_dev && h->ino == st.st_ino) { - return 0; - } - } - if ((type == NJS_FTW_D || type == NJS_FTW_DP) && fd_limit != 0) { if (dfd < 0) { errno = err; @@ -1335,7 +1340,7 @@ njs_ftw(char *path, njs_file_tree_walk_c d = fdopendir(dfd); if (njs_slow_path(d == NULL)) { - close(dfd); + (void) close(dfd); return -1; } From xeioex at nginx.com Thu Aug 13 13:42:34 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 13 Aug 2020 13:42:34 +0000 Subject: [njs] Improved error handling in njs_file_tree_walk(). Message-ID: details: https://hg.nginx.org/njs/rev/5a80b43b7098 branches: changeset: 1492:5a80b43b7098 user: Dmitry Volyntsev date: Thu Aug 13 13:40:36 2020 +0000 description: Improved error handling in njs_file_tree_walk(). This also fixes Coverity CID 1465877. diffstat: src/njs_fs.c | 75 +++++++++++++++++++++++++++++++---------------------------- 1 files changed, 39 insertions(+), 36 deletions(-) diffs (179 lines): diff -r 5cce5069440e -r 5a80b43b7098 src/njs_fs.c --- a/src/njs_fs.c Wed Aug 12 14:57:31 2020 +0000 +++ b/src/njs_fs.c Thu Aug 13 13:40:36 2020 +0000 @@ -81,7 +81,7 @@ typedef enum { } njs_ftw_type_t; -typedef int (*njs_file_tree_walk_cb_t)(const char *, const struct stat *, +typedef njs_int_t (*njs_file_tree_walk_cb_t)(const char *, const struct stat *, njs_ftw_type_t); @@ -1257,7 +1257,7 @@ static int njs_ftw(char *path, njs_file_tree_walk_cb_t cb, int fd_limit, njs_ftw_flags_t flags, njs_ftw_trace_t *parent) { - int type, ret, dfd, err; + int type, ret, dfd; DIR *d; size_t base, len, length; const char *d_name; @@ -1272,7 +1272,7 @@ njs_ftw(char *path, njs_file_tree_walk_c type = NJS_FTW_SLN; } else if (errno != EACCES) { - return -1; + return NJS_ERROR; } else { type = NJS_FTW_NS; @@ -1289,12 +1289,12 @@ njs_ftw(char *path, njs_file_tree_walk_c } if ((flags & NJS_FTW_MOUNT) && parent != NULL && st.st_dev != parent->dev) { - return 0; + return NJS_OK; } for (h = parent; h != NULL; h = h->chain) { if (h->dev == st.st_dev && h->ino == st.st_ino) { - return 0; + return NJS_OK; } } @@ -1305,43 +1305,32 @@ njs_ftw(char *path, njs_file_tree_walk_c trace.dev = st.st_dev; trace.ino = st.st_ino; - dfd = 0; - err = 0; + d = NULL; + dfd = -1; if (type == NJS_FTW_D || type == NJS_FTW_DP) { dfd = open(path, O_RDONLY); - err = errno; if (dfd < 0) { - if (err == EACCES) { - type = NJS_FTW_DNR; + if (errno != EACCES) { + return NJS_ERROR; } - } else if (fd_limit == 0) { - (void) close(dfd); + type = NJS_FTW_DNR; } } if (!(flags & NJS_FTW_DEPTH)) { ret = cb(path, &st, type); if (njs_slow_path(ret != 0)) { - if (dfd >= 0) { - (void) close(dfd); - } - - return ret; + goto done; } } - if ((type == NJS_FTW_D || type == NJS_FTW_DP) && fd_limit != 0) { - if (dfd < 0) { - errno = err; - return -1; - } - + if (type == NJS_FTW_D || type == NJS_FTW_DP) { d = fdopendir(dfd); if (njs_slow_path(d == NULL)) { - (void) close(dfd); - return -1; + ret = NJS_ERROR; + goto done; } for ( ;; ) { @@ -1362,21 +1351,23 @@ njs_ftw(char *path, njs_file_tree_walk_c if (njs_slow_path(length >= (PATH_MAX - len))) { errno = ENAMETOOLONG; - closedir(d); - return -1; + ret = NJS_ERROR; + goto done; } path[base] = '/'; strcpy(path + base + 1, d_name); - ret = njs_ftw(path, cb, fd_limit - 1, flags, &trace); - if (njs_slow_path(ret != 0)) { - closedir(d); - return ret; + if (fd_limit != 0) { + ret = njs_ftw(path, cb, fd_limit - 1, flags, &trace); + if (njs_slow_path(ret != 0)) { + goto done; + } } } closedir(d); + d = NULL; } path[len] = '\0'; @@ -1388,7 +1379,19 @@ njs_ftw(char *path, njs_file_tree_walk_c } } - return 0; + ret = NJS_OK; + +done: + + if (d != NULL) { + /* closedir() also closes underlying dfd. */ + (void) closedir(d); + + } else if (dfd >= 0) { + (void) close(dfd); + } + + return ret; } @@ -1411,17 +1414,17 @@ njs_file_tree_walk(const char *path, njs } -static int +static njs_int_t njs_fs_rmtree_cb(const char *path, const struct stat *sb, njs_ftw_type_t type) { njs_int_t ret; ret = remove(path); if (ret != 0) { - return -1; + return NJS_ERROR; } - return 0; + return NJS_OK; } @@ -1448,7 +1451,7 @@ njs_fs_rmtree(njs_vm_t *vm, const char * ret = njs_file_tree_walk(path, njs_fs_rmtree_cb, 16, NJS_FTW_PHYS | NJS_FTW_MOUNT | NJS_FTW_DEPTH); - if (ret == 0) { + if (ret == NJS_OK) { return NJS_OK; } From arut at nginx.com Fri Aug 14 09:51:30 2020 From: arut at nginx.com (Roman Arutyunyan) Date: Fri, 14 Aug 2020 09:51:30 +0000 Subject: [nginx] Version bump. Message-ID: details: https://hg.nginx.org/nginx/rev/8ca47f094d5b branches: changeset: 7701:8ca47f094d5b user: Roman Arutyunyan date: Fri Aug 14 12:45:52 2020 +0300 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r 7d46c9f56c9a -r 8ca47f094d5b src/core/nginx.h --- a/src/core/nginx.h Tue Aug 11 17:52:30 2020 +0300 +++ b/src/core/nginx.h Fri Aug 14 12:45:52 2020 +0300 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1019002 -#define NGINX_VERSION "1.19.2" +#define nginx_version 1019003 +#define NGINX_VERSION "1.19.3" #define NGINX_VER "nginx/" NGINX_VERSION #ifdef NGX_BUILD From arut at nginx.com Fri Aug 14 09:51:33 2020 From: arut at nginx.com (Roman Arutyunyan) Date: Fri, 14 Aug 2020 09:51:33 +0000 Subject: [nginx] Cache: ignore stale-if-error for 4xx and 5xx codes. Message-ID: details: https://hg.nginx.org/nginx/rev/7015f26aef90 branches: changeset: 7702:7015f26aef90 user: Roman Arutyunyan date: Wed Jul 29 13:28:04 2020 +0300 description: Cache: ignore stale-if-error for 4xx and 5xx codes. Previously the stale-if-error extension of the Cache-Control upstream header triggered the return of a stale response for all error conditions that can be specified in the proxy_cache_use_stale directive. The list of these errors includes both network/timeout/format errors, as well as some HTTP codes like 503, 504, 403, 429 etc. The latter prevented a cache entry from being updated by a response with any of these HTTP codes during the stale-if-error period. Now stale-if-error only works for network/timeout/format errors and ignores the upstream HTTP code. The return of a stale response for certain HTTP codes is still possible using the proxy_cache_use_stale directive. This change also applies to the stale-while-revalidate extension of the Cache-Control header, which triggers stale-if-error if it is missing. Reported at http://mailman.nginx.org/pipermail/nginx/2020-July/059723.html. diffstat: src/http/ngx_http_upstream.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 8ca47f094d5b -r 7015f26aef90 src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c Fri Aug 14 12:45:52 2020 +0300 +++ b/src/http/ngx_http_upstream.c Wed Jul 29 13:28:04 2020 +0300 @@ -2473,7 +2473,7 @@ ngx_http_upstream_test_next(ngx_http_req #if (NGX_HTTP_CACHE) if (u->cache_status == NGX_HTTP_CACHE_EXPIRED - && ((u->conf->cache_use_stale & un->mask) || r->cache->stale_error)) + && (u->conf->cache_use_stale & un->mask)) { ngx_int_t rc; From xeioex at nginx.com Mon Aug 17 11:24:13 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Mon, 17 Aug 2020 11:24:13 +0000 Subject: [njs] HTTP: fixed location merge. Message-ID: details: https://hg.nginx.org/njs/rev/eebf9e525358 branches: changeset: 1493:eebf9e525358 user: Dmitry Volyntsev date: Mon Aug 17 11:22:35 2020 +0000 description: HTTP: fixed location merge. diffstat: nginx/ngx_http_js_module.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diffs (14 lines): diff -r 5a80b43b7098 -r eebf9e525358 nginx/ngx_http_js_module.c --- a/nginx/ngx_http_js_module.c Thu Aug 13 13:40:36 2020 +0000 +++ b/nginx/ngx_http_js_module.c Mon Aug 17 11:22:35 2020 +0000 @@ -3363,5 +3363,10 @@ ngx_http_js_create_loc_conf(ngx_conf_t * static char * ngx_http_js_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child) { + ngx_http_js_loc_conf_t *prev = parent; + ngx_http_js_loc_conf_t *conf = child; + + ngx_conf_merge_str_value(conf->content, prev->content, ""); + return NGX_CONF_OK; } From vbart at nginx.com Mon Aug 17 11:48:02 2020 From: vbart at nginx.com (Valentin Bartenev) Date: Mon, 17 Aug 2020 11:48:02 +0000 Subject: [njs] Unicode case tables updated to version 13.0.0 (March 2020). Message-ID: details: https://hg.nginx.org/njs/rev/10a2c35d53e7 branches: changeset: 1494:10a2c35d53e7 user: Valentin Bartenev date: Mon Aug 17 14:44:29 2020 +0300 description: Unicode case tables updated to version 13.0.0 (March 2020). diffstat: src/njs_unicode_lower_case.h | 6 +++--- src/njs_unicode_upper_case.h | 4 ++-- src/njs_utf8.c | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diffs (50 lines): diff -r eebf9e525358 -r 10a2c35d53e7 src/njs_unicode_lower_case.h --- a/src/njs_unicode_lower_case.h Mon Aug 17 11:22:35 2020 +0000 +++ b/src/njs_unicode_lower_case.h Mon Aug 17 14:44:29 2020 +0300 @@ -570,13 +570,13 @@ static const uint32_t njs_unicode_lower 0x0a7a9, 0x0a7a9, 0x00266, 0x0025c, 0x00261, 0x0026c, 0x0026a, 0x0a7af, 0x0029e, 0x00287, 0x0029d, 0x0ab53, 0x0a7b5, 0x0a7b5, 0x0a7b7, 0x0a7b7, 0x0a7b9, 0x0a7b9, 0x0a7bb, 0x0a7bb, 0x0a7bd, 0x0a7bd, 0x0a7bf, 0x0a7bf, - 0x0a7c0, 0x0a7c1, 0x0a7c3, 0x0a7c3, 0x0a794, 0x00282, 0x01d8e, 0x0a7c7, - 0x0a7c8, 0x0a7c9, 0x0a7ca, 0x0a7cb, 0x0a7cc, 0x0a7cd, 0x0a7ce, 0x0a7cf, + 0x0a7c0, 0x0a7c1, 0x0a7c3, 0x0a7c3, 0x0a794, 0x00282, 0x01d8e, 0x0a7c8, + 0x0a7c8, 0x0a7ca, 0x0a7ca, 0x0a7cb, 0x0a7cc, 0x0a7cd, 0x0a7ce, 0x0a7cf, 0x0a7d0, 0x0a7d1, 0x0a7d2, 0x0a7d3, 0x0a7d4, 0x0a7d5, 0x0a7d6, 0x0a7d7, 0x0a7d8, 0x0a7d9, 0x0a7da, 0x0a7db, 0x0a7dc, 0x0a7dd, 0x0a7de, 0x0a7df, 0x0a7e0, 0x0a7e1, 0x0a7e2, 0x0a7e3, 0x0a7e4, 0x0a7e5, 0x0a7e6, 0x0a7e7, 0x0a7e8, 0x0a7e9, 0x0a7ea, 0x0a7eb, 0x0a7ec, 0x0a7ed, 0x0a7ee, 0x0a7ef, - 0x0a7f0, 0x0a7f1, 0x0a7f2, 0x0a7f3, 0x0a7f4, 0x0a7f5, 0x0a7f6, 0x0a7f7, + 0x0a7f0, 0x0a7f1, 0x0a7f2, 0x0a7f3, 0x0a7f4, 0x0a7f6, 0x0a7f6, 0x0a7f7, 0x0a7f8, 0x0a7f9, 0x0a7fa, 0x0a7fb, 0x0a7fc, 0x0a7fd, 0x0a7fe, 0x0a7ff, }; diff -r eebf9e525358 -r 10a2c35d53e7 src/njs_unicode_upper_case.h --- a/src/njs_unicode_upper_case.h Mon Aug 17 11:22:35 2020 +0000 +++ b/src/njs_unicode_upper_case.h Mon Aug 17 14:44:29 2020 +0300 @@ -681,12 +681,12 @@ static const uint32_t njs_unicode_upper 0x0a7b0, 0x0a7b1, 0x0a7b2, 0x0a7b3, 0x0a7b4, 0x0a7b4, 0x0a7b6, 0x0a7b6, 0x0a7b8, 0x0a7b8, 0x0a7ba, 0x0a7ba, 0x0a7bc, 0x0a7bc, 0x0a7be, 0x0a7be, 0x0a7c0, 0x0a7c1, 0x0a7c2, 0x0a7c2, 0x0a7c4, 0x0a7c5, 0x0a7c6, 0x0a7c7, - 0x0a7c8, 0x0a7c9, 0x0a7ca, 0x0a7cb, 0x0a7cc, 0x0a7cd, 0x0a7ce, 0x0a7cf, + 0x0a7c7, 0x0a7c9, 0x0a7c9, 0x0a7cb, 0x0a7cc, 0x0a7cd, 0x0a7ce, 0x0a7cf, 0x0a7d0, 0x0a7d1, 0x0a7d2, 0x0a7d3, 0x0a7d4, 0x0a7d5, 0x0a7d6, 0x0a7d7, 0x0a7d8, 0x0a7d9, 0x0a7da, 0x0a7db, 0x0a7dc, 0x0a7dd, 0x0a7de, 0x0a7df, 0x0a7e0, 0x0a7e1, 0x0a7e2, 0x0a7e3, 0x0a7e4, 0x0a7e5, 0x0a7e6, 0x0a7e7, 0x0a7e8, 0x0a7e9, 0x0a7ea, 0x0a7eb, 0x0a7ec, 0x0a7ed, 0x0a7ee, 0x0a7ef, - 0x0a7f0, 0x0a7f1, 0x0a7f2, 0x0a7f3, 0x0a7f4, 0x0a7f5, 0x0a7f6, 0x0a7f7, + 0x0a7f0, 0x0a7f1, 0x0a7f2, 0x0a7f3, 0x0a7f4, 0x0a7f5, 0x0a7f5, 0x0a7f7, 0x0a7f8, 0x0a7f9, 0x0a7fa, 0x0a7fb, 0x0a7fc, 0x0a7fd, 0x0a7fe, 0x0a7ff, }; diff -r eebf9e525358 -r 10a2c35d53e7 src/njs_utf8.c --- a/src/njs_utf8.c Mon Aug 17 11:22:35 2020 +0000 +++ b/src/njs_utf8.c Mon Aug 17 14:44:29 2020 +0300 @@ -9,7 +9,7 @@ /* * The njs_unicode_lower_case.h and njs_unicode_upper_case.h files are - * files auto-generated from the UnicodeData.txt file version 12.1.0 (May 2019) + * auto-generated from the UnicodeData.txt file version 13.0.0 (March 2020) * provided by Unicode, Inc.: * * ./njs_unicode_lower_case.pl UnicodeData.txt From vbart at nginx.com Mon Aug 17 16:56:38 2020 From: vbart at nginx.com (Valentin Bartenev) Date: Mon, 17 Aug 2020 16:56:38 +0000 Subject: [njs] Added support for numeric separators (ES12). Message-ID: details: https://hg.nginx.org/njs/rev/4818a450f4e6 branches: changeset: 1495:4818a450f4e6 user: Valentin Bartenev date: Mon Aug 17 19:55:46 2020 +0300 description: Added support for numeric separators (ES12). diffstat: src/njs_json.c | 2 +- src/njs_lexer.c | 10 +++- src/njs_number.c | 42 +++++++++++++----- src/njs_number.h | 6 +- src/njs_parser.c | 4 +- src/njs_string.c | 6 +- src/njs_strtod.c | 36 +++++++++++++++- src/njs_strtod.h | 3 +- src/test/njs_unit_test.c | 101 ++++++++++++++++++++++++++++++++++++++++++---- 9 files changed, 173 insertions(+), 37 deletions(-) diffs (518 lines): diff -r 10a2c35d53e7 -r 4818a450f4e6 src/njs_json.c --- a/src/njs_json.c Mon Aug 17 14:44:29 2020 +0300 +++ b/src/njs_json.c Mon Aug 17 19:55:46 2020 +0300 @@ -796,7 +796,7 @@ njs_json_parse_number(njs_json_parse_ctx } start = p; - num = njs_number_dec_parse(&p, ctx->end); + num = njs_number_dec_parse(&p, ctx->end, 0); if (p != start) { njs_set_number(value, sign * num); return p; diff -r 10a2c35d53e7 -r 4818a450f4e6 src/njs_lexer.c --- a/src/njs_lexer.c Mon Aug 17 14:44:29 2020 +0300 +++ b/src/njs_lexer.c Mon Aug 17 19:55:46 2020 +0300 @@ -787,7 +787,7 @@ njs_lexer_number(njs_lexer_t *lexer, njs goto illegal_token; } - token->number = njs_number_hex_parse(&p, lexer->end); + token->number = njs_number_hex_parse(&p, lexer->end, 1); goto done; } @@ -830,16 +830,20 @@ njs_lexer_number(njs_lexer_t *lexer, njs /* Legacy Octal literals are deprecated. */ - if (*p >= '0' && *p <= '9') { + if ((*p >= '0' && *p <= '9') || *p == '_') { goto illegal_trailer; } } p--; - token->number = njs_number_dec_parse(&p, lexer->end); + token->number = njs_number_dec_parse(&p, lexer->end, 1); done: + if (p[-1] == '_') { + p--; + } + lexer->start = (u_char *) p; token->text.length = p - token->text.start; diff -r 10a2c35d53e7 -r 4818a450f4e6 src/njs_number.c --- a/src/njs_number.c Mon Aug 17 14:44:29 2020 +0300 +++ b/src/njs_number.c Mon Aug 17 19:55:46 2020 +0300 @@ -54,9 +54,10 @@ njs_key_to_index(const njs_value_t *valu double -njs_number_dec_parse(const u_char **start, const u_char *end) +njs_number_dec_parse(const u_char **start, const u_char *end, + njs_bool_t literal) { - return njs_strtod(start, end); + return njs_strtod(start, end, literal); } @@ -65,22 +66,27 @@ njs_number_oct_parse(const u_char **star { u_char c; uint64_t num; - const u_char *p; + const u_char *p, *_; p = *start; num = 0; + _ = p - 1; - while (p < end) { + for (; p < end; p++) { /* Values less than '0' become >= 208. */ c = *p - '0'; if (njs_slow_path(c > 7)) { + if (*p == '_' && (p - _) > 1) { + _ = p; + continue; + } + break; } num = num * 8 + c; - p++; } *start = p; @@ -94,22 +100,27 @@ njs_number_bin_parse(const u_char **star { u_char c; uint64_t num; - const u_char *p; + const u_char *p, *_; p = *start; num = 0; + _ = p - 1; - while (p < end) { + for (; p < end; p++) { /* Values less than '0' become >= 208. */ c = *p - '0'; if (njs_slow_path(c > 1)) { + if (*p == '_' && (p - _) > 1) { + _ = p; + continue; + } + break; } num = num * 2 + c; - p++; } *start = p; @@ -119,24 +130,31 @@ njs_number_bin_parse(const u_char **star uint64_t -njs_number_hex_parse(const u_char **start, const u_char *end) +njs_number_hex_parse(const u_char **start, const u_char *end, + njs_bool_t literal) { uint64_t num; njs_int_t n; - const u_char *p; + const u_char *p, *_; p = *start; num = 0; + _ = p - 1; - while (p < end) { + for (; p < end; p++) { n = njs_char_to_hex(*p); + if (njs_slow_path(n < 0)) { + if (literal && *p == '_' && (p - _) > 1) { + _ = p; + continue; + } + break; } num = num * 16 + n; - p++; } *start = p; diff -r 10a2c35d53e7 -r 4818a450f4e6 src/njs_number.h --- a/src/njs_number.h Mon Aug 17 14:44:29 2020 +0300 +++ b/src/njs_number.h Mon Aug 17 19:55:46 2020 +0300 @@ -12,10 +12,12 @@ double njs_key_to_index(const njs_value_t *value); -double njs_number_dec_parse(const u_char **start, const u_char *end); +double njs_number_dec_parse(const u_char **start, const u_char *end, + njs_bool_t literal); uint64_t njs_number_oct_parse(const u_char **start, const u_char *end); uint64_t njs_number_bin_parse(const u_char **start, const u_char *end); -uint64_t njs_number_hex_parse(const u_char **start, const u_char *end); +uint64_t njs_number_hex_parse(const u_char **start, const u_char *end, + njs_bool_t literal); int64_t njs_number_radix_parse(const u_char **start, const u_char *end, uint8_t radix); njs_int_t njs_number_to_string(njs_vm_t *vm, njs_value_t *string, diff -r 10a2c35d53e7 -r 4818a450f4e6 src/njs_parser.c --- a/src/njs_parser.c Mon Aug 17 14:44:29 2020 +0300 +++ b/src/njs_parser.c Mon Aug 17 19:55:46 2020 +0300 @@ -8078,7 +8078,7 @@ njs_parser_escape_string_create(njs_pars hex_end = src + hex_length; hex: - cp = njs_number_hex_parse(&src, hex_end); + cp = njs_number_hex_parse(&src, hex_end, 0); /* Skip '}' character. */ @@ -8219,7 +8219,7 @@ njs_parser_escape_string_calc_length(njs hex: ptr = src; - cp = njs_number_hex_parse(&src, hex_end); + cp = njs_number_hex_parse(&src, hex_end, 0); if (hex_length != 0) { if (src != hex_end) { diff -r 10a2c35d53e7 -r 4818a450f4e6 src/njs_string.c --- a/src/njs_string.c Mon Aug 17 14:44:29 2020 +0300 +++ b/src/njs_string.c Mon Aug 17 19:55:46 2020 +0300 @@ -3718,11 +3718,11 @@ njs_string_to_number(const njs_value_t * && p + 2 < end && p[0] == '0' && (p[1] == 'x' || p[1] == 'X')) { p += 2; - num = njs_number_hex_parse(&p, end); + num = njs_number_hex_parse(&p, end, 0); } else { start = p; - num = njs_number_dec_parse(&p, end); + num = njs_number_dec_parse(&p, end, 0); if (p == start) { if (p + infinity > end || memcmp(p, "Infinity", infinity) != 0) { @@ -3821,7 +3821,7 @@ njs_string_to_index(const njs_value_t *v } } - num = njs_strtod(&p, end); + num = njs_strtod(&p, end, 0); if (p != end) { return NAN; } diff -r 10a2c35d53e7 -r 4818a450f4e6 src/njs_strtod.c --- a/src/njs_strtod.c Mon Aug 17 14:44:29 2020 +0300 +++ b/src/njs_strtod.c Mon Aug 17 19:55:46 2020 +0300 @@ -307,12 +307,12 @@ njs_strtod_internal(const u_char *start, double -njs_strtod(const u_char **start, const u_char *end) +njs_strtod(const u_char **start, const u_char *end, njs_bool_t literal) { int exponent, exp, insignf; u_char c, *pos; njs_bool_t minus; - const u_char *e, *p, *last; + const u_char *e, *p, *last, *_; u_char data[128]; exponent = 0; @@ -321,11 +321,25 @@ njs_strtod(const u_char **start, const u pos = data; last = data + sizeof(data); - for (p = *start; p < end; p++) { + p = *start; + _ = p - 2; + + for (; p < end; p++) { /* Values less than '0' become >= 208. */ c = *p - '0'; if (njs_slow_path(c > 9)) { + if (literal) { + if ((p - _) == 1) { + goto done; + } + + if (*p == '_') { + _ = p; + continue; + } + } + break; } @@ -339,12 +353,18 @@ njs_strtod(const u_char **start, const u /* Do not emit a '.', but adjust the exponent instead. */ if (p < end && *p == '.') { + _ = p; for (p++; p < end; p++) { /* Values less than '0' become >= 208. */ c = *p - '0'; if (njs_slow_path(c > 9)) { + if (literal && *p == '_' && (p - _) > 1) { + _ = p; + continue; + } + break; } @@ -388,6 +408,11 @@ njs_strtod(const u_char **start, const u c = *p - '0'; if (njs_slow_path(c > 9)) { + if (literal && *p == '_' && (p - _) > 1) { + _ = p; + continue; + } + break; } @@ -397,9 +422,14 @@ njs_strtod(const u_char **start, const u } exponent += minus ? -exp : exp; + + } else if (literal && *e == '_') { + p = e; } } +done: + *start = p; exponent += insignf; diff -r 10a2c35d53e7 -r 4818a450f4e6 src/njs_strtod.h --- a/src/njs_strtod.h Mon Aug 17 14:44:29 2020 +0300 +++ b/src/njs_strtod.h Mon Aug 17 19:55:46 2020 +0300 @@ -7,6 +7,7 @@ #ifndef _NJS_STRTOD_H_INCLUDED_ #define _NJS_STRTOD_H_INCLUDED_ -NJS_EXPORT double njs_strtod(const u_char **start, const u_char *end); +NJS_EXPORT double njs_strtod(const u_char **start, const u_char *end, + njs_bool_t literal); #endif /* _NJS_STRTOD_H_INCLUDED_ */ diff -r 10a2c35d53e7 -r 4818a450f4e6 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Mon Aug 17 14:44:29 2020 +0300 +++ b/src/test/njs_unit_test.c Mon Aug 17 19:55:46 2020 +0300 @@ -171,13 +171,13 @@ static njs_unit_test_t njs_test[] = { njs_str(".9"), njs_str("0.9") }, - { njs_str("-.01"), + { njs_str("-.0_1"), njs_str("-0.01") }, - { njs_str("0.000001"), + { njs_str("0.000_001"), njs_str("0.000001") }, - { njs_str("0.00000123456"), + { njs_str("0.00000_123456"), njs_str("0.00000123456") }, { njs_str("0.0000001"), @@ -186,10 +186,13 @@ static njs_unit_test_t njs_test[] = { njs_str("1.1000000"), njs_str("1.1") }, - { njs_str("99999999999999999999"), + { njs_str("1_0"), + njs_str("10") }, + + { njs_str("99_999_999_999_999_999_999"), njs_str("100000000000000000000") }, - { njs_str("99999999999999999999.111"), + { njs_str("9999999999999999999_9.1_1_1"), njs_str("100000000000000000000") }, { njs_str("999999999999999999999"), @@ -201,7 +204,7 @@ static njs_unit_test_t njs_test[] = { njs_str("18446744073709551616"), njs_str("18446744073709552000") }, - { njs_str("1.7976931348623157E+308"), + { njs_str("1.79769313_48623157E+3_0_8"), njs_str("1.7976931348623157e+308") }, { njs_str("+1"), @@ -213,6 +216,48 @@ static njs_unit_test_t njs_test[] = { njs_str("."), njs_str("SyntaxError: Unexpected token \".\" in 1") }, + { njs_str("0_1"), + njs_str("SyntaxError: Unexpected token \"0_\" in 1") }, + + { njs_str("1_"), + njs_str("SyntaxError: Unexpected token \"_\" in 1") }, + + { njs_str("1__0"), + njs_str("SyntaxError: Unexpected token \"__0\" in 1") }, + + { njs_str("._1"), + njs_str("SyntaxError: Unexpected token \".\" in 1") }, + + { njs_str(".1_"), + njs_str("SyntaxError: Unexpected token \"_\" in 1") }, + + { njs_str("1_.1"), + njs_str("SyntaxError: Unexpected token \"_\" in 1") }, + + { njs_str(".0__1"), + njs_str("SyntaxError: Unexpected token \"__1\" in 1") }, + + { njs_str("1e_1"), + njs_str("SyntaxError: Unexpected token \"_1\" in 1") }, + + { njs_str("1e-_1"), + njs_str("SyntaxError: Unexpected token \"_1\" in 1") }, + + { njs_str("1E1__0"), + njs_str("SyntaxError: Unexpected token \"__0\" in 1") }, + + { njs_str("1_e1"), + njs_str("SyntaxError: Unexpected token \"_e1\" in 1") }, + + { njs_str("1e1_"), + njs_str("SyntaxError: Unexpected token \"_\" in 1") }, + + { njs_str("-_1"), + njs_str("ReferenceError: \"_1\" is not defined in 1") }, + + { njs_str("_1"), + njs_str("ReferenceError: \"_1\" is not defined in 1") }, + /* Octal Numbers. */ { njs_str("0o0"), @@ -224,7 +269,7 @@ static njs_unit_test_t njs_test[] = { njs_str("0o011"), njs_str("9") }, - { njs_str("-0O777"), + { njs_str("-0O7_7_7"), njs_str("-511") }, { njs_str("0o"), @@ -233,6 +278,15 @@ static njs_unit_test_t njs_test[] = { njs_str("0O778"), njs_str("SyntaxError: Unexpected token \"0O778\" in 1") }, + { njs_str("0O_7"), + njs_str("SyntaxError: Unexpected token \"_7\" in 1") }, + + { njs_str("0o7_"), + njs_str("SyntaxError: Unexpected token \"_\" in 1") }, + + { njs_str("0o7__7"), + njs_str("SyntaxError: Unexpected token \"__7\" in 1") }, + /* Legacy Octal Numbers are deprecated. */ { njs_str("00"), @@ -247,6 +301,15 @@ static njs_unit_test_t njs_test[] = { njs_str("0011"), njs_str("SyntaxError: Unexpected token \"00\" in 1") }, + { njs_str("0_"), + njs_str("SyntaxError: Unexpected token \"0_\" in 1") }, + + { njs_str("0_1"), + njs_str("SyntaxError: Unexpected token \"0_\" in 1") }, + + { njs_str("00_1"), + njs_str("SyntaxError: Unexpected token \"00\" in 1") }, + /* Binary Numbers. */ { njs_str("0b0"), @@ -255,10 +318,10 @@ static njs_unit_test_t njs_test[] = { njs_str("0B10"), njs_str("2") }, - { njs_str("0b0101"), + { njs_str("0b0_1_0_1"), njs_str("5") }, - { njs_str("-0B11111111"), + { njs_str("-0B1111_1111"), njs_str("-255") }, { njs_str("0b"), @@ -267,6 +330,15 @@ static njs_unit_test_t njs_test[] = { njs_str("0B12"), njs_str("SyntaxError: Unexpected token \"0B12\" in 1") }, + { njs_str("0b_11"), + njs_str("SyntaxError: Unexpected token \"_11\" in 1") }, + + { njs_str("0B1__1"), + njs_str("SyntaxError: Unexpected token \"__1\" in 1") }, + + { njs_str("0b11_"), + njs_str("SyntaxError: Unexpected token \"_\" in 1") }, + /* Hex Numbers. */ { njs_str("0x0"), @@ -278,7 +350,7 @@ static njs_unit_test_t njs_test[] = { njs_str("0xffFF"), njs_str("65535") }, - { njs_str("0X0000BEEF"), + { njs_str("0X00_00_BE_EF"), njs_str("48879") }, { njs_str("0x"), @@ -290,6 +362,15 @@ static njs_unit_test_t njs_test[] = { njs_str("0x12g"), njs_str("SyntaxError: Unexpected token \"g\" in 1") }, + { njs_str("0X_ff"), + njs_str("SyntaxError: Unexpected token \"_ff\" in 1") }, + + { njs_str("0xff_"), + njs_str("SyntaxError: Unexpected token \"_\" in 1") }, + + { njs_str("0Xf__f"), + njs_str("SyntaxError: Unexpected token \"__f\" in 1") }, + { njs_str(""), njs_str("undefined") }, From dnj0496 at gmail.com Tue Aug 18 07:11:23 2020 From: dnj0496 at gmail.com (Dk Jack) Date: Tue, 18 Aug 2020 00:11:23 -0700 Subject: module context. Message-ID: Hi, I have a module. I am saving some data in my module context. However, in certain cases, I am doing an internal redirect. When I do an internal redirect my context is cleared as per the nginx development guide. Is there a way to save the context data. Since this data is specific to the request, I want to save it in the ngx_http_request_t if possible. My current solution is to save it as header values and retrieve them later after the internal redirect has happened. I am setting a special header to say, I've saved data. This is not an elegant solution but it works. The problem is, since I am saving it in headers, the header I added to save my context data is being sent to the origin as a header. After restoring my context, I tried setting the header key and value lengths to zero and also tried setting header hash to zero. None of these things seem to help and the header is still getting transmitted. Two questions: Is there a way to save module context data in ngx_http_request_t when an internal redirect happens? If not, is there a way to prevent the custom headers add by module from going out? Thanks for help. Dk. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Tue Aug 18 11:53:54 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 18 Aug 2020 11:53:54 +0000 Subject: [njs] Refactored iterator Array methods. Message-ID: details: https://hg.nginx.org/njs/rev/94499cbdf990 branches: changeset: 1496:94499cbdf990 user: Dmitry Volyntsev date: Tue Aug 18 11:52:17 2020 +0000 description: Refactored iterator Array methods. diffstat: src/njs_array.c | 1879 ++++++++++++++++++++------------------------- src/test/njs_unit_test.c | 104 +- 2 files changed, 873 insertions(+), 1110 deletions(-) diffs (truncated from 2304 to 1000 lines): diff -r 4818a450f4e6 -r 94499cbdf990 src/njs_array.c --- a/src/njs_array.c Mon Aug 17 19:55:46 2020 +0300 +++ b/src/njs_array.c Tue Aug 18 11:52:17 2020 +0000 @@ -8,7 +8,45 @@ #include -#define njs_fast_object(_sz) ((_sz) <= NJS_ARRAY_FAST_OBJECT_LENGTH) +#define njs_fast_object(_sz) ((_sz) <= NJS_ARRAY_FAST_OBJECT_LENGTH) + + +#define njs_array_func(type) \ + ((type << 1) | NJS_ARRAY_FUNC) + + +#define njs_array_arg(type) \ + ((type << 1) | NJS_ARRAY_ARG) + + +#define njs_array_type(magic) (magic >> 1) +#define njs_array_arg1(magic) (magic & 0x1) + + +typedef enum { + NJS_ARRAY_EVERY = 0, + NJS_ARRAY_SOME, + NJS_ARRAY_INCLUDES, + NJS_ARRAY_INDEX_OF, + NJS_ARRAY_FOR_EACH, + NJS_ARRAY_FIND, + NJS_ARRAY_FIND_INDEX, + NJS_ARRAY_REDUCE, + NJS_ARRAY_FILTER, + NJS_ARRAY_MAP, +} njs_array_iterator_fun_t; + + +typedef enum { + NJS_ARRAY_LAST_INDEX_OF = 0, + NJS_ARRAY_REDUCE_RIGHT, +} njs_array_reverse_iterator_fun_t; + + +typedef enum { + NJS_ARRAY_FUNC = 0, + NJS_ARRAY_ARG +} njs_array_iterator_arg_t; typedef struct { @@ -1769,380 +1807,6 @@ njs_array_indices(njs_vm_t *vm, njs_valu njs_inline njs_int_t -njs_array_object_handler(njs_vm_t *vm, njs_array_iterator_handler_t handler, - njs_array_iterator_args_t *args, njs_value_t *key, int64_t i) -{ - njs_int_t ret; - njs_value_t prop, *entry; - - if (key != NULL) { - ret = njs_value_property(vm, args->value, key, &prop); - if (njs_slow_path(ret == NJS_ERROR)) { - return ret; - } - - } else { - ret = njs_value_property_i64(vm, args->value, i, &prop); - if (njs_slow_path(ret == NJS_ERROR)) { - return ret; - } - } - - entry = (ret == NJS_OK) ? &prop : njs_value_arg(&njs_value_invalid); - - ret = handler(vm, args, entry, i); - if (njs_slow_path(ret != NJS_OK)) { - if (ret > 0) { - return NJS_DECLINED; - } - - return NJS_ERROR; - } - - return ret; -} - - -njs_inline njs_int_t -njs_array_iterator(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_array_iterator_handler_t handler) -{ - double idx; - int64_t length, i, from, to; - njs_int_t ret; - njs_array_t *array, *keys; - njs_value_t *value, *entry, prop, character, string_obj; - njs_object_t *object; - const u_char *p, *end, *pos; - njs_string_prop_t string_prop; - - value = args->value; - from = args->from; - to = args->to; - - if (njs_is_array(value)) { - array = njs_array(value); - - for (; from < to; from++) { - if (njs_slow_path(!array->object.fast_array)) { - goto process_object; - } - - if (njs_fast_path(from < array->length - && njs_is_valid(&array->start[from]))) - { - ret = handler(vm, args, &array->start[from], from); - - } else { - entry = njs_value_arg(&njs_value_invalid); - ret = njs_value_property_i64(vm, value, from, &prop); - if (njs_slow_path(ret != NJS_DECLINED)) { - if (ret == NJS_ERROR) { - return NJS_ERROR; - } - - entry = ∝ - } - - ret = handler(vm, args, entry, from); - } - - if (njs_slow_path(ret != NJS_OK)) { - if (ret > 0) { - return NJS_DECLINED; - } - - return NJS_ERROR; - } - } - - return NJS_OK; - } - - if (njs_is_string(value) || njs_is_object_string(value)) { - - if (njs_is_string(value)) { - object = njs_object_value_alloc(vm, value, NJS_STRING); - if (njs_slow_path(object == NULL)) { - return NJS_ERROR; - } - - njs_set_type_object(&string_obj, object, NJS_OBJECT_STRING); - - args->value = &string_obj; - } - else { - value = njs_object_value(value); - } - - length = njs_string_prop(&string_prop, value); - - p = string_prop.start; - end = p + string_prop.size; - - if ((size_t) length == string_prop.size) { - /* Byte or ASCII string. */ - - for (i = from; i < to; i++) { - /* This cannot fail. */ - (void) njs_string_new(vm, &character, p + i, 1, 1); - - ret = handler(vm, args, &character, i); - if (njs_slow_path(ret != NJS_OK)) { - if (ret > 0) { - return NJS_DECLINED; - } - - return NJS_ERROR; - } - } - - } else { - /* UTF-8 string. */ - - for (i = from; i < to; i++) { - pos = njs_utf8_next(p, end); - - /* This cannot fail. */ - (void) njs_string_new(vm, &character, p, pos - p, 1); - - ret = handler(vm, args, &character, i); - if (njs_slow_path(ret != NJS_OK)) { - if (ret > 0) { - return NJS_DECLINED; - } - - return NJS_ERROR; - } - - p = pos; - } - } - - return NJS_OK; - } - - if (!njs_is_object(value)) { - return NJS_OK; - } - -process_object: - - if (!njs_fast_object(to - from)) { - keys = njs_array_indices(vm, value); - if (njs_slow_path(keys == NULL)) { - return NJS_ERROR; - } - - for (i = 0; i < keys->length; i++) { - idx = njs_string_to_index(&keys->start[i]); - - if (idx < from || idx >= to) { - continue; - } - - ret = njs_array_object_handler(vm, handler, args, &keys->start[i], - idx); - if (njs_slow_path(ret != NJS_OK)) { - njs_array_destroy(vm, keys); - return ret; - } - } - - njs_array_destroy(vm, keys); - - return NJS_OK; - } - - for (i = from; i < to; i++) { - ret = njs_array_object_handler(vm, handler, args, NULL, i); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - return NJS_OK; -} - - -njs_inline njs_int_t -njs_array_reverse_iterator(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_array_iterator_handler_t handler) -{ - double idx; - int64_t i, from, to, length; - njs_int_t ret; - njs_array_t *array, *keys; - njs_value_t *entry, *value, prop, character, string_obj; - njs_object_t *object; - const u_char *p, *end, *pos; - njs_string_prop_t string_prop; - - value = args->value; - from = args->from; - to = args->to; - - if (njs_is_array(value)) { - array = njs_array(value); - - from += 1; - - while (from-- > to) { - if (njs_slow_path(!array->object.fast_array)) { - goto process_object; - } - - if (njs_fast_path(from < array->length - && njs_is_valid(&array->start[from]))) - { - ret = handler(vm, args, &array->start[from], from); - - } else { - entry = njs_value_arg(&njs_value_invalid); - ret = njs_value_property_i64(vm, value, from, &prop); - if (njs_slow_path(ret != NJS_DECLINED)) { - if (ret == NJS_ERROR) { - return NJS_ERROR; - } - - entry = ∝ - } - - ret = handler(vm, args, entry, from); - } - - if (njs_slow_path(ret != NJS_OK)) { - if (ret > 0) { - return NJS_DECLINED; - } - - return NJS_ERROR; - } - } - - return NJS_OK; - } - - if (njs_is_string(value) || njs_is_object_string(value)) { - - if (njs_is_string(value)) { - object = njs_object_value_alloc(vm, value, NJS_STRING); - if (njs_slow_path(object == NULL)) { - return NJS_ERROR; - } - - njs_set_type_object(&string_obj, object, NJS_OBJECT_STRING); - - args->value = &string_obj; - } - else { - value = njs_object_value(value); - } - - length = njs_string_prop(&string_prop, value); - end = string_prop.start + string_prop.size; - - if ((size_t) length == string_prop.size) { - /* Byte or ASCII string. */ - - p = string_prop.start + from; - - i = from + 1; - - while (i-- > to) { - /* This cannot fail. */ - (void) njs_string_new(vm, &character, p, 1, 1); - - ret = handler(vm, args, &character, i); - if (njs_slow_path(ret != NJS_OK)) { - if (ret > 0) { - return NJS_DECLINED; - } - - return NJS_ERROR; - } - - p--; - } - - } else { - /* UTF-8 string. */ - - p = njs_string_offset(string_prop.start, end, from); - p = njs_utf8_next(p, end); - - i = from + 1; - - while (i-- > to) { - pos = njs_utf8_prev(p); - - /* This cannot fail. */ - (void) njs_string_new(vm, &character, pos, p - pos , 1); - - ret = handler(vm, args, &character, i); - if (njs_slow_path(ret != NJS_OK)) { - if (ret > 0) { - return NJS_DECLINED; - } - - return NJS_ERROR; - } - - p = pos; - } - } - - return NJS_OK; - } - - if (!njs_is_object(value)) { - return NJS_OK; - } - -process_object: - - if (!njs_fast_object(from - to)) { - keys = njs_array_indices(vm, value); - if (njs_slow_path(keys == NULL)) { - return NJS_ERROR; - } - - i = keys->length; - - while (i > 0) { - idx = njs_string_to_index(&keys->start[--i]); - - if (idx < to || idx > from) { - continue; - } - - ret = njs_array_object_handler(vm, handler, args, &keys->start[i], - idx); - if (njs_slow_path(ret != NJS_OK)) { - njs_array_destroy(vm, keys); - return ret; - } - } - - njs_array_destroy(vm, keys); - - return NJS_OK; - } - - i = from + 1; - - while (i-- > to) { - ret = njs_array_object_handler(vm, handler, args, NULL, i); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - return NJS_OK; -} - - -njs_inline njs_int_t njs_is_concat_spreadable(njs_vm_t *vm, njs_value_t *value) { njs_int_t ret; @@ -2326,211 +1990,6 @@ njs_array_prototype_concat(njs_vm_t *vm, static njs_int_t -njs_array_handler_index_of(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, int64_t n) -{ - if (njs_values_strict_equal(args->argument, entry)) { - njs_set_number(&vm->retval, n); - - return 1; - } - - return NJS_OK; -} - - -static njs_int_t -njs_array_prototype_index_of(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - int64_t from, length; - njs_int_t ret; - njs_array_iterator_args_t iargs; - - iargs.value = njs_argument(args, 0); - - ret = njs_value_to_object(vm, iargs.value); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - iargs.argument = njs_arg(args, nargs, 1); - - ret = njs_value_length(vm, iargs.value, &length); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - ret = njs_value_to_integer(vm, njs_arg(args, nargs, 2), &from); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - if (length == 0 || from >= (int64_t) length) { - goto not_found; - } - - if (from < 0) { - from = length + from; - - if (from < 0) { - from = 0; - } - } - - iargs.from = from; - iargs.to = length; - - ret = njs_array_iterator(vm, &iargs, njs_array_handler_index_of); - if (njs_fast_path(ret != NJS_OK)) { - return (ret == NJS_DECLINED) ? NJS_OK : NJS_ERROR; - } - -not_found: - - njs_set_number(&vm->retval, -1); - - return ret; -} - - -static njs_int_t -njs_array_prototype_last_index_of(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) -{ - int64_t from, length; - njs_int_t ret; - njs_array_iterator_args_t iargs; - - iargs.value = njs_argument(args, 0); - - ret = njs_value_to_object(vm, iargs.value); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - iargs.argument = njs_arg(args, nargs, 1); - - ret = njs_value_length(vm, iargs.value, &length); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - if (length == 0) { - goto not_found; - } - - if (nargs > 2) { - ret = njs_value_to_integer(vm, njs_arg(args, nargs, 2), &from); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - } else { - from = length - 1; - } - - if (from >= 0) { - from = njs_min(from, length - 1); - - } else if (from < 0) { - from += length; - - if (from <= 0) { - goto not_found; - } - } - - iargs.from = from; - iargs.to = 0; - - ret = njs_array_reverse_iterator(vm, &iargs, njs_array_handler_index_of); - if (njs_fast_path(ret != NJS_OK)) { - return (ret == NJS_DECLINED) ? NJS_OK : NJS_ERROR; - } - -not_found: - - njs_set_number(&vm->retval, -1); - - return ret; -} - - -static njs_int_t -njs_array_handler_includes(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, int64_t n) -{ - if (!njs_is_valid(entry)) { - entry = njs_value_arg(&njs_value_undefined); - } - - if (njs_values_same_zero(args->argument, entry)) { - njs_set_true(&vm->retval); - - return 1; - } - - return NJS_OK; -} - - -static njs_int_t -njs_array_prototype_includes(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - int64_t from, length; - njs_int_t ret; - njs_array_iterator_args_t iargs; - - iargs.value = njs_argument(args, 0); - - ret = njs_value_to_object(vm, iargs.value); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - iargs.argument = njs_arg(args, nargs, 1); - - ret = njs_value_length(vm, iargs.value, &length); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - if (length == 0) { - goto not_found; - } - - ret = njs_value_to_integer(vm, njs_arg(args, nargs, 2), &from); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - if (from < 0) { - from += length; - - if (from < 0) { - from = 0; - } - } - - iargs.from = from; - iargs.to = length; - - ret = njs_array_iterator(vm, &iargs, njs_array_handler_includes); - if (njs_fast_path(ret != NJS_OK)) { - return (ret == NJS_DECLINED) ? NJS_OK : NJS_ERROR; - } - -not_found: - - njs_set_false(&vm->retval); - - return NJS_OK; -} - - -static njs_int_t njs_array_prototype_fill(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { @@ -2622,119 +2081,197 @@ njs_array_iterator_call(njs_vm_t *vm, nj } -njs_inline njs_int_t -njs_array_validate_args(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_array_iterator_args_t *iargs) +static njs_int_t +njs_array_object_handler(njs_vm_t *vm, njs_array_iterator_handler_t handler, + njs_array_iterator_args_t *args, njs_value_t *key, int64_t i) { - njs_int_t ret; - - iargs->value = njs_argument(args, 0); - - ret = njs_value_to_object(vm, iargs->value); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - ret = njs_value_length(vm, iargs->value, &iargs->to); - if (njs_slow_path(ret != NJS_OK)) { - return ret; + njs_int_t ret; + njs_value_t prop, *entry; + + if (key != NULL) { + ret = njs_value_property(vm, args->value, key, &prop); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + + } else { + ret = njs_value_property_i64(vm, args->value, i, &prop); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } } - if (njs_slow_path(!njs_is_function(njs_arg(args, nargs, 1)))) { - goto failed; + entry = (ret == NJS_OK) ? &prop : njs_value_arg(&njs_value_invalid); + + ret = handler(vm, args, entry, i); + if (njs_slow_path(ret != NJS_OK)) { + if (ret == NJS_DONE) { + return NJS_DONE; + } + + return NJS_ERROR; } - iargs->from = 0; - iargs->function = njs_function(njs_argument(args, 1)); - iargs->argument = njs_arg(args, nargs, 2); - - return NJS_OK; - -failed: - - njs_type_error(vm, "unexpected iterator arguments"); - - return NJS_ERROR; + return ret; } -static njs_int_t -njs_array_handler_for_each(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, int64_t n) +njs_inline njs_int_t +njs_array_iterator(njs_vm_t *vm, njs_array_iterator_args_t *args, + njs_array_iterator_handler_t handler) { - if (njs_is_valid(entry)) { - return njs_array_iterator_call(vm, args, entry, n); - } - - return NJS_OK; -} - - -static njs_int_t -njs_array_prototype_for_each(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - njs_array_iterator_args_t iargs; - - ret = njs_array_validate_args(vm, args, nargs, &iargs); - if (njs_slow_path(ret != NJS_OK)) { - return ret; + double idx; + int64_t length, i, from, to; + njs_int_t ret; + njs_array_t *array, *keys; + njs_value_t *value, *entry, prop, character, string_obj; + njs_object_t *object; + const u_char *p, *end, *pos; + njs_string_prop_t string_prop; + + value = args->value; + from = args->from; + to = args->to; + + if (njs_is_array(value)) { + array = njs_array(value); + + for (; from < to; from++) { + if (njs_slow_path(!array->object.fast_array)) { + goto process_object; + } + + if (njs_fast_path(from < array->length + && njs_is_valid(&array->start[from]))) + { + ret = handler(vm, args, &array->start[from], from); + + } else { + entry = njs_value_arg(&njs_value_invalid); + ret = njs_value_property_i64(vm, value, from, &prop); + if (njs_slow_path(ret != NJS_DECLINED)) { + if (ret == NJS_ERROR) { + return NJS_ERROR; + } + + entry = ∝ + } + + ret = handler(vm, args, entry, from); + } + + if (njs_slow_path(ret != NJS_OK)) { + if (ret == NJS_DONE) { + return NJS_DONE; + } + + return NJS_ERROR; + } + } + + return NJS_OK; } - ret = njs_array_iterator(vm, &iargs, njs_array_handler_for_each); - if (njs_slow_path(ret != NJS_OK)) { - return ret; + if (njs_is_string(value) || njs_is_object_string(value)) { + + if (njs_is_string(value)) { + object = njs_object_value_alloc(vm, value, NJS_STRING); + if (njs_slow_path(object == NULL)) { + return NJS_ERROR; + } + + njs_set_type_object(&string_obj, object, NJS_OBJECT_STRING); + + args->value = &string_obj; + } + else { + value = njs_object_value(value); + } + + length = njs_string_prop(&string_prop, value); + + p = string_prop.start; + end = p + string_prop.size; + + if ((size_t) length == string_prop.size) { + /* Byte or ASCII string. */ + + for (i = from; i < to; i++) { + /* This cannot fail. */ + (void) njs_string_new(vm, &character, p + i, 1, 1); + + ret = handler(vm, args, &character, i); + if (njs_slow_path(ret != NJS_OK)) { + if (ret == NJS_DONE) { + return NJS_DONE; + } + + return NJS_ERROR; + } + } + + } else { + /* UTF-8 string. */ + + for (i = from; i < to; i++) { + pos = njs_utf8_next(p, end); + + /* This cannot fail. */ + (void) njs_string_new(vm, &character, p, pos - p, 1); + + ret = handler(vm, args, &character, i); + if (njs_slow_path(ret != NJS_OK)) { + if (ret == NJS_DONE) { + return NJS_DONE; + } + + return NJS_ERROR; + } + + p = pos; + } + } + + return NJS_OK; } - njs_set_undefined(&vm->retval); - - return NJS_OK; -} - - -static njs_int_t -njs_array_handler_some(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, int64_t n) -{ - njs_int_t ret; - - if (njs_is_valid(entry)) { - ret = njs_array_iterator_call(vm, args, entry, n); + if (!njs_is_object(value)) { + return NJS_OK; + } + +process_object: + + if (!njs_fast_object(to - from)) { + keys = njs_array_indices(vm, value); + if (njs_slow_path(keys == NULL)) { + return NJS_ERROR; + } + + for (i = 0; i < keys->length; i++) { + idx = njs_string_to_index(&keys->start[i]); + + if (idx < from || idx >= to) { + continue; + } + + ret = njs_array_object_handler(vm, handler, args, &keys->start[i], + idx); + if (njs_slow_path(ret != NJS_OK)) { + njs_array_destroy(vm, keys); + return ret; + } + } + + njs_array_destroy(vm, keys); + + return NJS_OK; + } + + for (i = from; i < to; i++) { + ret = njs_array_object_handler(vm, handler, args, NULL, i); if (njs_slow_path(ret != NJS_OK)) { return ret; } - - if (njs_is_true(&vm->retval)) { - vm->retval = njs_value_true; - - return 1; - } - } - - return NJS_OK; -} - - -static njs_int_t -njs_array_prototype_some(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - njs_array_iterator_args_t iargs; - - ret = njs_array_validate_args(vm, args, nargs, &iargs); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - ret = njs_array_iterator(vm, &iargs, njs_array_handler_some); - if (njs_slow_path(ret == NJS_ERROR)) { - return ret; - } - - if (ret != NJS_DECLINED) { - vm->retval = njs_value_false; } return NJS_OK; @@ -2755,8 +2292,29 @@ njs_array_handler_every(njs_vm_t *vm, nj if (!njs_is_true(&vm->retval)) { vm->retval = njs_value_false; - - return 1; + return NJS_DONE; + } + } + + return NJS_OK; +} + + +static njs_int_t +njs_array_handler_some(njs_vm_t *vm, njs_array_iterator_args_t *args, + njs_value_t *entry, int64_t n) +{ + njs_int_t ret; + + if (njs_is_valid(entry)) { + ret = njs_array_iterator_call(vm, args, entry, n); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + if (njs_is_true(&vm->retval)) { + vm->retval = njs_value_true; + return NJS_DONE; } } @@ -2765,24 +2323,17 @@ njs_array_handler_every(njs_vm_t *vm, nj static njs_int_t -njs_array_prototype_every(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) +njs_array_handler_includes(njs_vm_t *vm, njs_array_iterator_args_t *args, + njs_value_t *entry, int64_t n) { - njs_int_t ret; - njs_array_iterator_args_t iargs; - - ret = njs_array_validate_args(vm, args, nargs, &iargs); - if (njs_slow_path(ret != NJS_OK)) { - return ret; + if (!njs_is_valid(entry)) { + entry = njs_value_arg(&njs_value_undefined); } - ret = njs_array_iterator(vm, &iargs, njs_array_handler_every); - if (njs_slow_path(ret == NJS_ERROR)) { - return ret; - } - - if (ret != NJS_DECLINED) { - vm->retval = njs_value_true; + if (njs_values_same_zero(args->argument, entry)) { + njs_set_true(&vm->retval); + + return NJS_DONE; From xeioex at nginx.com Tue Aug 18 16:54:44 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 18 Aug 2020 16:54:44 +0000 Subject: [njs] Fixed function "prototype" property handler while setting. Message-ID: details: https://hg.nginx.org/njs/rev/c9d18ab3eb85 branches: changeset: 1497:c9d18ab3eb85 user: Dmitry Volyntsev date: Tue Aug 18 16:53:46 2020 +0000 description: Fixed function "prototype" property handler while setting. njs_function_prototype_create() works as a getter and setter. As a getter the function is expected to create "prototype" property on the first access, it also sets F.prototype.constructor property to ensure F.prototype.constructor === F. Setting of "constructor" property is not needed in setter context, as it may overwrite existing "constructor" property in setval. This closes #333 issue on Github. diffstat: src/njs_function.c | 9 +++++---- src/test/njs_unit_test.c | 10 ++++++++++ 2 files changed, 15 insertions(+), 4 deletions(-) diffs (50 lines): diff -r 94499cbdf990 -r c9d18ab3eb85 src/njs_function.c --- a/src/njs_function.c Tue Aug 18 11:52:17 2020 +0000 +++ b/src/njs_function.c Tue Aug 18 16:53:46 2020 +0000 @@ -776,7 +776,7 @@ njs_function_frame_free(njs_vm_t *vm, nj static njs_value_t * -njs_function_property_prototype_create(njs_vm_t *vm, njs_lvlhsh_t *hash, +njs_function_property_prototype_set(njs_vm_t *vm, njs_lvlhsh_t *hash, njs_value_t *prototype) { njs_int_t ret; @@ -845,13 +845,14 @@ njs_function_prototype_create(njs_vm_t * return NJS_ERROR; } - proto = njs_function_property_prototype_create(vm, &function->object.hash, - setval); + proto = njs_function_property_prototype_set(vm, njs_object_hash(value), + setval); if (njs_slow_path(proto == NULL)) { return NJS_ERROR; } - if (njs_is_object(proto)) { + if (setval == &proto_value && njs_is_object(proto)) { + /* Only in getter context. */ cons = njs_property_constructor_create(vm, njs_object_hash(proto), value); if (njs_slow_path(cons == NULL)) { diff -r 94499cbdf990 -r c9d18ab3eb85 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Tue Aug 18 11:52:17 2020 +0000 +++ b/src/test/njs_unit_test.c Tue Aug 18 16:53:46 2020 +0000 @@ -9393,6 +9393,16 @@ static njs_unit_test_t njs_test[] = "x.t == 1 && y.t == 2"), njs_str("true") }, + { njs_str("function A(){}; A.tag = 'A'; var a = new A();" + "(function B(){}).prototype = A.prototype;" + "a.constructor.tag"), + njs_str("A") }, + + { njs_str("function A(){}; A.tag = 'A'; var a = new A();" + "(function B(){}).prototype = a.constructor.prototype;" + "a.constructor.tag"), + njs_str("A") }, + { njs_str("var x = {}, y = function() {}, z; y.prototype = x; z = new y();" "(z instanceof y) && (z.__proto__ == y.prototype) && (x.isPrototypeOf(z))"), njs_str("true") }, From xeioex at nginx.com Wed Aug 19 12:44:23 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 19 Aug 2020 12:44:23 +0000 Subject: [njs] Fixed njs_property_query() while quering symbol keys. Message-ID: details: https://hg.nginx.org/njs/rev/15ae88d60ca4 branches: changeset: 1498:15ae88d60ca4 user: Dmitry Volyntsev date: Wed Aug 19 12:43:23 2020 +0000 description: Fixed njs_property_query() while quering symbol keys. Since 37f1f20de909, njs_object_hash_test() requires lhq->key.start to be NULL to avoid hash collisions. The field is zeroed by njs_property_query_init(), but the field still might be non-zero when njs_property_query() is called inside a loop. The fix is to ensure lhq->key.start == NULL for symbol keys. This closes #268 issue on Github. diffstat: src/njs_value.c | 1 + src/test/njs_unit_test.c | 4 ++++ 2 files changed, 5 insertions(+), 0 deletions(-) diffs (25 lines): diff -r c9d18ab3eb85 -r 15ae88d60ca4 src/njs_value.c --- a/src/njs_value.c Tue Aug 18 16:53:46 2020 +0000 +++ b/src/njs_value.c Wed Aug 19 12:43:23 2020 +0000 @@ -568,6 +568,7 @@ njs_property_query(njs_vm_t *vm, njs_pro if (njs_is_symbol(key)) { pq->lhq.key_hash = njs_symbol_key(key); + pq->lhq.key.start = NULL; } else { njs_string_get(&pq->key, &pq->lhq.key); diff -r c9d18ab3eb85 -r 15ae88d60ca4 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Tue Aug 18 16:53:46 2020 +0000 +++ b/src/test/njs_unit_test.c Wed Aug 19 12:43:23 2020 +0000 @@ -13151,6 +13151,10 @@ static njs_unit_test_t njs_test[] = "[m.hasOwnProperty('abs'), delete m.abs, m.abs(-1)]"), njs_str("true,true,1") }, + { njs_str("var Q = Object.create({}, {a: {value: 'AAA'}, [Symbol.toStringTag]:{value: 'TAG'}});" + "Q[Symbol.toStringTag]"), + njs_str("TAG") }, + { njs_str("Object.getOwnPropertyDescriptor({a:1}, 'a').value"), njs_str("1") }, From eran.kornblau at kaltura.com Wed Aug 19 18:02:29 2020 From: eran.kornblau at kaltura.com (Eran Kornblau) Date: Wed, 19 Aug 2020 18:02:29 +0000 Subject: Exposing additional stub status metrics Message-ID: Hi all, I want to configure Nginx to output the stub status metrics in the format used by Prometheus. My thought was to simply use the builtin 'return' directive for this, however, I noticed that only 4 out of the 7 metrics are currently exposed as variables. I would like to submit a patch to expose the other 3... Does this make sense? Will the patch be accepted (assuming it follows the coding standards...)? Thank you Eran -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Fri Aug 21 13:08:35 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 21 Aug 2020 13:08:35 +0000 Subject: [njs] Added remaining methods for %TypedArray%.prototype. Message-ID: details: https://hg.nginx.org/njs/rev/95ec4ee01853 branches: changeset: 1499:95ec4ee01853 user: Dmitry Volyntsev date: Fri Aug 21 13:07:48 2020 +0000 description: Added remaining methods for %TypedArray%.prototype. The following methods were added: every(), filter(), find(), findIndex(), forEach(), includes(), indexOf(), lastIndexOf(), map(), reduce(), reduceRight(), reverse(), some(). diffstat: src/njs_typed_array.c | 714 +++++++++++++++++++++++++++++++++++++++++++++- src/njs_utils.c | 56 --- src/njs_utils.h | 57 +++ src/test/njs_unit_test.c | 391 ++++++++++++++++++++++--- 4 files changed, 1094 insertions(+), 124 deletions(-) diffs (truncated from 1329 to 1000 lines): diff -r 15ae88d60ca4 -r 95ec4ee01853 src/njs_typed_array.c --- a/src/njs_typed_array.c Wed Aug 19 12:43:23 2020 +0000 +++ b/src/njs_typed_array.c Fri Aug 21 13:07:48 2020 +0000 @@ -8,6 +8,17 @@ #include +typedef enum { + NJS_ARRAY_EVERY = 0, + NJS_ARRAY_FOR_EACH, + NJS_ARRAY_SOME, + NJS_ARRAY_FIND, + NJS_ARRAY_FIND_INDEX, + NJS_ARRAY_FILTER, + NJS_ARRAY_MAP, +} njs_array_iterator_fun_t; + + njs_typed_array_t * njs_typed_array_alloc(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_object_type_t type) @@ -826,6 +837,558 @@ njs_typed_array_prototype_copy_within(nj } +static njs_int_t +njs_typed_array_prototype_iterator(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t type) +{ + double val; + int64_t i, length; + njs_int_t ret; + njs_arr_t results; + njs_value_t *this, *this_arg, *r; + njs_value_t arguments[4], retval; + njs_function_t *function; + njs_typed_array_t *array, *dst; + + this = njs_argument(args, 0); + if (njs_slow_path(!njs_is_typed_array(this))) { + njs_type_error(vm, "this is not a typed array"); + return NJS_ERROR; + } + + dst = NULL; + array = njs_typed_array(this); + length = njs_typed_array_length(array); + + if (njs_slow_path(!njs_is_function(njs_arg(args, nargs, 1)))) { + njs_type_error(vm, "callback argument is not callable"); + return NJS_ERROR; + } + + function = njs_function(njs_argument(args, 1)); + this_arg = njs_arg(args, nargs, 2); + + results.separate = 0; + results.pointer = 0; + + switch (type) { + case NJS_ARRAY_MAP: + njs_set_number(&arguments[0], length); + ret = njs_typed_array_species_create(vm, this, njs_value_arg(arguments), + 1, &retval); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + + dst = njs_typed_array(&retval); + break; + + case NJS_ARRAY_FILTER: + r = njs_arr_init(vm->mem_pool, &results, NULL, 4, sizeof(njs_value_t)); + if (njs_slow_path(r == NULL)) { + return NJS_ERROR; + } + + default: + break; + } + + for (i = 0; i < length; i++) { + val = njs_typed_array_get(array, i); + + arguments[0] = *this_arg; + njs_set_number(&arguments[1], val); + njs_set_number(&arguments[2], i); + njs_set_typed_array(&arguments[3], array); + + ret = njs_function_apply(vm, function, arguments, 4, &vm->retval); + if (njs_slow_path(ret != NJS_OK)) { + goto exception; + } + + switch (type) { + case NJS_ARRAY_EVERY: + if (!njs_is_true(&vm->retval)) { + vm->retval = njs_value_false; + goto done; + } + + break; + + case NJS_ARRAY_FOR_EACH: + break; + + case NJS_ARRAY_SOME: + case NJS_ARRAY_FIND: + case NJS_ARRAY_FIND_INDEX: + if (!njs_is_true(&vm->retval)) { + continue; + } + + switch (type) { + case NJS_ARRAY_SOME: + vm->retval = njs_value_true; + break; + + case NJS_ARRAY_FIND: + njs_set_number(&vm->retval, val); + break; + + default: + njs_set_number(&vm->retval, i); + break; + } + + goto done; + + case NJS_ARRAY_MAP: + ret = njs_typed_array_set_value(vm, dst, i, &vm->retval); + if (njs_slow_path(ret != NJS_OK)) { + goto exception; + } + + break; + + default: + /* NJS_ARRAY_FILTER. */ + + if (!njs_is_true(&vm->retval)) { + continue; + } + + r = njs_arr_add(&results); + if (njs_slow_path(r == NULL)) { + goto exception; + } + + njs_set_number(r, val); + } + } + + /* Default values. */ + + switch (type) { + case NJS_ARRAY_EVERY: + vm->retval = njs_value_true; + break; + + case NJS_ARRAY_SOME: + vm->retval = njs_value_false; + break; + + case NJS_ARRAY_FOR_EACH: + case NJS_ARRAY_FIND: + njs_set_undefined(&vm->retval); + break; + + case NJS_ARRAY_FIND_INDEX: + njs_set_number(&vm->retval, -1); + break; + + case NJS_ARRAY_MAP: + case NJS_ARRAY_FILTER: + default: + if (type == NJS_ARRAY_FILTER) { + njs_set_number(&arguments[0], results.items); + ret = njs_typed_array_species_create(vm, this, + njs_value_arg(arguments), + 1, &retval); + if (njs_slow_path(ret != NJS_OK)) { + goto exception; + } + + dst = njs_typed_array(&retval); + + i = 0; + + while (i < results.items) { + r = njs_arr_item(&results, i); + ret = njs_typed_array_set_value(vm, dst, i++, r); + if (njs_slow_path(ret != NJS_OK)) { + goto exception; + } + } + } + + njs_set_typed_array(&vm->retval, dst); + break; + } + +done: + + ret = NJS_OK; + +exception: + + njs_arr_destroy(&results); + + return ret; +} + + +static njs_int_t +njs_typed_array_prototype_index_of(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t type) +{ + double v; + int64_t i, i64, from, to, index, increment, offset, length; + njs_int_t ret, integer; + njs_value_t *this; + const float *f32; + const double *f64; + const uint8_t *u8; + const uint16_t *u16; + const uint32_t *u32; + njs_typed_array_t *array; + njs_array_buffer_t *buffer; + + this = njs_argument(args, 0); + if (njs_slow_path(!njs_is_typed_array(this))) { + njs_type_error(vm, "this is not a typed array"); + return NJS_ERROR; + } + + index = -1; + array = njs_typed_array(this); + length = njs_typed_array_length(array); + + if (!njs_is_number(njs_arg(args, nargs, 1)) || length == 0) { + goto done; + } + + if (type & 2) { + /* lastIndexOf(). */ + + if (nargs > 2) { + ret = njs_value_to_integer(vm, njs_arg(args, nargs, 2), &from); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + } else { + from = length - 1; + } + + if (from >= 0) { + from = njs_min(from, length - 1); + + } else if (from < 0) { + from += length; + } + + to = -1; + increment = -1; + + if (from <= to) { + goto done; + } + + } else { + /* indexOf(), includes(). */ + + ret = njs_value_to_integer(vm, njs_arg(args, nargs, 2), &from); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + if (from < 0) { + from += length; + + if (from < 0) { + from = 0; + } + } + + to = length; + increment = 1; + + if (from >= to) { + goto done; + } + } + + v = njs_number(njs_argument(args, 1)); + + i64 = v; + integer = (v == i64); + + buffer = array->buffer; + offset = array->offset; + + switch (array->type) { + case NJS_OBJ_TYPE_INT8_ARRAY: + if (integer && ((int8_t) i64 == i64)) { + goto search8; + } + + break; + + case NJS_OBJ_TYPE_UINT8_CLAMPED_ARRAY: + case NJS_OBJ_TYPE_UINT8_ARRAY: + if (integer && ((uint8_t) i64 == i64)) { +search8: + u8 = &buffer->u.u8[0]; + for (i = from; i != to; i += increment) { + if (u8[offset + i] == (uint8_t) i64) { + index = i; + break; + } + } + } + + break; + + case NJS_OBJ_TYPE_INT16_ARRAY: + if (integer && ((int16_t) i64 == i64)) { + goto search16; + } + + break; + + case NJS_OBJ_TYPE_UINT16_ARRAY: + if (integer && ((uint16_t) i64 == i64)) { +search16: + u16 = &buffer->u.u16[0]; + for (i = from; i != to; i += increment) { + if (u16[offset + i] == (uint16_t) i64) { + index = i; + break; + } + } + } + + break; + + case NJS_OBJ_TYPE_INT32_ARRAY: + if (integer && ((int32_t) i64 == i64)) { + goto search32; + } + + break; + + case NJS_OBJ_TYPE_UINT32_ARRAY: + if (integer && ((uint32_t) i64 == i64)) { +search32: + u32 = &buffer->u.u32[0]; + for (i = from; i != to; i += increment) { + if (u32[offset + i] == (uint32_t) i64) { + index = i; + break; + } + } + } + + break; + + case NJS_OBJ_TYPE_FLOAT32_ARRAY: + f32 = &buffer->u.f32[0]; + + if (((float) v == v)) { + for (i = from; i != to; i += increment) { + if (f32[offset + i] == (float) v) { + index = i; + break; + } + } + + } else if ((type & 1) && isnan(v)) { + /* includes() handles NaN. */ + + for (i = from; i != to; i += increment) { + if (isnan(f32[offset + i])) { + index = i; + break; + } + } + } + + break; + + default: + + /* NJS_OBJ_TYPE_FLOAT64_ARRAY. */ + + f64 = &buffer->u.f64[0]; + + if ((type & 1) && isnan(v)) { + /* includes() handles NaN. */ + + for (i = from; i != to; i += increment) { + if (isnan(f64[offset + i])) { + index = i; + break; + } + } + + } else { + for (i = from; i != to; i += increment) { + if (f64[offset + i] == v) { + index = i; + break; + } + } + } + } + +done: + + /* Default values. */ + + if (type & 1) { + njs_set_boolean(&vm->retval, index != -1); + + } else { + njs_set_number(&vm->retval, index); + } + + return NJS_OK; +} + + +static njs_int_t +njs_typed_array_prototype_reduce(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t right) +{ + int64_t i, from, to, increment, length; + njs_int_t ret; + njs_value_t *this, accumulator; + njs_value_t arguments[5]; + njs_function_t *function; + njs_typed_array_t *array; + + this = njs_argument(args, 0); + if (njs_slow_path(!njs_is_typed_array(this))) { + njs_type_error(vm, "this is not a typed array"); + return NJS_ERROR; + } + + array = njs_typed_array(this); + length = njs_typed_array_length(array); + + if (njs_slow_path(!njs_is_function(njs_arg(args, nargs, 1)))) { + njs_type_error(vm, "callback argument is not callable"); + return NJS_ERROR; + } + + function = njs_function(njs_argument(args, 1)); + + if (length == 0 && nargs <= 2) { + njs_type_error(vm, "Reduce of empty object with no initial value"); + return NJS_ERROR; + } + + if (right) { + from = length - 1; + to = -1; + increment = -1; + + } else { + from = 0; + to = length; + increment = 1; + } + + if (nargs > 2) { + accumulator = *njs_argument(args, 2); + + } else { + njs_set_number(&accumulator, njs_typed_array_get(array, from)); + from += increment; + } + + for (i = from; i != to; i += increment) { + njs_set_undefined(&arguments[0]); + arguments[1] = accumulator; + njs_set_number(&arguments[2], njs_typed_array_get(array, i)); + njs_set_number(&arguments[3], i); + njs_set_typed_array(&arguments[4], array); + + ret = njs_function_apply(vm, function, arguments, 5, &accumulator); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + } + + vm->retval = accumulator; + + return NJS_OK; +} + + +static njs_int_t +njs_typed_array_prototype_reverse(njs_vm_t *vm, njs_value_t *args, + njs_uint_t nargs, njs_index_t unused) +{ + double *f64; + uint8_t *u8; + int64_t i, length; + uint16_t *u16; + uint32_t *u32; + njs_value_t *this; + njs_typed_array_t *array; + njs_array_buffer_t *buffer; + + this = njs_argument(args, 0); + if (njs_slow_path(!njs_is_typed_array(this))) { + njs_type_error(vm, "this is not a typed array"); + return NJS_ERROR; + } + + array = njs_typed_array(this); + length = njs_typed_array_length(array); + + buffer = array->buffer; + + switch (array->type) { + case NJS_OBJ_TYPE_UINT8_ARRAY: + case NJS_OBJ_TYPE_UINT8_CLAMPED_ARRAY: + case NJS_OBJ_TYPE_INT8_ARRAY: + u8 = &buffer->u.u8[array->offset]; + + for (i = 0; i < length / 2; i++) { + njs_swap_u8(&u8[i], &u8[length - i - 1], 0); + } + + break; + + case NJS_OBJ_TYPE_INT16_ARRAY: + case NJS_OBJ_TYPE_UINT16_ARRAY: + u16 = &buffer->u.u16[array->offset]; + + for (i = 0; i < length / 2; i++) { + njs_swap_u16(&u16[i], &u16[length - i - 1], 0); + } + + break; + + case NJS_OBJ_TYPE_INT32_ARRAY: + case NJS_OBJ_TYPE_UINT32_ARRAY: + case NJS_OBJ_TYPE_FLOAT32_ARRAY: + u32 = &buffer->u.u32[array->offset]; + + for (i = 0; i < length / 2; i++) { + njs_swap_u32(&u32[i], &u32[length - i - 1], 0); + } + + break; + + default: + + /* NJS_OBJ_TYPE_FLOAT64_ARRAY. */ + + f64 = &buffer->u.f64[array->offset]; + + for (i = 0; i < length / 2; i++) { + njs_swap_u64(&f64[i], &f64[length - i - 1], 0); + } + } + + njs_set_typed_array(&vm->retval, array); + + return NJS_OK; +} + + static int njs_typed_array_compare_i8(const void *a, const void *b, void *c) { @@ -1367,6 +1930,132 @@ static const njs_object_prop_t njs_type { .type = NJS_PROPERTY, + .name = njs_string("copyWithin"), + .value = njs_native_function(njs_typed_array_prototype_copy_within, 2), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("every"), + .value = njs_native_function2(njs_typed_array_prototype_iterator, 1, + NJS_ARRAY_EVERY), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("filter"), + .value = njs_native_function2(njs_typed_array_prototype_iterator, 1, + NJS_ARRAY_FILTER), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("find"), + .value = njs_native_function2(njs_typed_array_prototype_iterator, 1, + NJS_ARRAY_FIND), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("findIndex"), + .value = njs_native_function2(njs_typed_array_prototype_iterator, 1, + NJS_ARRAY_FIND_INDEX), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("forEach"), + .value = njs_native_function2(njs_typed_array_prototype_iterator, 1, + NJS_ARRAY_FOR_EACH), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("includes"), + .value = njs_native_function2(njs_typed_array_prototype_index_of, 1, 1), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("indexOf"), + .value = njs_native_function2(njs_typed_array_prototype_index_of, 1, 0), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("join"), + .value = njs_native_function(njs_typed_array_prototype_join, 1), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("fill"), + .value = njs_native_function(njs_typed_array_prototype_fill, 1), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("lastIndexOf"), + .value = njs_native_function2(njs_typed_array_prototype_index_of, 1, 2), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("map"), + .value = njs_native_function2(njs_typed_array_prototype_iterator, 1, + NJS_ARRAY_MAP), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("reduce"), + .value = njs_native_function2(njs_typed_array_prototype_reduce, 1, 0), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("reduceRight"), + .value = njs_native_function2(njs_typed_array_prototype_reduce, 1, 1), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("reverse"), + .value = njs_native_function(njs_typed_array_prototype_reverse, 0), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, .name = njs_string("set"), .value = njs_native_function(njs_typed_array_prototype_set, 2), .writable = 1, @@ -1383,8 +2072,9 @@ static const njs_object_prop_t njs_type { .type = NJS_PROPERTY, - .name = njs_string("subarray"), - .value = njs_native_function2(njs_typed_array_prototype_slice, 2, 0), + .name = njs_string("some"), + .value = njs_native_function2(njs_typed_array_prototype_iterator, 1, + NJS_ARRAY_SOME), .writable = 1, .configurable = 1, }, @@ -1399,24 +2089,8 @@ static const njs_object_prop_t njs_type { .type = NJS_PROPERTY, - .name = njs_string("copyWithin"), - .value = njs_native_function(njs_typed_array_prototype_copy_within, 2), - .writable = 1, - .configurable = 1, - }, - - { - .type = NJS_PROPERTY, - .name = njs_string("fill"), - .value = njs_native_function(njs_typed_array_prototype_fill, 1), - .writable = 1, - .configurable = 1, - }, - - { - .type = NJS_PROPERTY, - .name = njs_string("join"), - .value = njs_native_function(njs_typed_array_prototype_join, 1), + .name = njs_string("subarray"), + .value = njs_native_function2(njs_typed_array_prototype_slice, 2, 0), .writable = 1, .configurable = 1, }, diff -r 15ae88d60ca4 -r 95ec4ee01853 src/njs_utils.c --- a/src/njs_utils.c Wed Aug 19 12:43:23 2020 +0000 +++ b/src/njs_utils.c Fri Aug 21 13:07:48 2020 +0000 @@ -12,62 +12,6 @@ typedef void (*njs_swap_t) (void *a, voi njs_inline void -njs_swap_u8(void *a, void *b, size_t size) -{ - uint8_t u, *au, *bu; - - au = (uint8_t *) a; - bu = (uint8_t *) b; - - u = au[0]; - au[0] = bu[0]; - bu[0] = u; -} - - -njs_inline void -njs_swap_u16(void *a, void *b, size_t size) -{ - uint16_t u, *au, *bu; - - au = (uint16_t *) a; - bu = (uint16_t *) b; - - u = au[0]; - au[0] = bu[0]; - bu[0] = u; -} - - -njs_inline void -njs_swap_u32(void *a, void *b, size_t size) -{ - uint32_t u, *au, *bu; - - au = (uint32_t *) a; - bu = (uint32_t *) b; - - u = au[0]; - au[0] = bu[0]; - bu[0] = u; -} - - -njs_inline void -njs_swap_u64(void *a, void *b, size_t size) -{ - uint64_t u, *au, *bu; - - au = (uint64_t *) a; - bu = (uint64_t *) b; - - u = au[0]; - au[0] = bu[0]; - bu[0] = u; -} - - -njs_inline void njs_swap_u128(void *a, void *b, size_t size) { uint64_t u, v, *au, *bu; diff -r 15ae88d60ca4 -r 95ec4ee01853 src/njs_utils.h --- a/src/njs_utils.h Wed Aug 19 12:43:23 2020 +0000 +++ b/src/njs_utils.h Fri Aug 21 13:07:48 2020 +0000 @@ -15,4 +15,61 @@ void njs_qsort(void *base, size_t n, siz const char *njs_errno_string(int errnum); + +njs_inline void +njs_swap_u8(void *a, void *b, size_t size) +{ + uint8_t u, *au, *bu; + + au = (uint8_t *) a; + bu = (uint8_t *) b; + + u = au[0]; + au[0] = bu[0]; + bu[0] = u; +} + + +njs_inline void +njs_swap_u16(void *a, void *b, size_t size) +{ + uint16_t u, *au, *bu; + + au = (uint16_t *) a; + bu = (uint16_t *) b; + + u = au[0]; + au[0] = bu[0]; + bu[0] = u; +} + + +njs_inline void +njs_swap_u32(void *a, void *b, size_t size) +{ + uint32_t u, *au, *bu; + + au = (uint32_t *) a; + bu = (uint32_t *) b; + + u = au[0]; + au[0] = bu[0]; + bu[0] = u; +} + + +njs_inline void +njs_swap_u64(void *a, void *b, size_t size) +{ + uint64_t u, *au, *bu; + + au = (uint64_t *) a; + bu = (uint64_t *) b; + + u = au[0]; + au[0] = bu[0]; + bu[0] = u; +} + + #endif /* _NJS_UTILS_H_INCLUDED_ */ diff -r 15ae88d60ca4 -r 95ec4ee01853 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Wed Aug 19 12:43:23 2020 +0000 +++ b/src/test/njs_unit_test.c Fri Aug 21 13:07:48 2020 +0000 @@ -5895,6 +5895,312 @@ static njs_unit_test_t njs_test[] = " return a.toString() === '4,5,3,4,5'})"), njs_str("true") }, + { njs_str("Uint8Array.prototype.every.call(1)"), + njs_str("TypeError: this is not a typed array") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v([1,2,3])).every(e=>e>0) === true})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v([1,2,3])).every(function(e) {" + " if (this != undefined) {throw 'Oops';}" + " return e > 0}) === true})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v([1,2,3])).every(function(e) {" + " if (this != 'QQ') {throw 'Oops';}" + " return e > 0}, 'QQ') === true})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v([1,2,3])).every(e=>e>1) === false})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var orig = new v([255,255,1,2,3,255]);" + " var a = new v(orig.buffer, 2 * v.BYTES_PER_ELEMENT, 3);" + " return a.every(e=>e<4)})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var collect = []; (new v([42,43])).forEach(e=>collect.push(e)); " + " return collect.join('|') === '42|43'})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v([7,10,3,8,5])).filter(q=>q%2).join('|') === '7|3|5'})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var orig = new v([255,255,7,10,3,8,5,255]);" + " var a = new v(orig.buffer, 2 * v.BYTES_PER_ELEMENT, 5);" + " return a.filter(q=>q%2).join('|') === '7|3|5'})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var orig = new v([255,255,1,2,3,255]);" + " var a = new v(orig.buffer, 2 * v.BYTES_PER_ELEMENT, 3);" + " return a.find(e=>e>2) === 3})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var orig = new v([255,255,1,2,3,255]);" + " var a = new v(orig.buffer, 2 * v.BYTES_PER_ELEMENT, 3);" + " return a.find(e=>e===255) === undefined})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var orig = new v([255,255,1,2,3,255]);" + " var a = new v(orig.buffer, 2 * v.BYTES_PER_ELEMENT, 3);" + " return a.findIndex(e=>e>2) === 2})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var orig = new v([255,255,1,2,3,255]);" + " var a = new v(orig.buffer, 2 * v.BYTES_PER_ELEMENT, 3);" + " return a.findIndex(e=>e===255) === -1})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v([1,2,3])).some(e=>e==2)})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var orig = new v([255,255,1,2,3,255]);" + " var a = new v(orig.buffer, 2 * v.BYTES_PER_ELEMENT, 3);" + " return a.some(e=>e==255)})"), + njs_str("false") }, + + { njs_str("Uint8Array.prototype.includes.call(1)"), + njs_str("TypeError: this is not a typed array") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v()).includes(0, {valueOf(){throw 'Oops'}}) === false})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v([0,1,2,3])).includes(2) === true})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v([0,1,2,3])).includes(2,3) === false})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var a = new v(5);" + " return a.includes(0, 4) === true " + " && a.includes(0, 5) === false;})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{return (new v([0,1,2,3])).includes(-0) === true})"), + njs_str("true") }, + + { njs_str(NJS_FLOAT_TYPED_ARRAY_LIST + ".every(v=>{return (new v([42, 43, NaN, 41])).includes(NaN) === true})"), + njs_str("true") }, + + { njs_str(NJS_TYPED_ARRAY_LIST + ".every(v=>{var orig = new v([255,255,0,2,3,255]);" + " var a = new v(orig.buffer, 2 * v.BYTES_PER_ELEMENT, 3);" From xeioex at nginx.com Fri Aug 21 13:08:37 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 21 Aug 2020 13:08:37 +0000 Subject: [njs] Improved njs.dump() with built-in functions. Message-ID: details: https://hg.nginx.org/njs/rev/775c81dcbe61 branches: changeset: 1500:775c81dcbe61 user: Dmitry Volyntsev date: Fri Aug 21 13:07:54 2020 +0000 description: Improved njs.dump() with built-in functions. diffstat: src/njs_json.c | 22 ++++++++++++++++++++-- src/test/njs_unit_test.c | 14 ++++++++++++++ 2 files changed, 34 insertions(+), 2 deletions(-) diffs (63 lines): diff -r 95ec4ee01853 -r 775c81dcbe61 src/njs_json.c --- a/src/njs_json.c Fri Aug 21 13:07:48 2020 +0000 +++ b/src/njs_json.c Fri Aug 21 13:07:54 2020 +0000 @@ -1830,6 +1830,8 @@ njs_dump_terminal(njs_json_stringify_t * njs_typed_array_t *array; njs_string_prop_t string; + static const njs_value_t name_string = njs_string("name"); + njs_int_t (*to_string)(njs_vm_t *, njs_value_t *, const njs_value_t *); switch (value->type) { @@ -1933,8 +1935,24 @@ njs_dump_terminal(njs_json_stringify_t * break; case NJS_FUNCTION: - if (njs_function(value)->native) { - njs_chb_append_literal(chain, "[Function: native]"); + ret = njs_value_property(stringify->vm, value, + njs_value_arg(&name_string), &tag); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + + if (njs_is_string(&tag)) { + njs_string_get(&tag, &str); + + } else if (njs_function(value)->native) { + str = njs_str_value("native"); + + } else { + str = njs_str_value(""); + } + + if (str.length != 0) { + njs_chb_sprintf(chain, 32 + str.length, "[Function: %V]", &str); } else { njs_chb_append_literal(chain, "[Function]"); diff -r 95ec4ee01853 -r 775c81dcbe61 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Fri Aug 21 13:07:48 2020 +0000 +++ b/src/test/njs_unit_test.c Fri Aug 21 13:07:54 2020 +0000 @@ -16747,6 +16747,20 @@ static njs_unit_test_t njs_test[] = { njs_str("njs.dump(Object(Symbol.iterator))"), njs_str("[Symbol: Symbol(Symbol.iterator)]") }, + { njs_str("njs.dump(decodeURI)"), + njs_str("[Function: decodeURI]") }, + + { njs_str("delete decodeURI.name; njs.dump(decodeURI)"), + njs_str("[Function]") }, + + { njs_str("delete decodeURI.name; delete Function.prototype.name; " + "decodeURI.name = 1; njs.dump(decodeURI)"), + njs_str("[Function: native]") }, + + { njs_str("delete decodeURI.name; delete Function.prototype.name; " + "decodeURI.name = 'XXX'; njs.dump(decodeURI)"), + njs_str("[Function: XXX]") }, + /* Built-in methods name. */ { njs_str( From xeioex at nginx.com Tue Aug 25 12:08:06 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 25 Aug 2020 12:08:06 +0000 Subject: [njs] Fixed function "constructor" property handler while setting. Message-ID: details: https://hg.nginx.org/njs/rev/85bfd770a3da branches: changeset: 1501:85bfd770a3da user: Dmitry Volyntsev date: Mon Aug 24 11:28:21 2020 +0000 description: Fixed function "constructor" property handler while setting. diffstat: src/njs_function.c | 3 +-- src/njs_object.c | 38 +++++++++++++++++++++++++------------- src/njs_object.h | 2 +- 3 files changed, 27 insertions(+), 16 deletions(-) diffs (93 lines): diff -r 775c81dcbe61 -r 85bfd770a3da src/njs_function.c --- a/src/njs_function.c Fri Aug 21 13:07:54 2020 +0000 +++ b/src/njs_function.c Mon Aug 24 11:28:21 2020 +0000 @@ -853,8 +853,7 @@ njs_function_prototype_create(njs_vm_t * if (setval == &proto_value && njs_is_object(proto)) { /* Only in getter context. */ - cons = njs_property_constructor_create(vm, njs_object_hash(proto), - value); + cons = njs_property_constructor_set(vm, njs_object_hash(proto), value); if (njs_slow_path(cons == NULL)) { return NJS_ERROR; } diff -r 775c81dcbe61 -r 85bfd770a3da src/njs_object.c --- a/src/njs_object.c Fri Aug 21 13:07:54 2020 +0000 +++ b/src/njs_object.c Mon Aug 24 11:28:21 2020 +0000 @@ -2216,6 +2216,22 @@ njs_object_prototype_create_constructor( njs_object_t *object; njs_object_prototype_t *prototype; + if (setval != NULL) { + if (!njs_is_object(value)) { + njs_type_error(vm, "Cannot create propery \"constructor\" on %s", + njs_type_string(value->type)); + return NJS_ERROR; + } + + cons = njs_property_constructor_set(vm, njs_object_hash(value), setval); + if (njs_slow_path(cons == NULL)) { + return NJS_ERROR; + } + + *retval = *cons; + return NJS_OK; + } + if (njs_is_object(value)) { object = njs_object(value); @@ -2231,8 +2247,6 @@ njs_object_prototype_create_constructor( } while (object != NULL); - njs_thread_log_alert("prototype not found"); - return NJS_ERROR; } else { @@ -2242,23 +2256,21 @@ njs_object_prototype_create_constructor( found: - if (setval == NULL) { - njs_set_function(&constructor, &vm->constructors[index]); - setval = &constructor; + njs_set_function(&constructor, &vm->constructors[index]); + setval = &constructor; + + cons = njs_property_constructor_set(vm, &prototype->object.hash, setval); + if (njs_slow_path(cons == NULL)) { + return NJS_ERROR; } - cons = njs_property_constructor_create(vm, &prototype->object.hash, setval); - if (njs_fast_path(cons != NULL)) { - *retval = *cons; - return NJS_OK; - } - - return NJS_ERROR; + *retval = *cons; + return NJS_OK; } njs_value_t * -njs_property_constructor_create(njs_vm_t *vm, njs_lvlhsh_t *hash, +njs_property_constructor_set(njs_vm_t *vm, njs_lvlhsh_t *hash, njs_value_t *constructor) { njs_int_t ret; diff -r 775c81dcbe61 -r 85bfd770a3da src/njs_object.h --- a/src/njs_object.h Fri Aug 21 13:07:54 2020 +0000 +++ b/src/njs_object.h Mon Aug 24 11:28:21 2020 +0000 @@ -63,7 +63,7 @@ njs_int_t njs_object_prototype_proto(njs njs_int_t njs_object_prototype_create_constructor(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, njs_value_t *setval, njs_value_t *retval); -njs_value_t *njs_property_constructor_create(njs_vm_t *vm, njs_lvlhsh_t *hash, +njs_value_t *njs_property_constructor_set(njs_vm_t *vm, njs_lvlhsh_t *hash, njs_value_t *constructor); njs_int_t njs_object_prototype_to_string(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused); From xeioex at nginx.com Wed Aug 26 14:59:04 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 26 Aug 2020 14:59:04 +0000 Subject: [njs] Adapted expect tests after 775c81dcbe61. Message-ID: details: https://hg.nginx.org/njs/rev/781c6b5e81cc branches: changeset: 1502:781c6b5e81cc user: Dmitry Volyntsev date: Wed Aug 26 14:55:28 2020 +0000 description: Adapted expect tests after 775c81dcbe61. diffstat: test/njs_expect_test.exp | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 85bfd770a3da -r 781c6b5e81cc test/njs_expect_test.exp --- a/test/njs_expect_test.exp Mon Aug 24 11:28:21 2020 +0000 +++ b/test/njs_expect_test.exp Wed Aug 26 14:55:28 2020 +0000 @@ -329,7 +329,7 @@ njs_test { {"var o = {toString: function(){}, log: console.log}\r\n" "undefined\r\n>> "} {"o\r\n" - "o\r\n{\r\n toString: \\\[Function],\r\n log: \\\[Function: native]\r\n}"} + "o\r\n{\r\n toString: \\\[Function],\r\n log: \\\[Function: log]\r\n}"} } njs_test { From xeioex at nginx.com Wed Aug 26 14:59:06 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 26 Aug 2020 14:59:06 +0000 Subject: [njs] Generator: simplified runtime errors generation. Message-ID: details: https://hg.nginx.org/njs/rev/a2d12799c9f7 branches: changeset: 1503:a2d12799c9f7 user: Dmitry Volyntsev date: Wed Aug 26 14:55:47 2020 +0000 description: Generator: simplified runtime errors generation. diffstat: src/njs_disassembler.c | 24 ++++++++++++++++-- src/njs_generator.c | 25 +++++------------- src/njs_vmcode.c | 25 +++++++----------- src/njs_vmcode.h | 12 +++++--- src/test/njs_unit_test.c | 62 ++++++++++++++++++++++++------------------------ test/njs_expect_test.exp | 14 ++++++---- 6 files changed, 84 insertions(+), 78 deletions(-) diffs (452 lines): diff -r 781c6b5e81cc -r a2d12799c9f7 src/njs_disassembler.c --- a/src/njs_disassembler.c Wed Aug 26 14:55:28 2020 +0000 +++ b/src/njs_disassembler.c Wed Aug 26 14:55:47 2020 +0000 @@ -167,8 +167,10 @@ njs_disassemble(njs_vm_code_t *code) uint32_t line; njs_str_t *name; njs_uint_t n; + const char *type; njs_code_name_t *code_name; njs_vmcode_jump_t *jump; + njs_vmcode_error_t *error; njs_vmcode_1addr_t *code1; njs_vmcode_2addr_t *code2; njs_vmcode_3addr_t *code3; @@ -451,10 +453,26 @@ njs_disassemble(njs_vm_code_t *code) continue; } - if (operation == NJS_VMCODE_REFERENCE_ERROR) { - njs_printf("%5uD | %05uz REFERENCE ERROR\n", line, p - start); + if (operation == NJS_VMCODE_ERROR) { + error = (njs_vmcode_error_t *) p; + + switch (error->type) { + case NJS_OBJ_TYPE_REF_ERROR: + type = "REFERENCE"; + break; - p += sizeof(njs_vmcode_reference_error_t); + case NJS_OBJ_TYPE_TYPE_ERROR: + type = "TYPE"; + break; + + case NJS_OBJ_TYPE_ERROR: + default: + type = ""; + } + + njs_printf("%5uD | %05uz %s ERROR\n", line, p - start, type); + + p += sizeof(njs_vmcode_error_t); continue; } diff -r 781c6b5e81cc -r a2d12799c9f7 src/njs_generator.c --- a/src/njs_generator.c Wed Aug 26 14:55:28 2020 +0000 +++ b/src/njs_generator.c Wed Aug 26 14:55:47 2020 +0000 @@ -3459,9 +3459,8 @@ static njs_int_t njs_generate_reference_error(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node) { - njs_jump_off_t ret; - const njs_lexer_entry_t *lex_entry; - njs_vmcode_reference_error_t *ref_err; + njs_vmcode_error_t *ref_err; + const njs_lexer_entry_t *lex_entry; if (njs_slow_path(!node->u.reference.not_defined)) { njs_internal_error(vm, "variable is not defined but not_defined " @@ -3469,24 +3468,14 @@ njs_generate_reference_error(njs_vm_t *v return NJS_ERROR; } - njs_generate_code(generator, njs_vmcode_reference_error_t, ref_err, - NJS_VMCODE_REFERENCE_ERROR, 0, NULL); - - ref_err->token_line = node->token_line; - - ref_err->file.length = node->scope->file.length; - - if (ref_err->file.length != 0) { - ret = njs_name_copy(vm, &ref_err->file, &node->scope->file); - if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; - } - } - + njs_generate_code(generator, njs_vmcode_error_t, ref_err, NJS_VMCODE_ERROR, + 0, NULL); + + ref_err->type = NJS_OBJ_TYPE_REF_ERROR; lex_entry = njs_lexer_entry(node->u.reference.unique_id); if (njs_slow_path(lex_entry == NULL)) { return NJS_ERROR; } - return njs_name_copy(vm, &ref_err->name, &lex_entry->name); + return njs_name_copy(vm, &ref_err->u.name, &lex_entry->name); } diff -r 781c6b5e81cc -r a2d12799c9f7 src/njs_vmcode.c --- a/src/njs_vmcode.c Wed Aug 26 14:55:28 2020 +0000 +++ b/src/njs_vmcode.c Wed Aug 26 14:55:47 2020 +0000 @@ -49,7 +49,7 @@ static njs_jump_off_t njs_vmcode_try_end njs_value_t *offset); static njs_jump_off_t njs_vmcode_finally(njs_vm_t *vm, njs_value_t *invld, njs_value_t *retval, u_char *pc); -static void njs_vmcode_reference_error(njs_vm_t *vm, u_char *pc); +static void njs_vmcode_error(njs_vm_t *vm, u_char *pc); /* * These functions are forbidden to inline to minimize JavaScript VM @@ -217,7 +217,7 @@ next: pc += sizeof(njs_vmcode_prop_get_t); if (ret == NJS_OK) { - pc += sizeof(njs_vmcode_reference_error_t); + pc += sizeof(njs_vmcode_error_t); } goto next; @@ -888,8 +888,8 @@ next: break; - case NJS_VMCODE_REFERENCE_ERROR: - njs_vmcode_reference_error(vm, pc); + case NJS_VMCODE_ERROR: + njs_vmcode_error(vm, pc); goto error; default: @@ -1859,21 +1859,16 @@ njs_vmcode_finally(njs_vm_t *vm, njs_val static void -njs_vmcode_reference_error(njs_vm_t *vm, u_char *pc) +njs_vmcode_error(njs_vm_t *vm, u_char *pc) { - njs_str_t *file; - njs_vmcode_reference_error_t *ref_err; + njs_vmcode_error_t *err; - ref_err = (njs_vmcode_reference_error_t *) pc; - - file = &ref_err->file; + err = (njs_vmcode_error_t *) pc; - if (file->length != 0 && !vm->options.quiet) { - njs_reference_error(vm, "\"%V\" is not defined in %V:%uD", - &ref_err->name, file, ref_err->token_line); + if (err->type == NJS_OBJ_TYPE_REF_ERROR) { + njs_reference_error(vm, "\"%V\" is not defined", &err->u.name); } else { - njs_reference_error(vm, "\"%V\" is not defined in %uD", &ref_err->name, - ref_err->token_line); + njs_error_fmt_new(vm, &vm->retval, err->type, "%V", &err->u.message); } } diff -r 781c6b5e81cc -r a2d12799c9f7 src/njs_vmcode.h --- a/src/njs_vmcode.h Wed Aug 26 14:55:28 2020 +0000 +++ b/src/njs_vmcode.h Wed Aug 26 14:55:47 2020 +0000 @@ -61,7 +61,7 @@ typedef uint8_t #define NJS_VMCODE_TRY_END VMCODE0(37) #define NJS_VMCODE_CATCH VMCODE0(38) #define NJS_VMCODE_FINALLY VMCODE0(39) -#define NJS_VMCODE_REFERENCE_ERROR VMCODE0(40) +#define NJS_VMCODE_ERROR VMCODE0(40) #define NJS_VMCODE_NORET 127 @@ -386,10 +386,12 @@ typedef struct { typedef struct { njs_vmcode_t code; - njs_str_t name; - njs_str_t file; - uint32_t token_line; -} njs_vmcode_reference_error_t; + njs_object_type_t type; + union { + njs_str_t name; + njs_str_t message; + } u; +} njs_vmcode_error_t; njs_int_t njs_vmcode_interpreter(njs_vm_t *vm, u_char *pc); diff -r 781c6b5e81cc -r a2d12799c9f7 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Wed Aug 26 14:55:28 2020 +0000 +++ b/src/test/njs_unit_test.c Wed Aug 26 14:55:47 2020 +0000 @@ -253,10 +253,10 @@ static njs_unit_test_t njs_test[] = njs_str("SyntaxError: Unexpected token \"_\" in 1") }, { njs_str("-_1"), - njs_str("ReferenceError: \"_1\" is not defined in 1") }, + njs_str("ReferenceError: \"_1\" is not defined") }, { njs_str("_1"), - njs_str("ReferenceError: \"_1\" is not defined in 1") }, + njs_str("ReferenceError: \"_1\" is not defined") }, /* Octal Numbers. */ @@ -2645,25 +2645,25 @@ static njs_unit_test_t njs_test[] = njs_str("1") }, { njs_str("a"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("\na"), - njs_str("ReferenceError: \"a\" is not defined in 2") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("\n\na"), - njs_str("ReferenceError: \"a\" is not defined in 3") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("a + a"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("a = b + 1"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("a = a + 1"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("a += 1"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("a += 1; var a = 2"), njs_str("undefined") }, @@ -3418,10 +3418,10 @@ static njs_unit_test_t njs_test[] = njs_str("undefined") }, { njs_str("typeof a; a"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("typeof a; a = 1"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, /**/ @@ -3446,7 +3446,7 @@ static njs_unit_test_t njs_test[] = njs_str("1 undefined") }, { njs_str("a = 1"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("var a; a = 1; a"), njs_str("1") }, @@ -3610,7 +3610,7 @@ static njs_unit_test_t njs_test[] = njs_str("{a:1,b:undefined}") }, { njs_str("var a = 1, b = 2; ({a,b,c})"), - njs_str("ReferenceError: \"c\" is not defined in 1") }, + njs_str("ReferenceError: \"c\" is not defined") }, { njs_str("var a = 1, b = 2; njs.dump({a,b,c:3})"), njs_str("{a:1,b:2,c:3}") }, @@ -9123,7 +9123,7 @@ static njs_unit_test_t njs_test[] = njs_str("undefined") }, { njs_str("function f() { var a = f2(); } f();"), - njs_str("ReferenceError: \"f2\" is not defined in 1") }, + njs_str("ReferenceError: \"f2\" is not defined") }, { njs_str("typeof Buffer !== 'undefined' ? Buffer : function Buffer(){}"), njs_str("[object Function]") }, @@ -9132,19 +9132,19 @@ static njs_unit_test_t njs_test[] = njs_str("123") }, { njs_str("1 == 1 ? func() : '123'"), - njs_str("ReferenceError: \"func\" is not defined in 1") }, + njs_str("ReferenceError: \"func\" is not defined") }, { njs_str("function f(){ if (1 == 1) { 1 == 2 ? some_var : '123' } }; f()"), njs_str("undefined") }, { njs_str("function f(){ if (1 == 1) { 1 == 1 ? some_var : '123' } }; f()"), - njs_str("ReferenceError: \"some_var\" is not defined in 1") }, + njs_str("ReferenceError: \"some_var\" is not defined") }, { njs_str("function f(){ if (1 == 1) { 1 == 2 ? some_func() : '123' } }; f()"), njs_str("undefined") }, { njs_str("function f(){ if (1 == 1) { 1 == 1 ? some_func() : '123' } }; f()"), - njs_str("ReferenceError: \"some_func\" is not defined in 1") }, + njs_str("ReferenceError: \"some_func\" is not defined") }, { njs_str("(function(){ function f() {return f}; return f()})()"), njs_str("[object Function]") }, @@ -9269,7 +9269,7 @@ static njs_unit_test_t njs_test[] = njs_str("20") }, { njs_str("var f = function b(a) { a *= 2; return a }; b(10)"), - njs_str("ReferenceError: \"b\" is not defined in 1") }, + njs_str("ReferenceError: \"b\" is not defined") }, { njs_str("var f; f = function(a) { a *= 2; return a }; f(10)"), njs_str("20") }, @@ -9342,7 +9342,7 @@ static njs_unit_test_t njs_test[] = { njs_str("var a = +function f(a) { return a + 1 }(2);" "var b = f(5); a"), - njs_str("ReferenceError: \"f\" is not defined in 1") }, + njs_str("ReferenceError: \"f\" is not defined") }, { njs_str("var o = { f: function(a) { return a * 2 } }; o.f(5)"), njs_str("10") }, @@ -11448,7 +11448,7 @@ static njs_unit_test_t njs_test[] = njs_str("1") }, { njs_str("delete this.Array; Array"), - njs_str("ReferenceError: \"Array\" is not defined in 1") }, + njs_str("ReferenceError: \"Array\" is not defined") }, { njs_str("Array.__proto__ === Function.prototype"), njs_str("true") }, @@ -14641,7 +14641,7 @@ static njs_unit_test_t njs_test[] = njs_str("InternalError: Not implemented") }, { njs_str("delete this.eval; eval"), - njs_str("ReferenceError: \"eval\" is not defined in 1") }, + njs_str("ReferenceError: \"eval\" is not defined") }, { njs_str("var d = Object.getOwnPropertyDescriptor(this, 'eval');" "d.writable && !d.enumerable && d.configurable"), @@ -15882,7 +15882,7 @@ static njs_unit_test_t njs_test[] = njs_str("1") }, { njs_str("delete this.JSON; JSON"), - njs_str("ReferenceError: \"JSON\" is not defined in 1") }, + njs_str("ReferenceError: \"JSON\" is not defined") }, /* Top-level constructors. */ @@ -17585,10 +17585,10 @@ static njs_unit_test_t njs_test[] = njs_str("SyntaxError: Unexpected token \"}\" in 1") }, { njs_str("{{}a = b++}"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("{{}a = b--}"), - njs_str("ReferenceError: \"a\" is not defined in 1") }, + njs_str("ReferenceError: \"a\" is not defined") }, { njs_str("{{}a =}"), njs_str("SyntaxError: Unexpected token \"}\" in 1") }, @@ -17672,7 +17672,7 @@ static njs_unit_test_t njs_test[] = njs_str("SyntaxError: Unexpected end of input in 1") }, { njs_str("`${{a: 1, b}}`"), - njs_str("ReferenceError: \"b\" is not defined in 1") }, + njs_str("ReferenceError: \"b\" is not defined") }, { njs_str("`${{a: 1, b:}}`"), njs_str("SyntaxError: Unexpected token \"}\" in 1") }, @@ -18528,7 +18528,7 @@ static njs_unit_test_t njs_shared_test[ njs_str("false") }, { njs_str("isFin()"), - njs_str("ReferenceError: \"isFin\" is not defined in 1") }, + njs_str("ReferenceError: \"isFin\" is not defined") }, { njs_str("isNaN(function(){})"), njs_str("true") }, @@ -19022,11 +19022,11 @@ static njs_unit_test_t njs_shell_test[] " at main (:3)\n") }, { njs_str("1\n+a" ENTER), - njs_str("ReferenceError: \"a\" is not defined in 2\n" + njs_str("ReferenceError: \"a\" is not defined\n" " at main (:2)\n") }, { njs_str("\n`\n${Object}\n${a}`" ENTER), - njs_str("ReferenceError: \"a\" is not defined in 4\n" + njs_str("ReferenceError: \"a\" is not defined\n" " at main (:4)\n") }, { njs_str("function log(v) {}\nlog({}\n.a\n.a)" ENTER), @@ -19034,15 +19034,15 @@ static njs_unit_test_t njs_shell_test[] " at main (:4)\n") }, { njs_str("\nfor (var i = 0;\n i < a;\n i++) { }\n" ENTER), - njs_str("ReferenceError: \"a\" is not defined in 3\n" + njs_str("ReferenceError: \"a\" is not defined\n" " at main (:3)\n") }, { njs_str("\nfor (var i = 0;\n i < 5;\n a) {\n }" ENTER), - njs_str("ReferenceError: \"a\" is not defined in 4\n" + njs_str("ReferenceError: \"a\" is not defined\n" " at main (:4)\n") }, { njs_str("Math\n.min(1,\na)" ENTER), - njs_str("ReferenceError: \"a\" is not defined in 3\n" + njs_str("ReferenceError: \"a\" is not defined\n" " at Math.min (native)\n" " at main (:3)\n") }, }; diff -r 781c6b5e81cc -r a2d12799c9f7 test/njs_expect_test.exp --- a/test/njs_expect_test.exp Wed Aug 26 14:55:28 2020 +0000 +++ b/test/njs_expect_test.exp Wed Aug 26 14:55:47 2020 +0000 @@ -747,12 +747,12 @@ njs_test { njs_run {"-c" "setTimeout(() => {console.log('A'.repeat(1024))}, 0); ref"} \ "^Thrown: -ReferenceError: \"ref\" is not defined in string:1 +ReferenceError: \"ref\" is not defined at main \\\(string:1\\\)\n$" njs_run {"-c" "setTimeout(() => {ref}, 0); setTimeout(() => {console.log('A'.repeat(1024))}, 0)"} \ "^Thrown: -ReferenceError: \"ref\" is not defined in string:1 +ReferenceError: \"ref\" is not defined at anonymous \\\(string:1\\\) at main \\\(string:1\\\)\n$" @@ -816,6 +816,8 @@ njs_run {"test/script_args.js" "A" "B"} njs_test { {"1+1\r\n" " 1 | 00000 ADD*\r\n*2"} + {"__unknown\r\n" + " 1 | 00000 GLOBAL GET*\r\n*REFERENCE ERROR*"} {"for (var n in [1]) {try {break} finally{}}\r\n" " 1 | 00000 ARRAY*\r\n*TRY BREAK*PROP NEXT*-*\r\n\r\nundefined"} {"(function() {try {return} finally{}})()\r\n" @@ -831,9 +833,9 @@ njs_test { # # For example: # {"import ref from 'ref_exception.js'\r\n" -# "ReferenceError: \"undeclared\" is not defined in ref_exception.js:1"} +# "ReferenceError: \"undeclared\" is not defined"} # {"ref\r\n" -# "ReferenceError: \"ref\" is not defined in shell:1\r\n"} +# "ReferenceError: \"ref\" is not defined\r\n"} njs_test { {"import lib1 from 'lib1.js'; import lib2 from 'lib1.js'\r\n" @@ -851,11 +853,11 @@ njs_test { {"import m from 'export_non_default.js'\r\n" "Non-default export is not supported in export_non_default.js:3\r\n"} {"import ref from 'ref_exception.js'\r\n" - "ReferenceError: \"undeclared\" is not defined in ref_exception.js:1"} + "ReferenceError: \"undeclared\" is not defined"} {"var ref\r\n" "undefined\r\n"} {"import ref from 'ref_exception.js'\r\n" - "ReferenceError: \"undeclared\" is not defined in ref_exception.js:1"} + "ReferenceError: \"undeclared\" is not defined"} {"import m from 'declaration_exception.js'\r\n" "SyntaxError: \"f\" has already been declared in declaration_exception.js:6"} {"import m from 'loading_exception.js'\r\n" From xeioex at nginx.com Wed Aug 26 14:59:08 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 26 Aug 2020 14:59:08 +0000 Subject: [njs] Fixed String.prototype.indexOf() for byte-strings. Message-ID: details: https://hg.nginx.org/njs/rev/657d446001da branches: changeset: 1504:657d446001da user: Dmitry Volyntsev date: Wed Aug 26 14:56:47 2020 +0000 description: Fixed String.prototype.indexOf() for byte-strings. The issue was introduced in e8a941b394a3 (0.4.3). This closes #335 issue on Github. diffstat: src/njs_string.c | 6 +++--- src/test/njs_unit_test.c | 3 +++ 2 files changed, 6 insertions(+), 3 deletions(-) diffs (38 lines): diff -r a2d12799c9f7 -r 657d446001da src/njs_string.c --- a/src/njs_string.c Wed Aug 26 14:55:47 2020 +0000 +++ b/src/njs_string.c Wed Aug 26 14:56:47 2020 +0000 @@ -1999,7 +1999,7 @@ static njs_int_t njs_string_prototype_index_of(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { - int64_t from; + int64_t from, length; njs_int_t ret; njs_value_t *this, *search, *pos, search_lvalue, pos_lvalue; njs_string_prop_t string, s; @@ -2029,10 +2029,10 @@ njs_string_prototype_index_of(njs_vm_t * return ret; } - (void) njs_string_prop(&string, this); + length = njs_string_prop(&string, this); (void) njs_string_prop(&s, search); - from = njs_min(njs_max(from, 0), (int64_t) string.length); + from = njs_min(njs_max(from, 0), length); njs_set_number(&vm->retval, njs_string_index_of(&string, &s, from)); diff -r a2d12799c9f7 -r 657d446001da src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Wed Aug 26 14:55:47 2020 +0000 +++ b/src/test/njs_unit_test.c Wed Aug 26 14:56:47 2020 +0000 @@ -7649,6 +7649,9 @@ static njs_unit_test_t njs_test[] = { njs_str("var r = new String('undefined').indexOf(x); var x; r"), njs_str("0") }, + { njs_str("'a a'.toUTF8().indexOf('a', 1)"), + njs_str("2") }, + { njs_str("'abc'.lastIndexOf('abcdef')"), njs_str("-1") }, From jaderhs5 at gmail.com Wed Aug 26 15:04:10 2020 From: jaderhs5 at gmail.com (Jader H. Silva) Date: Wed, 26 Aug 2020 12:04:10 -0300 Subject: Print milliseconds on error messages timestamps Message-ID: Hi all, Are there any plans to display the timestamp millisecond in error log messages (or in debug logs) on platforms that support it? Thank you*,* Jader H. Silva -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.borisov at nginx.com Wed Aug 26 18:26:57 2020 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Wed, 26 Aug 2020 18:26:57 +0000 Subject: [njs] Improved UTF-8 encoding/decoding. Message-ID: details: https://hg.nginx.org/njs/rev/b98eb205a37b branches: changeset: 1505:b98eb205a37b user: Alexander Borisov date: Wed Aug 26 21:05:46 2020 +0300 description: Improved UTF-8 encoding/decoding. diffstat: src/njs_encoding.c | 135 ++++++---------------------------------------------- src/njs_parser.c | 18 +----- src/njs_utf8.c | 101 ++++++++++++++++++++++++--------------- src/njs_utf8.h | 37 ++++++++++++- 4 files changed, 117 insertions(+), 174 deletions(-) diffs (443 lines): diff -r 657d446001da -r b98eb205a37b src/njs_encoding.c --- a/src/njs_encoding.c Wed Aug 26 14:56:47 2020 +0000 +++ b/src/njs_encoding.c Wed Aug 26 21:05:46 2020 +0300 @@ -18,7 +18,6 @@ typedef struct { njs_bool_t fatal; njs_bool_t ignore_bom; - uint32_t codepoint; njs_unicode_decode_t ctx; } njs_encoding_decode_t; @@ -87,11 +86,10 @@ njs_text_encoder_encode(njs_vm_t *vm, nj njs_index_t unused) { u_char *dst; - int64_t size; - uint32_t cp; + size_t size; njs_int_t ret; njs_value_t *this, *input, value; - const u_char *p, *start, *end; + const u_char *start, *end; njs_string_prop_t prop; njs_typed_array_t *array; njs_unicode_decode_t ctx; @@ -126,30 +124,9 @@ njs_text_encoder_encode(njs_vm_t *vm, nj end = start + prop.size; } - p = start; - - cp = 0; - size = 0; - njs_utf8_decode_init(&ctx); - while (p < end) { - cp = njs_utf8_decode(&ctx, &p, end); - - if (cp > NJS_UNICODE_MAX_CODEPOINT) { - if (cp == NJS_UNICODE_CONTINUE) { - continue; - } - - cp = NJS_UNICODE_REPLACEMENT; - } - - size += njs_utf8_size(cp); - } - - if (cp == NJS_UNICODE_CONTINUE) { - size += njs_utf8_size(NJS_UNICODE_REPLACEMENT); - } + (void) njs_utf8_stream_length(&ctx, start, end - start, 1, 0, &size); njs_set_number(&value, size); @@ -161,23 +138,7 @@ njs_text_encoder_encode(njs_vm_t *vm, nj dst = njs_typed_array_buffer(array)->u.u8; njs_utf8_decode_init(&ctx); - while (start < end) { - cp = njs_utf8_decode(&ctx, &start, end); - - if (cp > NJS_UNICODE_MAX_CODEPOINT) { - if (cp == NJS_UNICODE_CONTINUE) { - continue; - } - - cp = NJS_UNICODE_REPLACEMENT; - } - - dst = njs_utf8_encode(dst, cp); - } - - if (cp == NJS_UNICODE_CONTINUE) { - (void) njs_utf8_encode(dst, NJS_UNICODE_REPLACEMENT); - } + (void) njs_utf8_stream_encode(&ctx, start, end, dst, 1, 0); njs_set_typed_array(&vm->retval, array); @@ -410,7 +371,6 @@ njs_text_decoder_constructor(njs_vm_t *v return ret; } - data->codepoint = 0; njs_utf8_decode_init(&data->ctx); njs_set_data(&ov->value, data, NJS_DATA_TAG_TEXT_DECODER); @@ -573,12 +533,12 @@ njs_text_decoder_decode(njs_vm_t *vm, nj njs_index_t unused) { u_char *dst; - uint32_t length, cp; - uint64_t size; + size_t size; + ssize_t length; njs_int_t ret; njs_bool_t stream; njs_value_t retval, *this, *typed_array, *options; - const u_char *start, *end, *p; + const u_char *start, *end; njs_unicode_decode_t ctx; njs_encoding_decode_t *data; const njs_typed_array_t *array; @@ -632,52 +592,18 @@ njs_text_decoder_decode(njs_vm_t *vm, nj data = njs_object_data(this); ctx = data->ctx; - cp = data->codepoint; - - size = 0; - length = 0; - - p = start; /* Looking for BOM. */ - if (!data->ignore_bom && p + 3 <= end) { - cp = njs_utf8_decode(&ctx, &p, end); - - if (cp == NJS_UNICODE_BOM) { - start = p; - - } else { - p = start; - } + if (!data->ignore_bom) { + start += njs_utf8_bom(start, end); } - while (p < end) { - cp = njs_utf8_decode(&ctx, &p, end); - - if (njs_slow_path(cp > NJS_UNICODE_MAX_CODEPOINT)) { - if (cp == NJS_UNICODE_CONTINUE) { - break; - } - - if (data->fatal) { - goto fatal; - } - - cp = NJS_UNICODE_REPLACEMENT; - } - - size += njs_utf8_size(cp); - length++; - } - - if (cp == NJS_UNICODE_CONTINUE && !stream) { - if (data->fatal) { - goto fatal; - } - - size += njs_utf8_size(NJS_UNICODE_REPLACEMENT); - length++; + length = njs_utf8_stream_length(&ctx, start, end - start, !stream, + data->fatal, &size); + if (length == -1) { + njs_type_error(vm, "The encoded data was not valid"); + return NJS_ERROR; } dst = njs_string_alloc(vm, &vm->retval, size, length); @@ -685,40 +611,13 @@ njs_text_decoder_decode(njs_vm_t *vm, nj return NJS_ERROR; } - while (start < end) { - cp = njs_utf8_decode(&data->ctx, &start, end); + (void) njs_utf8_stream_encode(&data->ctx, start, end, dst, !stream, 0); - if (cp > NJS_UNICODE_MAX_CODEPOINT) { - if (cp == NJS_UNICODE_CONTINUE) { - break; - } - - cp = NJS_UNICODE_REPLACEMENT; - } - - dst = njs_utf8_encode(dst, cp); + if (!stream) { + njs_utf8_decode_init(&data->ctx); } - if (stream) { - data->codepoint = cp; - return NJS_OK; - } - - if (cp == NJS_UNICODE_CONTINUE) { - (void) njs_utf8_encode(dst, NJS_UNICODE_REPLACEMENT); - } - - data->codepoint = 0; - - njs_utf8_decode_init(&data->ctx); - return NJS_OK; - -fatal: - - njs_type_error(vm, "The encoded data was not valid"); - - return NJS_ERROR; } diff -r 657d446001da -r b98eb205a37b src/njs_parser.c --- a/src/njs_parser.c Wed Aug 26 14:56:47 2020 +0000 +++ b/src/njs_parser.c Wed Aug 26 21:05:46 2020 +0300 @@ -7897,15 +7897,16 @@ njs_parser_string_create(njs_vm_t *vm, n njs_value_t *value) { u_char *dst; - ssize_t size, length; - uint32_t cp; + size_t size, length; njs_str_t *src; const u_char *p, *end; njs_unicode_decode_t ctx; src = &token->text; - length = njs_utf8_safe_length(src->start, src->length, &size); + njs_utf8_decode_init(&ctx); + + length = njs_utf8_stream_length(&ctx, src->start, src->length, 1, 0, &size); dst = njs_string_alloc(vm, value, size, length); if (njs_slow_path(dst == NULL)) { @@ -7917,16 +7918,7 @@ njs_parser_string_create(njs_vm_t *vm, n njs_utf8_decode_init(&ctx); - while (p < end) { - cp = njs_utf8_decode(&ctx, &p, end); - - if (cp <= NJS_UNICODE_MAX_CODEPOINT) { - dst = njs_utf8_encode(dst, cp); - - } else { - dst = njs_utf8_encode(dst, NJS_UNICODE_REPLACEMENT); - } - } + (void) njs_utf8_stream_encode(&ctx, p, end, dst, 1, 0); if (length > NJS_STRING_MAP_STRIDE && size != length) { njs_string_offset_map_init(value->long_string.data->start, size); diff -r 657d446001da -r b98eb205a37b src/njs_utf8.c --- a/src/njs_utf8.c Wed Aug 26 14:56:47 2020 +0000 +++ b/src/njs_utf8.c Wed Aug 26 21:05:46 2020 +0300 @@ -213,6 +213,43 @@ failed: return NJS_UNICODE_ERROR; } + +u_char * +njs_utf8_stream_encode(njs_unicode_decode_t *ctx, const u_char *start, + const u_char *end, u_char *dst, njs_bool_t last, njs_bool_t fatal) +{ + uint32_t cp; + + while (start < end) { + cp = njs_utf8_decode(ctx, &start, end); + + if (cp > NJS_UNICODE_MAX_CODEPOINT) { + if (cp == NJS_UNICODE_CONTINUE) { + break; + } + + if (fatal) { + return NULL; + } + + cp = NJS_UNICODE_REPLACEMENT; + } + + dst = njs_utf8_encode(dst, cp); + } + + if (last && ctx->need != 0x00) { + if (fatal) { + return NULL; + } + + dst = njs_utf8_encode(dst, NJS_UNICODE_REPLACEMENT); + } + + return dst; +} + + /* * njs_utf8_casecmp() tests only up to the minimum of given lengths, but * requires lengths of both strings because otherwise njs_utf8_decode() @@ -314,57 +351,43 @@ njs_utf8_upper_case(const u_char **start ssize_t -njs_utf8_length(const u_char *p, size_t len) +njs_utf8_stream_length(njs_unicode_decode_t *ctx, const u_char *p, size_t len, + njs_bool_t last, njs_bool_t fatal, size_t *out_size) { - ssize_t length; - const u_char *end; - njs_unicode_decode_t ctx; - - length = 0; - - end = p + len; - - njs_utf8_decode_init(&ctx); - - while (p < end) { - if (njs_slow_path(njs_utf8_decode(&ctx, &p, end) - > NJS_UNICODE_MAX_CODEPOINT)) - { - return -1; - } - - length++; - } - - return length; -} - - -ssize_t -njs_utf8_safe_length(const u_char *p, size_t len, ssize_t *out_size) -{ - ssize_t size, length; - uint32_t codepoint; - const u_char *end; - njs_unicode_decode_t ctx; + size_t size, length; + uint32_t codepoint; + const u_char *end; size = 0; length = 0; end = p + len; - njs_utf8_decode_init(&ctx); - while (p < end) { - codepoint = njs_utf8_decode(&ctx, &p, end); + codepoint = njs_utf8_decode(ctx, &p, end); - if (codepoint <= NJS_UNICODE_MAX_CODEPOINT) { - size += njs_utf8_size(codepoint); + if (codepoint > NJS_UNICODE_MAX_CODEPOINT) { + if (codepoint == NJS_UNICODE_CONTINUE) { + break; + } - } else { - size += njs_utf8_size(NJS_UNICODE_REPLACEMENT); + if (fatal) { + return -1; + } + + codepoint = NJS_UNICODE_REPLACEMENT; } + size += njs_utf8_size(codepoint); + length++; + } + + if (last && ctx->need != 0x00) { + if (fatal) { + return -1; + } + + size += njs_utf8_size(NJS_UNICODE_REPLACEMENT); length++; } diff -r 657d446001da -r b98eb205a37b src/njs_utf8.h --- a/src/njs_utf8.h Wed Aug 26 14:56:47 2020 +0000 +++ b/src/njs_utf8.h Wed Aug 26 21:05:46 2020 +0300 @@ -8,18 +8,21 @@ #define _NJS_UTF8_H_INCLUDED_ -NJS_EXPORT u_char *njs_utf8_encode(u_char *p, uint32_t u); NJS_EXPORT uint32_t njs_utf8_decode(njs_unicode_decode_t *ctx, const u_char **data, const u_char *end); +NJS_EXPORT u_char *njs_utf8_encode(u_char *p, uint32_t u); +NJS_EXPORT u_char *njs_utf8_stream_encode(njs_unicode_decode_t *ctx, + const u_char *start, const u_char *end, u_char *dst, njs_bool_t last, + njs_bool_t fatal); NJS_EXPORT njs_int_t njs_utf8_casecmp(const u_char *start1, const u_char *start2, size_t len1, size_t len2); NJS_EXPORT uint32_t njs_utf8_lower_case(const u_char **start, const u_char *end); NJS_EXPORT uint32_t njs_utf8_upper_case(const u_char **start, const u_char *end); -NJS_EXPORT ssize_t njs_utf8_length(const u_char *p, size_t len); -NJS_EXPORT ssize_t njs_utf8_safe_length(const u_char *p, size_t len, - ssize_t *out_size); +NJS_EXPORT ssize_t njs_utf8_stream_length(njs_unicode_decode_t *ctx, + const u_char *p, size_t len, njs_bool_t last, njs_bool_t fatal, + size_t *out_size); NJS_EXPORT njs_bool_t njs_utf8_is_valid(const u_char *p, size_t len); @@ -119,6 +122,32 @@ njs_utf8_decode_init(njs_unicode_decode_ } +njs_inline ssize_t +njs_utf8_length(const u_char *p, size_t len) +{ + njs_unicode_decode_t ctx; + + njs_utf8_decode_init(&ctx); + + return njs_utf8_stream_length(&ctx, p, len, 1, 1, NULL); +} + + +njs_inline size_t +njs_utf8_bom(const u_char *start, const u_char *end) +{ + if (start + 3 > end) { + return 0; + } + + if (start[0] == 0xEF && start[1] == 0xBB && start[2] == 0xBF) { + return 3; + } + + return 0; +} + + njs_inline size_t njs_utf8_size(uint32_t cp) { From alexander.borisov at nginx.com Wed Aug 26 18:26:59 2020 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Wed, 26 Aug 2020 18:26:59 +0000 Subject: [njs] Fixed return value for string base64/base64url encoding. Message-ID: details: https://hg.nginx.org/njs/rev/fbee1514cf43 branches: changeset: 1506:fbee1514cf43 user: Alexander Borisov date: Wed Aug 26 21:11:03 2020 +0300 description: Fixed return value for string base64/base64url encoding. diffstat: src/njs_string.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (21 lines): diff -r b98eb205a37b -r fbee1514cf43 src/njs_string.c --- a/src/njs_string.c Wed Aug 26 21:05:46 2020 +0300 +++ b/src/njs_string.c Wed Aug 26 21:11:03 2020 +0300 @@ -317,7 +317,7 @@ njs_string_base64(njs_vm_t *vm, njs_valu dst.length = njs_base64_encoded_length(src->length); - dst.start = njs_string_alloc(vm, &vm->retval, dst.length, dst.length); + dst.start = njs_string_alloc(vm, value, dst.length, dst.length); if (njs_slow_path(dst.start == NULL)) { return NJS_ERROR; } @@ -348,7 +348,7 @@ njs_string_base64url(njs_vm_t *vm, njs_v dst.length = njs_base64_encoded_length(src->length) - padding; - dst.start = njs_string_alloc(vm, &vm->retval, dst.length, dst.length); + dst.start = njs_string_alloc(vm, value, dst.length, dst.length); if (njs_slow_path(dst.start == NULL)) { return NJS_ERROR; } From junho at cloudflare.com Thu Aug 27 00:24:32 2020 From: junho at cloudflare.com (=?iso-8859-1?q?Junho_Choi?=) Date: Wed, 26 Aug 2020 17:24:32 -0700 Subject: [PATCH] HTTP/2 autotuning request body buffer Message-ID: <1739da077a8e27425867.1598487872@C02V518AHTDD> # HG changeset patch # User Junho Choi # Date 1598482301 25200 # Wed Aug 26 15:51:41 2020 -0700 # Node ID 1739da077a8e27425867d4804ffc39f8b9bd59ac # Parent 7015f26aef904e2ec17b4b6f6387fd3b8298f79d HTTP/2 autotuning request body buffer Currently when unbuffered proxy of request buffer is set (proxy_request_buffering=off), H2 receiver window size is set to client_body_buffer_size and http2_body_preread_size (which is greater). This can result in a suboptimal performance when a large request body is uploaded and the connection's BDP (bandwidth x delay) is much larger than current request body buffer. This patch is try to resize request body buffer based on the measured BDP, allowing to have bigger H2 receiver window for faster upload. Our result showed it's effective when the connection has a high BDP much larger than client_body_buffer_size. With a small BDP close to client_body_buffer_size, the improvement may be smaller, but still not worse than current fixed buffer size. When there is a large gap between current BDP and client_body_buffer_size, it can archive 2x or higher upload speed. This is effective when an end user's upload link is very fast. To enable, you need to add --with-http_v2_module (requirement) and --with-http_v2_autotune_upload to configure script when building. ./configure ... \ --with-http_v2_module --with-http_v2_autotune_upload ... To use this feature, add http2_max_client_body_buffer_size directive in nginx.conf (same level of client_body_buffer_size). By default this is 0 (disabled and no optimizing). This value is still capped to a hard coded default of NGX_HTTP_V2_MAX_CLIENT_BODY_BUFFER_SIZE (64MB). For example this value can be set to 8m, to allow request buffer growing up to 8MB but no more, to prevent from overusing memory in a very high BDP. Example nginx.conf: http2_body_preread_size 64k client_body_buffer_size 128k http2_max_client_body_buffer_size 8m This feature requires TCP_INFO. For more information, please see: https://blog.cloudflare.com/delivering-http-2-upload-speed-improvements/ diff -r 7015f26aef90 -r 1739da077a8e auto/modules --- a/auto/modules Wed Jul 29 13:28:04 2020 +0300 +++ b/auto/modules Wed Aug 26 15:51:41 2020 -0700 @@ -420,6 +420,15 @@ ngx_module_libs= ngx_module_link=$HTTP_V2 + if [ $HTTP_V2 = YES -a $HTTP_V2_AUTOTUNE_UPLOAD = YES ]; then + have=NGX_HTTP_V2_AUTOTUNE_UPLOAD . auto/have + + ngx_module_deps="$ngx_module_deps \ + src/http/v2/ngx_autotune_upload.h" + ngx_module_srcs="$ngx_module_srcs \ + src/http/v2/ngx_autotune_upload.c" + fi + . auto/module fi diff -r 7015f26aef90 -r 1739da077a8e auto/options --- a/auto/options Wed Jul 29 13:28:04 2020 +0300 +++ b/auto/options Wed Aug 26 15:51:41 2020 -0700 @@ -59,6 +59,7 @@ HTTP_GZIP=YES HTTP_SSL=NO HTTP_V2=NO +HTTP_V2_AUTOTUNE_UPLOAD=NO HTTP_SSI=YES HTTP_REALIP=NO HTTP_XSLT=NO @@ -224,6 +225,7 @@ --with-http_ssl_module) HTTP_SSL=YES ;; --with-http_v2_module) HTTP_V2=YES ;; + --with-http_v2_autotune_upload) HTTP_V2_AUTOTUNE_UPLOAD=YES;; --with-http_realip_module) HTTP_REALIP=YES ;; --with-http_addition_module) HTTP_ADDITION=YES ;; --with-http_xslt_module) HTTP_XSLT=YES ;; diff -r 7015f26aef90 -r 1739da077a8e src/http/v2/ngx_autotune_upload.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/http/v2/ngx_autotune_upload.c Wed Aug 26 15:51:41 2020 -0700 @@ -0,0 +1,198 @@ +/* + * Copyright (C) 2020 Cloudflare, Inc. + */ + +#include +#include +#include + +static void *ngx_prealloc(ngx_pool_t *pool, void *p, size_t size); +static void *ngx_realloc(void *oldp, size_t size, ngx_log_t *log); + +static ngx_int_t ngx_resize_buf(ngx_pool_t *pool, ngx_buf_t *buf, size_t nsize); + + +static void * +ngx_prealloc(ngx_pool_t *pool, void *p, size_t size) +{ + ngx_pool_large_t *l; + void *newp; + + for (l = pool->large; l; l = l->next) { + if (p == l->alloc) { + ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, pool->log, 0, + "prealloc: %p", l->alloc); + + newp = ngx_realloc(l->alloc, size, pool->log); + if (newp) { + l->alloc = newp; + + return newp; + } else { + return NULL; + } + } + } + + /* not found */ + return NULL; +} + + +static void * +ngx_realloc(void *oldp, size_t size, ngx_log_t *log) +{ + void *newp; + + newp = realloc(oldp, size); + if (newp == NULL) { + ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, + "realloc(%uz) failed", size); + } + + ngx_log_debug2(NGX_LOG_DEBUG_ALLOC, log, 0, "realloc: %p:%uz", newp, size); + + return newp; +} + + +/* resize the buffer to the new size */ +static ngx_int_t +ngx_resize_buf(ngx_pool_t *pool, ngx_buf_t *buf, size_t nsize) +{ + void *nbuf = ngx_prealloc(pool, buf->start, nsize); + + if (!nbuf) { + return NGX_ERROR; + } + + /* if buf->start is moved to a new location */ + if (nbuf != buf->start) { + buf->pos = (u_char *)nbuf + (buf->pos - buf->start); + buf->last = (u_char *)nbuf + (buf->last - buf->start); + } + + /* resize buffer */ + buf->start = nbuf; + buf->end = (u_char *)nbuf + nsize; + + return NGX_OK; +} + + +/* get current TCP RTT (ms) of the connection */ +ngx_int_t +ngx_tcp_rtt_ms(int fd) +{ +#if (NGX_HAVE_TCP_INFO) + struct tcp_info ti; + socklen_t len; + + len = sizeof(struct tcp_info); + if (getsockopt(fd, IPPROTO_TCP, TCP_INFO, &ti, &len) == 0) { + return ti.tcpi_rtt / 1000; + } +#endif + + return NGX_ERROR; +} + + +/* return current timestamp (ms) */ +ngx_msec_int_t +ngx_timestamp_ms() +{ + ngx_time_t *tp = ngx_timeofday(); + + return tp->sec * 1000 + tp->msec; +} + + +/* + * double the buffer size based on the current BDP. + * returns the new window size if resized. + * returns the current window size if not resized. + * if resizing fails, returns 0. + */ +size_t +ngx_autotune_client_body_buffer(ngx_http_request_t *r, + size_t window) +{ + ngx_buf_t *buf; + ngx_http_v2_stream_t *stream; + ngx_msec_int_t ts_now; + ngx_http_v2_loc_conf_t *h2lcf; + size_t max_window; + + h2lcf = ngx_http_get_module_loc_conf(r, ngx_http_v2_module); + max_window = h2lcf->max_client_body_buffer_size; + + /* no autotuning configured */ + if (!max_window) { + return window; + } + + /* if max_window is smaller than the current window, do nothing */ + if (window >= max_window) { + return window; + } + + stream = r->stream; + buf = r->request_body->buf; + + /* if rtt is not available, do nothing */ + if (stream->rtt == NGX_ERROR) { + return window; + } + + ts_now = ngx_timestamp_ms(); + + if (ts_now >= (stream->ts_checkpoint + stream->rtt)) { + size_t cur_win = (buf->end - buf->start); + size_t new_win = ngx_min(cur_win * 2 , max_window); + + /* if already on the max size, do nothing */ + if (cur_win >= max_window) { + return window; + } + + /* min rtt is 1ms to prevent BDP from becoming zero. */ + ngx_uint_t rtt = ngx_max(stream->rtt, 1); + + /* + * elapsed time (ms) from last checkpoint. mininum value is 1 to + * prevent from dividing by zero in BDP calculation + */ + ngx_uint_t elapsed = ngx_max(ts_now - stream->ts_checkpoint, 1); + + /* calculate BDP (bytes) = rtt * bw */ + ngx_uint_t bdp = rtt * stream->bytes_body_read / elapsed; + + ngx_log_debug4(NGX_LOG_DEBUG_HTTP, stream->connection->connection->log, 0, + "http2 autotune sid:%ui rtt:%z bdp:%z win:%z", + stream->node->id, stream->rtt, bdp, window); + + stream->bytes_body_read = 0; + stream->ts_checkpoint = ts_now; + + /* + * check if we need to bump the buffer size + * based on the heuristic condition + */ + if (bdp > (window / 4)) { + if (ngx_resize_buf(r->pool, buf, new_win) != NGX_OK) { + return 0; + } + + ngx_log_debug4(NGX_LOG_DEBUG_HTTP, + stream->connection->connection->log, 0, + "http2 autotune sid:%ui rtt:%z resized:%z->%z", + stream->node->id, stream->rtt, window, + window + (new_win - cur_win)); + + return window + (new_win - cur_win); + } + } + + return window; +} diff -r 7015f26aef90 -r 1739da077a8e src/http/v2/ngx_autotune_upload.h --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/http/v2/ngx_autotune_upload.h Wed Aug 26 15:51:41 2020 -0700 @@ -0,0 +1,25 @@ +/* + * Copyright (C) 2020 Cloudflare, Inc. + */ + +#ifndef _NGX_AUTOTUNE_UPLOAD_H_INCLUDED_ +#define _NGX_AUTOTUNE_UPLOAD_H_INCLUDED_ + +#include + + +/* the maximum size of the receiver window */ +#define NGX_HTTP_V2_MAX_CLIENT_BODY_BUFFER_SIZE (64*1024*1024) + + +/* get current TCP RTT (ms) of the connection */ +ngx_int_t ngx_tcp_rtt_ms(int fd); + +/* return current timestamp (ms) */ +ngx_msec_int_t ngx_timestamp_ms(); + +/* auto resize the buffer */ +size_t ngx_autotune_client_body_buffer(ngx_http_request_t *r, size_t window); + + +#endif diff -r 7015f26aef90 -r 1739da077a8e src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c Wed Jul 29 13:28:04 2020 +0300 +++ b/src/http/v2/ngx_http_v2.c Wed Aug 26 15:51:41 2020 -0700 @@ -11,6 +11,10 @@ #include +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) +#include +#endif + typedef struct { ngx_str_t name; ngx_uint_t offset; @@ -1122,6 +1126,10 @@ pos += size; h2c->state.length -= size; +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) + stream->bytes_body_read += size; +#endif + if (h2c->state.length) { return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_read_data); @@ -3211,6 +3219,12 @@ h2c->priority_limit += h2scf->concurrent_streams; +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) + stream->bytes_body_read = 0; + stream->rtt = ngx_tcp_rtt_ms(r->connection->fd); + stream->ts_checkpoint = ngx_timestamp_ms(); +#endif + return stream; } @@ -4323,6 +4337,15 @@ return NGX_AGAIN; } +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) + window = ngx_autotune_client_body_buffer(r, window); + + /* resizing failed */ + if (!window) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } +#endif + if (ngx_http_v2_send_window_update(h2c, stream->node->id, window - stream->recv_window) == NGX_ERROR) diff -r 7015f26aef90 -r 1739da077a8e src/http/v2/ngx_http_v2.h --- a/src/http/v2/ngx_http_v2.h Wed Jul 29 13:28:04 2020 +0300 +++ b/src/http/v2/ngx_http_v2.h Wed Aug 26 15:51:41 2020 -0700 @@ -210,6 +210,15 @@ ngx_pool_t *pool; +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) + /* how much client request body read */ + ngx_uint_t bytes_body_read; + /* timestamp of next checkpoint */ + ngx_msec_int_t ts_checkpoint; + /* rtt(ms) of the connection */ + ngx_int_t rtt; +#endif + unsigned waiting:1; unsigned blocked:1; unsigned exhausted:1; diff -r 7015f26aef90 -r 1739da077a8e src/http/v2/ngx_http_v2_module.c --- a/src/http/v2/ngx_http_v2_module.c Wed Jul 29 13:28:04 2020 +0300 +++ b/src/http/v2/ngx_http_v2_module.c Wed Aug 26 15:51:41 2020 -0700 @@ -10,6 +10,9 @@ #include #include +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) +#include +#endif static ngx_int_t ngx_http_v2_add_variables(ngx_conf_t *cf); @@ -38,6 +41,10 @@ static char *ngx_http_v2_chunk_size(ngx_conf_t *cf, void *post, void *data); static char *ngx_http_v2_spdy_deprecated(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) +static char *ngx_http_v2_max_client_body_buffer_size(ngx_conf_t *cf, void *post, + void *data); +#endif static ngx_conf_post_t ngx_http_v2_recv_buffer_size_post = @@ -50,6 +57,10 @@ { ngx_http_v2_streams_index_mask }; static ngx_conf_post_t ngx_http_v2_chunk_size_post = { ngx_http_v2_chunk_size }; +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) +static ngx_conf_post_t ngx_http_v2_max_client_body_buffer_size_post = + { ngx_http_v2_max_client_body_buffer_size }; +#endif static ngx_command_t ngx_http_v2_commands[] = { @@ -208,6 +219,15 @@ 0, NULL }, +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) + { ngx_string("http2_max_client_body_buffer_size"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_size_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_v2_loc_conf_t, max_client_body_buffer_size), + &ngx_http_v2_max_client_body_buffer_size_post }, +#endif + ngx_null_command }; @@ -423,6 +443,10 @@ h2lcf->push_preload = NGX_CONF_UNSET; h2lcf->push = NGX_CONF_UNSET; +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) + h2lcf->max_client_body_buffer_size = NGX_CONF_UNSET_SIZE; +#endif + return h2lcf; } @@ -443,6 +467,12 @@ ngx_conf_merge_value(conf->push_preload, prev->push_preload, 0); +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) + /* default is 0: no auto tuning */ + ngx_conf_merge_size_value(conf->max_client_body_buffer_size, + prev->max_client_body_buffer_size, 0); +#endif + return NGX_CONF_OK; } @@ -608,3 +638,19 @@ return NGX_CONF_OK; } + + +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) +static char * +ngx_http_v2_max_client_body_buffer_size(ngx_conf_t *cf, void *post, + void *data) +{ + size_t *sp = data; + + if (*sp > NGX_HTTP_V2_MAX_CLIENT_BODY_BUFFER_SIZE) { + *sp = NGX_HTTP_V2_MAX_CLIENT_BODY_BUFFER_SIZE; + } + + return NGX_CONF_OK; +} +#endif diff -r 7015f26aef90 -r 1739da077a8e src/http/v2/ngx_http_v2_module.h --- a/src/http/v2/ngx_http_v2_module.h Wed Jul 29 13:28:04 2020 +0300 +++ b/src/http/v2/ngx_http_v2_module.h Wed Aug 26 15:51:41 2020 -0700 @@ -41,6 +41,10 @@ ngx_flag_t push; ngx_array_t *pushes; + +#if (NGX_HTTP_V2_AUTOTUNE_UPLOAD) + size_t max_client_body_buffer_size; +#endif } ngx_http_v2_loc_conf_t; From am55444 at protonmail.com Thu Aug 27 06:37:56 2020 From: am55444 at protonmail.com (am55444) Date: Thu, 27 Aug 2020 06:37:56 +0000 Subject: c->error on 412 responses from not_modified_filter Message-ID: I am trying to understand reasoning behind nginx setting ngx_connection_t->error (via filter_finalize check in ngx_http_finalize_request) for 412 responses in ngx_http_not_modified_filter. From the initial introduction of filter_finalize, looks like it was intended to handle large responses for image filter. However I couldn't find c->error being read anywhere after ngx_http_terminate_request (which is called from ngx_http_finalize_request). https://github.com/nginx/nginx/blob/554916301c424f02b1cabc073845b64f8681099b/src/http/ngx_http_request.c#L2477 Can someone please point me to the right direction? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From eran.kornblau at kaltura.com Thu Aug 27 10:25:35 2020 From: eran.kornblau at kaltura.com (Eran Kornblau) Date: Thu, 27 Aug 2020 10:25:35 +0000 Subject: Exposing additional stub status metrics In-Reply-To: References: Message-ID: Ping Thanks Eran From: nginx-devel On Behalf Of Eran Kornblau Sent: Wednesday, August 19, 2020 9:02 PM To: nginx-devel at nginx.org Subject: Exposing additional stub status metrics Hi all, I want to configure Nginx to output the stub status metrics in the format used by Prometheus. My thought was to simply use the builtin 'return' directive for this, however, I noticed that only 4 out of the 7 metrics are currently exposed as variables. I would like to submit a patch to expose the other 3... Does this make sense? Will the patch be accepted (assuming it follows the coding standards...)? Thank you Eran -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Thu Aug 27 14:43:52 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 27 Aug 2020 14:43:52 +0000 Subject: [njs] Parser: fixed njs_queue_remove() macro usage. Message-ID: details: https://hg.nginx.org/njs/rev/da7d4c125a9c branches: changeset: 1507:da7d4c125a9c user: Dmitry Volyntsev date: Thu Aug 27 14:43:19 2020 +0000 description: Parser: fixed njs_queue_remove() macro usage. The issue caused segfault when NJS_DEBUG is defined. The issue was introduced in 86f55a7dc4a4. diffstat: src/njs_parser.c | 4 +++- src/njs_parser.h | 5 +++-- 2 files changed, 6 insertions(+), 3 deletions(-) diffs (40 lines): diff -r fbee1514cf43 -r da7d4c125a9c src/njs_parser.c --- a/src/njs_parser.c Wed Aug 26 21:11:03 2020 +0300 +++ b/src/njs_parser.c Thu Aug 27 14:43:19 2020 +0000 @@ -480,13 +480,15 @@ njs_parser_not_supported(njs_parser_t *p njs_inline njs_int_t njs_parser_reject(njs_parser_t *parser) { + njs_queue_link_t *link; njs_parser_stack_entry_t *entry; while (!njs_queue_is_empty(&parser->stack)) { entry = njs_queue_link_data(njs_queue_first(&parser->stack), njs_parser_stack_entry_t, link); - njs_queue_remove(njs_queue_first(&parser->stack)); + link = njs_queue_first(&parser->stack); + njs_queue_remove(link); if (!entry->optional) { njs_parser_next(parser, entry->state); diff -r fbee1514cf43 -r da7d4c125a9c src/njs_parser.h --- a/src/njs_parser.h Wed Aug 26 21:11:03 2020 +0300 +++ b/src/njs_parser.h Thu Aug 27 14:43:19 2020 +0000 @@ -280,13 +280,14 @@ njs_parser_height(njs_parser_t *parser, njs_inline njs_int_t njs_parser_stack_pop(njs_parser_t *parser) { + njs_queue_link_t *link; njs_parser_stack_entry_t *entry; - entry = njs_queue_link_data(njs_queue_first(&parser->stack), njs_parser_stack_entry_t, link); - njs_queue_remove(njs_queue_first(&parser->stack)); + link = njs_queue_first(&parser->stack); + njs_queue_remove(link); #ifdef NJS_PARSER_DEBUG njs_printf(" stack_pop(%d)\n", From xeioex at nginx.com Thu Aug 27 14:43:54 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 27 Aug 2020 14:43:54 +0000 Subject: [njs] Splitting debug and debug-memory configure options. Message-ID: details: https://hg.nginx.org/njs/rev/6f561e3d947e branches: changeset: 1508:6f561e3d947e user: Dmitry Volyntsev date: Thu Aug 27 14:43:22 2020 +0000 description: Splitting debug and debug-memory configure options. diffstat: auto/cc | 4 +++- auto/help | 2 ++ auto/options | 2 ++ 3 files changed, 7 insertions(+), 1 deletions(-) diffs (46 lines): diff -r da7d4c125a9c -r 6f561e3d947e auto/cc --- a/auto/cc Thu Aug 27 14:43:19 2020 +0000 +++ b/auto/cc Thu Aug 27 14:43:22 2020 +0000 @@ -170,8 +170,10 @@ case $NJS_CC_NAME in esac if [ "$NJS_DEBUG" = "YES" ]; then - NJS_CC_OPT="$NJS_CC_OPT -O0" njs_define=NJS_DEBUG . auto/define +fi + +if [ "$NJS_DEBUG_MEMORY" = "YES" ]; then njs_define=NJS_DEBUG_MEMORY . auto/define fi diff -r da7d4c125a9c -r 6f561e3d947e auto/help --- a/auto/help Thu Aug 27 14:43:19 2020 +0000 +++ b/auto/help Thu Aug 27 14:43:22 2020 +0000 @@ -16,6 +16,8 @@ default: "$NJS_LD_OPT" --debug=YES enables additional runtime checks, \ default: "$NJS_DEBUG" + --debug-memory=YES enables memory alloc debug, \ +default: "$NJS_DEBUG_MEMORY" --address-sanitizer=YES enables build with address sanitizer, \ default: "$NJS_ADDRESS_SANITIZER" END diff -r da7d4c125a9c -r 6f561e3d947e auto/options --- a/auto/options Thu Aug 27 14:43:19 2020 +0000 +++ b/auto/options Thu Aug 27 14:43:22 2020 +0000 @@ -7,6 +7,7 @@ NJS_CC_OPT=${NJS_CC_OPT:--O} NJS_LD_OPT=${NJS_CC_OPT:--O} NJS_DEBUG=NO +NJS_DEBUG_MEMORY=NO NJS_ADDRESS_SANITIZER=NO NJS_CONFIGURE_OPTIONS= @@ -25,6 +26,7 @@ do --ar=*) AR="$value" ;; --debug=*) NJS_DEBUG="$value" ;; + --debug-memory=*) NJS_DEBUG_MEMORY="$value" ;; --address-sanitizer=*) NJS_ADDRESS_SANITIZER="$value" ;; --help) From xeioex at nginx.com Thu Aug 27 14:43:55 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 27 Aug 2020 14:43:55 +0000 Subject: [njs] Introduced njs_assert() macro. Message-ID: details: https://hg.nginx.org/njs/rev/fa50cc5b1128 branches: changeset: 1509:fa50cc5b1128 user: Dmitry Volyntsev date: Thu Aug 27 14:43:23 2020 +0000 description: Introduced njs_assert() macro. diffstat: src/njs_assert.h | 27 +++++++++++++++++++++++++++ src/njs_main.h | 1 + 2 files changed, 28 insertions(+), 0 deletions(-) diffs (42 lines): diff -r 6f561e3d947e -r fa50cc5b1128 src/njs_assert.h --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/njs_assert.h Thu Aug 27 14:43:23 2020 +0000 @@ -0,0 +1,27 @@ + +/* + * Copyright (C) Dmitry Volyntsev + * Copyright (C) NGINX, Inc. + */ + +#ifndef _NJS_ASSERT_H_INCLUDED_ +#define _NJS_ASSERT_H_INCLUDED_ + +#if (NJS_DEBUG) + +#define njs_assert(condition) \ + do { \ + if (!(condition)) { \ + njs_stderror("Assertion \"%s\" failed at %s:%d\n", #condition, \ + __FILE__, __LINE__); \ + abort(); \ + } \ + } while (0) + +#else + +#define njs_assert(condition) (void) (condition) + +#endif + +#endif /* _NJS_ASSERT_H_INCLUDED_ */ diff -r 6f561e3d947e -r fa50cc5b1128 src/njs_main.h --- a/src/njs_main.h Thu Aug 27 14:43:22 2020 +0000 +++ b/src/njs_main.h Thu Aug 27 14:43:23 2020 +0000 @@ -36,6 +36,7 @@ #include #include #include +#include #include #include From xeioex at nginx.com Fri Aug 28 11:52:51 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 28 Aug 2020 11:52:51 +0000 Subject: [njs] Fixed double close in njs_ftw() introduced in 5a80b43b7098. Message-ID: details: https://hg.nginx.org/njs/rev/0a5145a78a36 branches: changeset: 1510:0a5145a78a36 user: Dmitry Volyntsev date: Fri Aug 28 11:47:50 2020 +0000 description: Fixed double close in njs_ftw() introduced in 5a80b43b7098. This correctly fixes Coverity CID 1465877. diffstat: src/njs_fs.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diffs (14 lines): diff -r fa50cc5b1128 -r 0a5145a78a36 src/njs_fs.c --- a/src/njs_fs.c Thu Aug 27 14:43:23 2020 +0000 +++ b/src/njs_fs.c Fri Aug 28 11:47:50 2020 +0000 @@ -1366,8 +1366,9 @@ njs_ftw(char *path, njs_file_tree_walk_c } } - closedir(d); + (void) closedir(d); d = NULL; + dfd = -1; } path[len] = '\0'; From xeioex at nginx.com Fri Aug 28 11:52:52 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 28 Aug 2020 11:52:52 +0000 Subject: [njs] Aligning switch statements in njs_typed_array_prototype_iterator(). Message-ID: details: https://hg.nginx.org/njs/rev/b0092a58d4fa branches: changeset: 1511:b0092a58d4fa user: Dmitry Volyntsev date: Fri Aug 28 11:48:22 2020 +0000 description: Aligning switch statements in njs_typed_array_prototype_iterator(). Merging NJS_ARRAY_FILTER case to default case for all switch statements. This makes code more static-analyzer friendly. diffstat: src/njs_typed_array.c | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-) diffs (18 lines): diff -r 0a5145a78a36 -r b0092a58d4fa src/njs_typed_array.c --- a/src/njs_typed_array.c Fri Aug 28 11:47:50 2020 +0000 +++ b/src/njs_typed_array.c Fri Aug 28 11:48:22 2020 +0000 @@ -884,13 +884,11 @@ njs_typed_array_prototype_iterator(njs_v break; case NJS_ARRAY_FILTER: + default: r = njs_arr_init(vm->mem_pool, &results, NULL, 4, sizeof(njs_value_t)); if (njs_slow_path(r == NULL)) { return NJS_ERROR; } - - default: - break; } for (i = 0; i < length; i++) { From xeioex at nginx.com Fri Aug 28 11:52:54 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 28 Aug 2020 11:52:54 +0000 Subject: [njs] Introduced copy-on-write for ArrayBuffer. Message-ID: details: https://hg.nginx.org/njs/rev/d3e1f95c0c5c branches: changeset: 1512:d3e1f95c0c5c user: Dmitry Volyntsev date: Fri Aug 28 11:51:35 2020 +0000 description: Introduced copy-on-write for ArrayBuffer. diffstat: src/njs.h | 7 + src/njs_array_buffer.c | 24 ++++ src/njs_array_buffer.h | 2 +- src/njs_object.c | 4 +- src/njs_typed_array.c | 210 ++++++++++++++++++++++++++++++++++++----- src/njs_typed_array.h | 61 +----------- src/njs_value.c | 11 +- src/njs_value.h | 4 - src/njs_vm.c | 8 + src/test/njs_benchmark.c | 9 +- src/test/njs_externals_test.c | 18 +++ src/test/njs_unit_test.c | 31 ++++++ 12 files changed, 288 insertions(+), 101 deletions(-) diffs (720 lines): diff -r b0092a58d4fa -r d3e1f95c0c5c src/njs.h --- a/src/njs.h Fri Aug 28 11:48:22 2020 +0000 +++ b/src/njs.h Fri Aug 28 11:51:35 2020 +0000 @@ -317,6 +317,13 @@ NJS_EXPORT njs_int_t njs_vm_value_string njs_value_t *value, uintptr_t *next); /* + * Sets a Uint8Array value. + * start data is not copied and should not be freed. + */ +NJS_EXPORT njs_int_t njs_vm_value_typed_array_uint8_set(njs_vm_t *vm, + njs_value_t *value, const u_char *start, uint32_t size); + +/* * Converts a value to string. */ NJS_EXPORT njs_int_t njs_vm_value_to_string(njs_vm_t *vm, njs_str_t *dst, diff -r b0092a58d4fa -r d3e1f95c0c5c src/njs_array_buffer.c --- a/src/njs_array_buffer.c Fri Aug 28 11:48:22 2020 +0000 +++ b/src/njs_array_buffer.c Fri Aug 28 11:51:35 2020 +0000 @@ -111,6 +111,30 @@ njs_array_buffer_is_view(njs_vm_t *vm, n } +njs_int_t +njs_array_buffer_writable(njs_vm_t *vm, njs_array_buffer_t *buffer) +{ + void *dst; + + if (!buffer->object.shared) { + return NJS_OK; + } + + dst = njs_mp_alloc(vm->mem_pool, buffer->size); + if (njs_slow_path(dst == NULL)) { + njs_memory_error(vm); + return NJS_ERROR; + } + + memcpy(dst, buffer->u.data, buffer->size); + + buffer->object.shared = 0; + buffer->u.data = dst; + + return NJS_OK; +} + + static const njs_object_prop_t njs_array_buffer_constructor_properties[] = { { diff -r b0092a58d4fa -r d3e1f95c0c5c src/njs_array_buffer.h --- a/src/njs_array_buffer.h Fri Aug 28 11:48:22 2020 +0000 +++ b/src/njs_array_buffer.h Fri Aug 28 11:51:35 2020 +0000 @@ -13,7 +13,7 @@ njs_array_buffer_t *njs_array_buffer_alloc(njs_vm_t *vm, uint64_t size); - +njs_int_t njs_array_buffer_writable(njs_vm_t *vm, njs_array_buffer_t *buffer); njs_inline njs_array_buffer_t * njs_array_buffer_slice(njs_vm_t *vm, njs_array_buffer_t *this, int64_t start, diff -r b0092a58d4fa -r d3e1f95c0c5c src/njs_object.c --- a/src/njs_object.c Fri Aug 28 11:48:22 2020 +0000 +++ b/src/njs_object.c Fri Aug 28 11:51:35 2020 +0000 @@ -668,7 +668,7 @@ njs_object_enumerate_typed_array(njs_vm_ case NJS_ENUM_VALUES: for (i = 0; i < length; i++) { - njs_set_number(item++, njs_typed_array_get(array, i)); + njs_set_number(item++, njs_typed_array_prop(array, i)); } break; @@ -681,7 +681,7 @@ njs_object_enumerate_typed_array(njs_vm_ } njs_uint32_to_string(&entry->start[0], i); - njs_set_number(&entry->start[1], njs_typed_array_get(array, i)); + njs_set_number(&entry->start[1], njs_typed_array_prop(array, i)); njs_set_array(item++, entry); } diff -r b0092a58d4fa -r d3e1f95c0c5c src/njs_typed_array.c --- a/src/njs_typed_array.c Fri Aug 28 11:48:22 2020 +0000 +++ b/src/njs_typed_array.c Fri Aug 28 11:51:35 2020 +0000 @@ -19,6 +19,10 @@ typedef enum { } njs_array_iterator_fun_t; +static void njs_typed_array_prop_set(njs_vm_t *vm, njs_typed_array_t *array, + uint32_t index, double v); + + njs_typed_array_t * njs_typed_array_alloc(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_object_type_t type) @@ -93,7 +97,7 @@ njs_typed_array_alloc(njs_vm_t *vm, njs_ size = (uint64_t) njs_typed_array_length(src_tarray) * element_size; } else if (njs_is_object(value)) { - if (njs_is_array(value) && njs_object_hash_is_empty(value)) { + if (njs_is_fast_array(value)) { src_array = njs_array(value); length = src_array->length; @@ -136,8 +140,8 @@ njs_typed_array_alloc(njs_vm_t *vm, njs_ if (type != src_tarray->type) { length = njs_typed_array_length(src_tarray); for (i = 0; i < length; i++) { - njs_typed_array_set(array, i, - njs_typed_array_get(src_tarray, i)); + njs_typed_array_prop_set(vm, array, i, + njs_typed_array_prop(src_tarray, i)); } } else { @@ -152,7 +156,7 @@ njs_typed_array_alloc(njs_vm_t *vm, njs_ } if (ret == NJS_OK) { - njs_typed_array_set(array, i, num); + njs_typed_array_prop_set(vm, array, i, num); } } @@ -172,7 +176,7 @@ njs_typed_array_alloc(njs_vm_t *vm, njs_ } } - njs_typed_array_set(array, i, num); + njs_typed_array_prop_set(vm, array, i, num); } } @@ -193,6 +197,53 @@ memory_error: } +njs_int_t +njs_typed_array_uint8_set(njs_vm_t *vm, njs_value_t *value, const u_char *start, + uint32_t size) +{ + njs_object_t *proto; + njs_typed_array_t *array; + njs_array_buffer_t *buffer; + + array = njs_mp_zalloc(vm->mem_pool, sizeof(njs_typed_array_t) + + sizeof(njs_array_buffer_t)); + if (njs_slow_path(array == NULL)) { + njs_memory_error(vm); + return NJS_ERROR; + } + + buffer = (njs_array_buffer_t *) &array[1]; + + proto = &vm->prototypes[NJS_OBJ_TYPE_ARRAY_BUFFER].object; + + njs_lvlhsh_init(&buffer->object.hash); + njs_lvlhsh_init(&buffer->object.shared_hash); + buffer->object.__proto__ = proto; + buffer->object.slots = NULL; + buffer->object.type = NJS_ARRAY_BUFFER; + buffer->object.shared = 1; + buffer->object.extensible = 1; + buffer->object.error_data = 0; + buffer->object.fast_array = 0; + buffer->u.data = (void *) start; + buffer->size = size; + + array->buffer = buffer; + array->byte_length = size; + array->type = NJS_OBJ_TYPE_UINT8_ARRAY; + njs_lvlhsh_init(&array->object.hash); + njs_lvlhsh_init(&array->object.shared_hash); + array->object.__proto__ = &vm->prototypes[array->type].object; + array->object.type = NJS_TYPED_ARRAY; + array->object.extensible = 1; + array->object.fast_array = 1; + + njs_set_typed_array(value, array); + + return NJS_OK; +} + + static njs_int_t njs_typed_array_constructor(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t magic) @@ -225,6 +276,23 @@ njs_typed_array_get_this(njs_vm_t *vm, n } +njs_array_buffer_t * +njs_typed_array_writable(njs_vm_t *vm, njs_typed_array_t *array) +{ + njs_int_t ret; + njs_array_buffer_t *buffer; + + buffer = array->buffer; + + ret = njs_array_buffer_writable(vm, buffer); + if (njs_slow_path(ret != NJS_OK)) { + return NULL; + } + + return buffer; +} + + static const njs_value_t njs_typed_array_uint8_tag = njs_string("Uint8Array"); static const njs_value_t njs_typed_array_uint8_clamped_tag = njs_long_string("Uint8ClampedArray"); @@ -360,19 +428,82 @@ njs_typed_array_prototype_byte_offset(nj } +static void +njs_typed_array_prop_set(njs_vm_t *vm, njs_typed_array_t *array, uint32_t index, + double v) +{ + int8_t i8; + int16_t i16; + int32_t i32; + njs_array_buffer_t *buffer; + + buffer = array->buffer; + index += array->offset; + + njs_assert(!buffer->object.shared); + + switch (array->type) { + case NJS_OBJ_TYPE_UINT8_CLAMPED_ARRAY: + if (isnan(v) || v < 0) { + v = 0; + } else if (v > 255) { + v = 255; + } + + buffer->u.u8[index] = lrint(v); + + break; + + case NJS_OBJ_TYPE_UINT8_ARRAY: + case NJS_OBJ_TYPE_INT8_ARRAY: + i8 = njs_number_to_int32(v); + buffer->u.u8[index] = i8; + break; + + case NJS_OBJ_TYPE_UINT16_ARRAY: + case NJS_OBJ_TYPE_INT16_ARRAY: + i16 = njs_number_to_int32(v); + buffer->u.u16[index] = i16; + break; + + case NJS_OBJ_TYPE_UINT32_ARRAY: + case NJS_OBJ_TYPE_INT32_ARRAY: + i32 = njs_number_to_int32(v); + buffer->u.u32[index] = i32; + break; + + case NJS_OBJ_TYPE_FLOAT32_ARRAY: + buffer->u.f32[index] = v; + break; + + default: + + /* NJS_OBJ_TYPE_FLOAT64_ARRAY. */ + + buffer->u.f64[index] = v; + } +} + + njs_int_t njs_typed_array_set_value(njs_vm_t *vm, njs_typed_array_t *array, uint32_t index, njs_value_t *setval) { - double num; - njs_int_t ret; + double num; + njs_int_t ret; + njs_array_buffer_t *buffer; ret = njs_value_to_number(vm, setval, &num); if (njs_slow_path(ret != NJS_OK)) { return ret; } - njs_typed_array_set(array, index, num); + buffer = njs_typed_array_writable(vm, array); + if (njs_slow_path(buffer == NULL)) { + return NJS_ERROR; + } + + njs_typed_array_prop_set(vm, array, index, num); njs_set_number(setval, num); @@ -384,12 +515,13 @@ static njs_int_t njs_typed_array_prototype_set(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { - double num; - int64_t i, length, src_length, offset; - njs_int_t ret; - njs_value_t *this, *src, *value, prop; - njs_array_t *array; - njs_typed_array_t *self, *src_tarray; + double num; + int64_t i, length, src_length, offset; + njs_int_t ret; + njs_value_t *this, *src, *value, prop; + njs_array_t *array; + njs_typed_array_t *self, *src_tarray; + njs_array_buffer_t *buffer; this = njs_argument(args, 0); if (njs_slow_path(!njs_is_typed_array(this))) { @@ -411,6 +543,11 @@ njs_typed_array_prototype_set(njs_vm_t * return NJS_ERROR; } + buffer = njs_typed_array_writable(vm, self); + if (njs_slow_path(buffer == NULL)) { + return NJS_ERROR; + } + length = njs_typed_array_length(self); if (njs_is_typed_array(src)) { @@ -427,12 +564,12 @@ njs_typed_array_prototype_set(njs_vm_t * length = njs_min(njs_typed_array_length(src_tarray), length - offset); for (i = 0; i < length; i++) { - njs_typed_array_set(self, offset + i, - njs_typed_array_get(src_tarray, i)); + njs_typed_array_prop_set(vm, self, offset + i, + njs_typed_array_prop(src_tarray, i)); } } else { - if (njs_is_array(src) && njs_object_hash_is_empty(src)) { + if (njs_is_fast_array(src)) { array = njs_array(src); src_length = array->length; @@ -448,7 +585,7 @@ njs_typed_array_prototype_set(njs_vm_t * for (i = 0; i < length; i++) { ret = njs_value_to_number(vm, &array->start[i], &num); if (ret == NJS_OK) { - njs_typed_array_set(self, offset + i, num); + njs_typed_array_prop_set(vm, self, offset + i, num); } } @@ -489,7 +626,7 @@ njs_typed_array_prototype_set(njs_vm_t * } } - njs_typed_array_set(self, offset + i, num); + njs_typed_array_prop_set(vm, self, offset + i, num); } } @@ -552,9 +689,13 @@ njs_typed_array_prototype_fill(njs_vm_t end = (end < 0) ? njs_max(length + end, 0) : njs_min(end, length); + buffer = njs_typed_array_writable(vm, array); + if (njs_slow_path(buffer == NULL)) { + return NJS_ERROR; + } + njs_set_typed_array(&vm->retval, array); - buffer = array->buffer; offset = array->offset; switch (array->type) { @@ -744,8 +885,8 @@ njs_typed_array_prototype_slice(njs_vm_t } else { for (i = 0; i < count; i++) { - njs_typed_array_set(new_array, i, - njs_typed_array_get(array, i + start)); + njs_typed_array_prop_set(vm, new_array, i, + njs_typed_array_prop(array, i + start)); } } @@ -824,7 +965,11 @@ njs_typed_array_prototype_copy_within(nj return NJS_OK; } - buffer = njs_typed_array_buffer(array); + buffer = njs_typed_array_writable(vm, array); + if (njs_slow_path(buffer == NULL)) { + return NJS_ERROR; + } + element_size = njs_typed_array_element_size(array->type); to = (to + array->offset) * element_size; @@ -892,7 +1037,7 @@ njs_typed_array_prototype_iterator(njs_v } for (i = 0; i < length; i++) { - val = njs_typed_array_get(array, i); + val = njs_typed_array_prop(array, i); arguments[0] = *this_arg; njs_set_number(&arguments[1], val); @@ -1290,14 +1435,14 @@ njs_typed_array_prototype_reduce(njs_vm_ accumulator = *njs_argument(args, 2); } else { - njs_set_number(&accumulator, njs_typed_array_get(array, from)); + njs_set_number(&accumulator, njs_typed_array_prop(array, from)); from += increment; } for (i = from; i != to; i += increment) { njs_set_undefined(&arguments[0]); arguments[1] = accumulator; - njs_set_number(&arguments[2], njs_typed_array_get(array, i)); + njs_set_number(&arguments[2], njs_typed_array_prop(array, i)); njs_set_number(&arguments[3], i); njs_set_typed_array(&arguments[4], array); @@ -1335,7 +1480,10 @@ njs_typed_array_prototype_reverse(njs_vm array = njs_typed_array(this); length = njs_typed_array_length(array); - buffer = array->buffer; + buffer = njs_typed_array_writable(vm, array); + if (njs_slow_path(buffer == NULL)) { + return NJS_ERROR; + } switch (array->type) { case NJS_OBJ_TYPE_UINT8_ARRAY: @@ -1676,8 +1824,12 @@ njs_typed_array_prototype_sort(njs_vm_t break; } + buffer = njs_typed_array_writable(vm, array); + if (njs_slow_path(buffer == NULL)) { + return NJS_ERROR; + } + length = njs_typed_array_length(array); - buffer = njs_typed_array_buffer(array); element_size = njs_typed_array_element_size(array->type); base = &buffer->u.u8[array->offset * element_size]; orig = base; @@ -1734,7 +1886,7 @@ njs_typed_array_to_chain(njs_vm_t *vm, n } for (i = 0; i < arr_length; i++) { - njs_number_to_chain(vm, chain, njs_typed_array_get(array, i)); + njs_number_to_chain(vm, chain, njs_typed_array_prop(array, i)); njs_chb_append(chain, separator.start, separator.size); } diff -r b0092a58d4fa -r d3e1f95c0c5c src/njs_typed_array.h --- a/src/njs_typed_array.h Fri Aug 28 11:48:22 2020 +0000 +++ b/src/njs_typed_array.h Fri Aug 28 11:51:35 2020 +0000 @@ -10,6 +10,10 @@ njs_typed_array_t *njs_typed_array_alloc(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_object_type_t type); +njs_int_t njs_typed_array_uint8_set(njs_vm_t *vm, njs_value_t *value, + const u_char *start, uint32_t size); +njs_array_buffer_t *njs_typed_array_writable(njs_vm_t *vm, + njs_typed_array_t *array); njs_int_t njs_typed_array_set_value(njs_vm_t *vm, njs_typed_array_t *array, uint32_t index, njs_value_t *setval); njs_int_t njs_typed_array_to_chain(njs_vm_t *vm, njs_chb_t *chain, @@ -51,7 +55,7 @@ njs_typed_array_length(const njs_typed_a njs_inline double -njs_typed_array_get(const njs_typed_array_t *array, uint32_t index) +njs_typed_array_prop(const njs_typed_array_t *array, uint32_t index) { njs_array_buffer_t *buffer; @@ -91,61 +95,6 @@ njs_typed_array_get(const njs_typed_arra } -njs_inline void -njs_typed_array_set(njs_typed_array_t *array, uint32_t index, double v) -{ - int8_t i8; - int16_t i16; - int32_t i32; - njs_array_buffer_t *buffer; - - index += array->offset; - - buffer = array->buffer; - - switch (array->type) { - case NJS_OBJ_TYPE_UINT8_CLAMPED_ARRAY: - if (isnan(v) || v < 0) { - v = 0; - } else if (v > 255) { - v = 255; - } - - buffer->u.u8[index] = lrint(v); - - break; - - case NJS_OBJ_TYPE_UINT8_ARRAY: - case NJS_OBJ_TYPE_INT8_ARRAY: - i8 = njs_number_to_int32(v); - buffer->u.u8[index] = i8; - break; - - case NJS_OBJ_TYPE_UINT16_ARRAY: - case NJS_OBJ_TYPE_INT16_ARRAY: - i16 = njs_number_to_int32(v); - buffer->u.u16[index] = i16; - break; - - case NJS_OBJ_TYPE_UINT32_ARRAY: - case NJS_OBJ_TYPE_INT32_ARRAY: - i32 = njs_number_to_int32(v); - buffer->u.u32[index] = i32; - break; - - case NJS_OBJ_TYPE_FLOAT32_ARRAY: - buffer->u.f32[index] = v; - break; - - default: - - /* NJS_OBJ_TYPE_FLOAT64_ARRAY. */ - - buffer->u.f64[index] = v; - } -} - - extern const njs_object_type_init_t njs_typed_array_type_init; extern const njs_object_type_init_t njs_typed_array_u8_type_init; extern const njs_object_type_init_t njs_typed_array_u8clamped_type_init; diff -r b0092a58d4fa -r d3e1f95c0c5c src/njs_value.c --- a/src/njs_value.c Fri Aug 28 11:48:22 2020 +0000 +++ b/src/njs_value.c Fri Aug 28 11:51:35 2020 +0000 @@ -812,7 +812,7 @@ njs_typed_array_property_query(njs_vm_t prop = &pq->scratch; if (pq->query == NJS_PROPERTY_QUERY_GET) { - njs_set_number(&prop->value, njs_typed_array_get(array, index)); + njs_set_number(&prop->value, njs_typed_array_prop(array, index)); prop->type = NJS_PROPERTY; } else { @@ -961,7 +961,7 @@ njs_value_property(njs_vm_t *vm, njs_val goto slow_path; } - njs_set_number(retval, njs_typed_array_get(tarray, index)); + njs_set_number(retval, njs_typed_array_prop(tarray, index)); return NJS_OK; } @@ -1084,12 +1084,7 @@ njs_value_property_set(njs_vm_t *vm, njs tarray = njs_typed_array(value); if (njs_fast_path(index < njs_typed_array_length(tarray))) { - ret = njs_value_to_number(vm, setval, &num); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - njs_typed_array_set(tarray, index, num); + return njs_typed_array_set_value(vm, tarray, index, setval); } return NJS_OK; diff -r b0092a58d4fa -r d3e1f95c0c5c src/njs_value.h --- a/src/njs_value.h Fri Aug 28 11:48:22 2020 +0000 +++ b/src/njs_value.h Fri Aug 28 11:51:35 2020 +0000 @@ -721,10 +721,6 @@ typedef struct { ((value)->data.u.object->slots) -#define njs_object_hash_is_empty(value) \ - (njs_lvlhsh_is_empty(njs_object_hash(value))) - - #define njs_array(value) \ ((value)->data.u.array) diff -r b0092a58d4fa -r d3e1f95c0c5c src/njs_vm.c --- a/src/njs_vm.c Fri Aug 28 11:48:22 2020 +0000 +++ b/src/njs_vm.c Fri Aug 28 11:51:35 2020 +0000 @@ -696,6 +696,14 @@ njs_vm_value_string_set(njs_vm_t *vm, nj } +njs_int_t +njs_vm_value_typed_array_uint8_set(njs_vm_t *vm, njs_value_t *value, + const u_char *start, uint32_t size) +{ + return njs_typed_array_uint8_set(vm, value, start, size); +} + + u_char * njs_vm_value_string_alloc(njs_vm_t *vm, njs_value_t *value, uint32_t size) { diff -r b0092a58d4fa -r d3e1f95c0c5c src/test/njs_benchmark.c --- a/src/test/njs_benchmark.c Fri Aug 28 11:48:22 2020 +0000 +++ b/src/test/njs_benchmark.c Fri Aug 28 11:51:35 2020 +0000 @@ -253,7 +253,7 @@ static njs_benchmark_test_t njs_test[] 1 }, { "typed array 10M", - njs_str("var arr = new Uint8Array(10000000);" + njs_str("var arr = new Uint8Array(10**7);" "var count = 0, length = arr.length;" "arr.fill(2);" "for (var i = 0; i < length; i++) { count += arr[i]; }" @@ -261,6 +261,13 @@ static njs_benchmark_test_t njs_test[] njs_str("20000000"), 1 }, + { "typed array 10M set", + njs_str("var arr = new Uint32Array(10**7);" + "var length = arr.length;" + "for (var i = 0; i < length; i++) { arr[i] = i; }"), + njs_str("undefined"), + 1 }, + { "external property ($shared.uri)", njs_str("$shared.uri"), njs_str("shared"), diff -r b0092a58d4fa -r d3e1f95c0c5c src/test/njs_externals_test.c --- a/src/test/njs_externals_test.c Fri Aug 28 11:48:22 2020 +0000 +++ b/src/test/njs_externals_test.c Fri Aug 28 11:51:35 2020 +0000 @@ -194,6 +194,15 @@ njs_unit_test_r_host(njs_vm_t *vm, njs_o static njs_int_t +njs_unit_test_r_u8buffer(njs_vm_t *vm, njs_object_prop_t *prop, + njs_value_t *value, njs_value_t *setval, njs_value_t *retval) +{ + return njs_vm_value_typed_array_uint8_set(vm, retval, + (u_char *) "???????????", 22); +} + + +static njs_int_t njs_unit_test_r_vars(njs_vm_t *vm, njs_object_prop_t *self, njs_value_t *value, njs_value_t *setval, njs_value_t *retval) { @@ -554,6 +563,15 @@ static njs_external_t njs_unit_test_r_e }, { + .flags = NJS_EXTERN_PROPERTY, + .name.string = njs_str("u8buffer"), + .enumerable = 1, + .u.property = { + .handler = njs_unit_test_r_u8buffer, + } + }, + + { .flags = NJS_EXTERN_METHOD, .name.string = njs_str("method"), .writable = 1, diff -r b0092a58d4fa -r d3e1f95c0c5c src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Fri Aug 28 11:48:22 2020 +0000 +++ b/src/test/njs_unit_test.c Fri Aug 28 11:51:35 2020 +0000 @@ -18502,6 +18502,37 @@ static njs_unit_test_t njs_externals_te { njs_str("var r = JSON.parse(JSON.stringify($r));" "[r.uri, r.host, r.props.a, njs.dump(r.vars), njs.dump(r.consts), r.header['02']]"), njs_str("???,???????????,1,{},{},02|???") }, + + { njs_str("var s = (new TextDecoder()).decode($r.u8buffer); [s, s.length]"), + njs_str("???????????,11") }, + + { njs_str("var b = $r.u8buffer; " + "b[4] = '@'.codePointAt(0); b[5] = '#'.codePointAt(0);" + "var s = (new TextDecoder()).decode(b); [s, s.length]"), + njs_str("??@#????????,12") }, + + { njs_str("var b = $r.u8buffer; " + "b.copyWithin(16,0,6);" + "var s = (new TextDecoder()).decode(b); [s, s.length]"), + njs_str("???????????,11") }, + + { njs_str("var b = $r.u8buffer; " + "b.fill('#'.codePointAt(0));" + "var s = (new TextDecoder()).decode(b); [s, s.length]"), + njs_str("######################,22") }, + + { njs_str("var b = $r.u8buffer; " + "b.set(['@'.codePointAt(0), '#'.codePointAt(0)], 4);" + "var s = (new TextDecoder()).decode(b); [s, s.length]"), + njs_str("??@#????????,12") }, + + { njs_str("var b = $r.u8buffer; " + "var u16 = new Uint16Array(b.buffer); u16.reverse();" + "var s = (new TextDecoder()).decode(u16); [s, s.length]"), + njs_str("???????????,11") }, + + { njs_str("$r.u8buffer.sort().slice(0,3)"), + njs_str("129,144,145") }, }; static njs_unit_test_t njs_shared_test[] = From mdounin at mdounin.ru Mon Aug 31 01:49:31 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 31 Aug 2020 04:49:31 +0300 Subject: Print milliseconds on error messages timestamps In-Reply-To: References: Message-ID: <20200831014931.GV12747@mdounin.ru> Hello! On Wed, Aug 26, 2020 at 12:04:10PM -0300, Jader H. Silva wrote: > Are there any plans to display the timestamp millisecond in error log > messages (or in debug logs) on platforms that support it? As of now, there are no plans. In debug logs, milliseconds passed between event loop iterations can be seen in the "timer delta: ..." log lines. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Mon Aug 31 02:06:55 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 31 Aug 2020 05:06:55 +0300 Subject: c->error on 412 responses from not_modified_filter In-Reply-To: References: Message-ID: <20200831020655.GW12747@mdounin.ru> Hello! On Thu, Aug 27, 2020 at 06:37:56AM +0000, am55444 wrote: > I am trying to understand reasoning behind nginx setting ngx_connection_t->error (via filter_finalize check in ngx_http_finalize_request) for 412 responses in ngx_http_not_modified_filter. > > From the initial introduction of filter_finalize, looks like it was intended to handle large responses for image filter. However I couldn't find c->error being read anywhere after ngx_http_terminate_request (which is called from ngx_http_finalize_request). > https://github.com/nginx/nginx/blob/554916301c424f02b1cabc073845b64f8681099b/src/http/ngx_http_request.c#L2477 > > Can someone please point me to the right direction? The c->error flag is set by filter finalization to prevent further output of the original response after the response produced by the filter finalization is sent. The c->error flag is tested by the ngx_http_write_filter() function. (Note that this is a hack / a misuse of the c->error flag. And this hack is known to cause various bad effects. Notably, it prevents connections from being kept alive.) -- Maxim Dounin http://mdounin.ru/ From maksim.yevmenkin at gmail.com Mon Aug 31 18:08:13 2020 From: maksim.yevmenkin at gmail.com (Maksim Yevmenkin) Date: Mon, 31 Aug 2020 11:08:13 -0700 Subject: possible problem with ngx_palloc_small() Message-ID: Hello, a colleague of mine sent me this == There is a problem in ngx_palloc_small() if it is called with arg 'align' set true when the small buffer is almost exhausted such that there are less bytes available in that buffer than the change in alignment consumes In that case, 'm' (the alignment adjusted start of the remainder of the buffer) may move beyond the 'end' marker, meaning that p->d.end - m becomes -ve. Unfortunately, that subtraction is cast to a size_t (unsigned) and so its comparison to '>= size' is very likely true, meaning that the p->d.last is advanced beyond p->d.end and so memory already utilised is returned. iI that happens to trample over bytes used for say the p->large->next...->next chain, then a BUS error is likely It seems that this can be addressed by : @@ -160,7 +160,7 @@ ngx_palloc_small(ngx_pool_t *pool, size_t size, ngx_uint_t align) m = ngx_align_ptr(m, NGX_ALIGNMENT); } - if ((size_t) (p->d.end - m) >= size) { + if (p->d.end >= (size + m)) { p->d.last = m + size; return m; == can someone please share thoughts, comments, etc? thanks, max