From mdounin at mdounin.ru Mon Oct 1 12:32:54 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 1 Oct 2018 15:32:54 +0300 Subject: Bring SPDY back in nginx-1.14.0 In-Reply-To: <636e17f1-bc63-74b7-f5c1-725b202ef1fd@xtremenitro.org> References: <636e17f1-bc63-74b7-f5c1-725b202ef1fd@xtremenitro.org> Message-ID: <20181001123254.GS56558@mdounin.ru> Hello! On Sat, Sep 29, 2018 at 08:29:56AM +0700, Dewangga Alam wrote: > Currently, I am using nginx-1.8.1 in production. And I need SPDY to > works on nginx-1.14.0 co-exists with http/2. Because I still need SPDY > for compatibility, and for future deployment, I will migrate it all to > HTTP/2. > > To make this possible, so I was create patch to make SPDY and HTTP/2 run > together. > > I was read cloudflare patch regarding this issue (spdy & http/2) and > make some adjustment (patch attached). > > But, got some errors when compiling. The error message was : [...] Just to make sure you understand it: As per https://caniuse.com/#search=spdy - SPDY is only supported by about 20% of browsers in use, and most of these browsers already support HTTP/2. Most modern browsers do not have SPDY support. In contrast, HTTP/2 is supported by more than 80% of browsers in use. And most modern browsers which do not support HTTP/2 - do not support SPDY either. Moreover, SPDY code you are trying to bring back is not maintained for a long time now, since it was replaced by HTTP/2 in nginx 1.9.5. And there are various known problems fixed in the HTTP/2 code since then, so likely there are similar problems in the SPDY code. Summing the above, trying to bring back SPDY support is mostly meaningless and might be dangerous. It might be a better idea to focus on migrating to HTTP/2 instead. -- Maxim Dounin http://mdounin.ru/ From dewanggaba at xtremenitro.org Mon Oct 1 16:13:58 2018 From: dewanggaba at xtremenitro.org (Dewangga Alam) Date: Mon, 1 Oct 2018 23:13:58 +0700 Subject: Bring SPDY back in nginx-1.14.0 In-Reply-To: <20181001123254.GS56558@mdounin.ru> References: <636e17f1-bc63-74b7-f5c1-725b202ef1fd@xtremenitro.org> <20181001123254.GS56558@mdounin.ru> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello! On 01/10/18 19.32, Maxim Dounin wrote: > Hello! > > On Sat, Sep 29, 2018 at 08:29:56AM +0700, Dewangga Alam wrote: > >> Currently, I am using nginx-1.8.1 in production. And I need SPDY to >> works on nginx-1.14.0 co-exists with http/2. Because I still need SPD Y >> for compatibility, and for future deployment, I will migrate it all t o >> HTTP/2. >> >> To make this possible, so I was create patch to make SPDY and HTTP/2 run >> together. >> >> I was read cloudflare patch regarding this issue (spdy & http/2) and >> make some adjustment (patch attached). >> >> But, got some errors when compiling. The error message was : > > [...] > > Just to make sure you understand it: > > As per https://caniuse.com/#search=spdy - SPDY is only supported by > about 20% of browsers in use, and most of these browsers already > support HTTP/2. Most modern browsers do not have SPDY support. > > In contrast, HTTP/2 is supported by more than 80% of browsers in > use. And most modern browsers which do not support HTTP/2 - do > not support SPDY either. > > Moreover, SPDY code you are trying to bring back is not maintained > for a long time now, since it was replaced by HTTP/2 in nginx > 1.9.5. And there are various known problems fixed in the HTTP/2 > code since then, so likely there are similar problems in the SPDY > code. > > Summing the above, trying to bring back SPDY support is mostly > meaningless and might be dangerous. It might be a better idea to > focus on migrating to HTTP/2 instead. > Many thanks for very clear explanation, Maxim. We will focus on migrating to HTTP/2. -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEZpxiw/Jg6pEte5xQ5X/SIKAozXAFAluyR8YACgkQ5X/SIKAo zXCFWg/+OOCivq1wL4iNhf3yrpfnMfvzottON1kuu1tR8FCZoO2/rls59eVWOBmE 7I3QjfYQ0CPM7w7n0iB83Em93gCnvWXY/djX2PVoi++o6Vy9ISNlqGCXxBVwneDx p3yNJm+uzvzHisNYTHRc2vnwXCq4fa4eoVxrnm8v7I/VD8F6D2SdCZ5HwM+qZW09 fdbwLwGeBRjbZIVFlXU3lZcpRPUs+42of2dqtUIzGs1qLPotgYVW7HnV2ULnBaqo IMK22kMs7kecnd3WLgTG2uu6n0abcf1UNG0QbYMWwzuyh/W6K61lGXC54pRI1/Xy BtTiWzE1XQP7BabALO5BbhpKz7WcpRFGN+405j7E0s8/ZwhylwwISoN2jaEg+zmZ 1C4Kv/fdFD96s1IPtuXy1IhXYg3OwBI9nNkaDSSNR4ipNhg7uTlMR/6eMFFWlaCW twc/FbR0GG7UoafwiKuZuNRuZBbjPALn2WVv+FSamaDYzrwR7PnTsB2/D8Rzc62D u28Za09k4pCL36GQ5mjL4i+NoO8gxENm+e43GaMPRnWn8h3d1nDtJoODvPol8YBk zR4Wef2PrzzYmPdEDMhS2SSmuIA0aMyTzWeRiMJFXeD8QRs31Q1dXu86sJKkiyS9 QK3G/qpYsxy1ze+aXKF3n/a0hd3KDT8Iaa39Lzgmyb2tWtFaE4Y= =kt7/ -----END PGP SIGNATURE----- From ru at nginx.com Tue Oct 2 10:33:33 2018 From: ru at nginx.com (Ruslan Ermilov) Date: Tue, 02 Oct 2018 10:33:33 +0000 Subject: [nginx] Fixed off-by-one error in shared zone initialization. Message-ID: details: http://hg.nginx.org/nginx/rev/7bf3c323cb6e branches: changeset: 7366:7bf3c323cb6e user: Ruslan Ermilov date: Tue Oct 02 13:32:52 2018 +0300 description: Fixed off-by-one error in shared zone initialization. On systems without atomic ops, not enough space was allocated for mutex's file name during shared zone initialization. diffstat: src/core/ngx_cycle.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diffs (13 lines): diff -r cd4fa2fab8d8 -r 7bf3c323cb6e src/core/ngx_cycle.c --- a/src/core/ngx_cycle.c Tue Sep 25 14:07:59 2018 +0300 +++ b/src/core/ngx_cycle.c Tue Oct 02 13:32:52 2018 +0300 @@ -921,7 +921,8 @@ ngx_init_zone_pool(ngx_cycle_t *cycle, n #else - file = ngx_pnalloc(cycle->pool, cycle->lock_file.len + zn->shm.name.len); + file = ngx_pnalloc(cycle->pool, + cycle->lock_file.len + zn->shm.name.len + 1); if (file == NULL) { return NGX_ERROR; } From mdounin at mdounin.ru Tue Oct 2 15:12:52 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 02 Oct 2018 15:12:52 +0000 Subject: [nginx] SSL: fixed segfault on renegotiation (ticket #1646). Message-ID: details: http://hg.nginx.org/nginx/rev/bf1ac3dc1e68 branches: changeset: 7367:bf1ac3dc1e68 user: Maxim Dounin date: Tue Oct 02 17:46:18 2018 +0300 description: SSL: fixed segfault on renegotiation (ticket #1646). In e3ba4026c02d (1.15.4) nginx own renegotiation checks were disabled if SSL_OP_NO_RENEGOTIATION is available. But since SSL_OP_NO_RENEGOTIATION is only set on a connection, not in an SSL context, SSL_clear_option() removed it as long as a matching virtual server was found. This resulted in a segmentation fault similar to the one fixed in a6902a941279 (1.9.8), affecting nginx built with OpenSSL 1.1.0h or higher. To fix this, SSL_OP_NO_RENEGOTIATION is now explicitly set in ngx_http_ssl_servername() after adjusting options. Additionally, instead of c->ssl->renegotiation we now check c->ssl->handshaked, which seems to be a more correct flag to test, and will prevent the segmentation fault from happening even if SSL_OP_NO_RENEGOTIATION is not working. diffstat: src/http/ngx_http_request.c | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diffs (23 lines): diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -854,7 +854,7 @@ ngx_http_ssl_servername(ngx_ssl_conn_t * c = ngx_ssl_get_connection(ssl_conn); - if (c->ssl->renegotiation) { + if (c->ssl->handshaked) { return SSL_TLSEXT_ERR_NOACK; } @@ -919,6 +919,10 @@ ngx_http_ssl_servername(ngx_ssl_conn_t * #endif SSL_set_options(ssl_conn, SSL_CTX_get_options(sscf->ssl.ctx)); + +#ifdef SSL_OP_NO_RENEGOTIATION + SSL_set_options(ssl_conn, SSL_OP_NO_RENEGOTIATION); +#endif } return SSL_TLSEXT_ERR_OK; From mdounin at mdounin.ru Tue Oct 2 15:16:30 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 02 Oct 2018 15:16:30 +0000 Subject: [nginx] nginx-1.15.5-RELEASE Message-ID: details: http://hg.nginx.org/nginx/rev/f062e43d74fc branches: changeset: 7368:f062e43d74fc user: Maxim Dounin date: Tue Oct 02 18:13:51 2018 +0300 description: nginx-1.15.5-RELEASE diffstat: docs/xml/nginx/changes.xml | 27 +++++++++++++++++++++++++++ 1 files changed, 27 insertions(+), 0 deletions(-) diffs (37 lines): diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml +++ b/docs/xml/nginx/changes.xml @@ -5,6 +5,33 @@ + + + + +??? ????????????? OpenSSL 1.1.0h ? ????? +? ??????? ???????? ??? ????????? segmentation fault; +?????? ????????? ? 1.15.4. + + +a segmentation fault might occur in a worker process +when using OpenSSL 1.1.0h or newer; +the bug had appeared in 1.15.4. + + + + + +?????????????? ????????????? ??????. + + +of minor potential bugs. + + + + + + From mdounin at mdounin.ru Tue Oct 2 15:16:31 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 02 Oct 2018 15:16:31 +0000 Subject: [nginx] release-1.15.5 tag Message-ID: details: http://hg.nginx.org/nginx/rev/64721be763b6 branches: changeset: 7369:64721be763b6 user: Maxim Dounin date: Tue Oct 02 18:13:52 2018 +0300 description: release-1.15.5 tag diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -430,3 +430,4 @@ 4189160cb946bb38d0bc0a452b5eb4cdd8979fb5 b234199c7ed8a156a6bb98f7ff58302c857c954f release-1.15.2 28b3e17ca7eba1e6a0891afde0e4bc5bcc99c861 release-1.15.3 49d49835653857daa418e68d6cbfed4958c78fca release-1.15.4 +f062e43d74fc2578bb100a9e82a953efa1eb9e4e release-1.15.5 From xeioex at nginx.com Tue Oct 2 17:35:12 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 02 Oct 2018 17:35:12 +0000 Subject: [njs] Added nxt_memset(). Message-ID: details: http://hg.nginx.org/njs/rev/891285348d58 branches: changeset: 616:891285348d58 user: Dmitry Volyntsev date: Tue Oct 02 20:33:14 2018 +0300 description: Added nxt_memset(). diffstat: njs/njs_json.c | 2 +- njs/njs_string.c | 4 ++-- nxt/nxt_dtoa.c | 4 ++-- nxt/nxt_mem_cache_pool.c | 2 +- nxt/nxt_string.h | 3 +++ nxt/test/rbtree_unit_test.c | 3 ++- 6 files changed, 11 insertions(+), 7 deletions(-) diffs (94 lines): diff -r 21858b20db0c -r 891285348d58 njs/njs_json.c --- a/njs/njs_json.c Tue Oct 02 20:28:10 2018 +0300 +++ b/njs/njs_json.c Tue Oct 02 20:33:14 2018 +0300 @@ -2316,7 +2316,7 @@ njs_vm_value_dump(njs_vm_t *vm, nxt_str_ goto memory_error; } - memset(stringify->space.start, ' ', indent); + nxt_memset(stringify->space.start, ' ', indent); if (nxt_array_init(&stringify->stack, NULL, 4, sizeof(njs_json_state_t), &njs_array_mem_proto, vm->mem_cache_pool) diff -r 21858b20db0c -r 891285348d58 njs/njs_string.c --- a/njs/njs_string.c Tue Oct 02 20:28:10 2018 +0300 +++ b/njs/njs_string.c Tue Oct 02 20:33:14 2018 +0300 @@ -2477,10 +2477,10 @@ njs_string_prototype_pad(njs_vm_t *vm, n memcpy(start, string.start, string.size); if (nargs == 2) { - memset(p, ' ', padding); + nxt_memset(p, ' ', padding); } else if (pad_string.size == 1) { - memset(p, pad_string.start[0], padding); + nxt_memset(p, pad_string.start[0], padding); } else { while (n != 0) { diff -r 21858b20db0c -r 891285348d58 nxt/nxt_dtoa.c --- a/nxt/nxt_dtoa.c Tue Oct 02 20:28:10 2018 +0300 +++ b/nxt/nxt_dtoa.c Tue Oct 02 20:33:14 2018 +0300 @@ -280,7 +280,7 @@ nxt_prettify(char *start, size_t len, in /* 1234e7 -> 12340000000 */ if (kk - length > 0) { - memset(&start[length], '0', kk - length); + nxt_memset(&start[length], '0', kk - length); } return kk; @@ -305,7 +305,7 @@ nxt_prettify(char *start, size_t len, in start[1] = '.'; if (offset - 2 > 0) { - memset(&start[2], '0', offset - 2); + nxt_memset(&start[2], '0', offset - 2); } return (length + offset); diff -r 21858b20db0c -r 891285348d58 nxt/nxt_mem_cache_pool.c --- a/nxt/nxt_mem_cache_pool.c Tue Oct 02 20:28:10 2018 +0300 +++ b/nxt/nxt_mem_cache_pool.c Tue Oct 02 20:33:14 2018 +0300 @@ -131,7 +131,7 @@ struct nxt_mem_cache_pool_s { #define nxt_mem_cache_free_junk(p, size) \ - memset((p), 0x5A, size) + nxt_memset((p), 0x5A, size) #define nxt_is_power_of_two(value) \ diff -r 21858b20db0c -r 891285348d58 nxt/nxt_string.h --- a/nxt/nxt_string.h Tue Oct 02 20:28:10 2018 +0300 +++ b/nxt/nxt_string.h Tue Oct 02 20:33:14 2018 +0300 @@ -44,6 +44,9 @@ nxt_upper_case(u_char c) #define nxt_cpymem(dst, src, n) (((u_char *) memcpy(dst, src, n)) + (n)) +#define nxt_memset(buf, c, length) (void) (memset(buf, c, length)) + + #define nxt_memzero(buf, length) (void) (memset(buf, 0, length)) diff -r 21858b20db0c -r 891285348d58 nxt/test/rbtree_unit_test.c --- a/nxt/test/rbtree_unit_test.c Tue Oct 02 20:28:10 2018 +0300 +++ b/nxt/test/rbtree_unit_test.c Tue Oct 02 20:33:14 2018 +0300 @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -102,7 +103,7 @@ rbtree_unit_test(nxt_uint_t n) for (i = 0; i < n; i++) { nxt_rbtree_delete(&tree, &items[i].node); - memset(&items[i], 0xA5, sizeof(nxt_rbtree_test_t)); + nxt_memset(&items[i], 0xA5, sizeof(nxt_rbtree_test_t)); } if (!nxt_rbtree_is_empty(&tree)) { From xeioex at nginx.com Tue Oct 2 17:35:12 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 02 Oct 2018 17:35:12 +0000 Subject: [njs] Added nxt_memzero() and nxt_explicit_memzero(). Message-ID: details: http://hg.nginx.org/njs/rev/21858b20db0c branches: changeset: 615:21858b20db0c user: Dmitry Volyntsev date: Tue Oct 02 20:28:10 2018 +0300 description: Added nxt_memzero() and nxt_explicit_memzero(). Thanks to David CARLIER. diffstat: njs/njs.c | 4 ++-- njs/njs_crypto.c | 5 +++-- njs/njs_date.c | 4 ++-- njs/njs_function.c | 2 +- njs/njs_shell.c | 4 ++-- njs/test/njs_benchmark.c | 2 +- njs/test/njs_interactive_test.c | 2 +- njs/test/njs_unit_test.c | 4 ++-- nxt/auto/configure | 1 + nxt/auto/explicit_bzero | 40 ++++++++++++++++++++++++++++++++++++++++ nxt/nxt_array.c | 3 ++- nxt/nxt_lvlhsh.c | 2 +- nxt/nxt_lvlhsh.h | 2 +- nxt/nxt_md5.c | 7 ++++--- nxt/nxt_mem_cache_pool.c | 5 +++-- nxt/nxt_sha1.c | 7 ++++--- nxt/nxt_sha2.c | 7 ++++--- nxt/nxt_string.h | 15 +++++++++++++++ nxt/test/lvlhsh_unit_test.c | 4 ++-- 19 files changed, 91 insertions(+), 29 deletions(-) diffs (420 lines): diff -r c4035cc63370 -r 21858b20db0c njs/njs.c --- a/njs/njs.c Thu Sep 27 17:37:55 2018 +0300 +++ b/njs/njs.c Tue Oct 02 20:28:10 2018 +0300 @@ -28,7 +28,7 @@ njs_zalloc(void *mem, size_t size) p = nxt_malloc(size); if (p != NULL) { - memset(p, 0, size); + nxt_memzero(p, size); } return p; @@ -392,7 +392,7 @@ njs_vm_init(njs_vm_t *vm) return NXT_ERROR; } - memset(frame, 0, NJS_GLOBAL_FRAME_SIZE); + nxt_memzero(frame, NJS_GLOBAL_FRAME_SIZE); vm->top_frame = &frame->native; vm->active_frame = frame; diff -r c4035cc63370 -r 21858b20db0c njs/njs_crypto.c --- a/njs/njs_crypto.c Thu Sep 27 17:37:55 2018 +0300 +++ b/njs/njs_crypto.c Tue Oct 02 20:28:10 2018 +0300 @@ -422,11 +422,12 @@ njs_crypto_create_hmac(njs_vm_t *vm, njs alg->final(digest, &ctx->u); memcpy(key_buf, digest, alg->size); - memset(key_buf + alg->size, 0, sizeof(key_buf) - alg->size); + nxt_explicit_memzero(key_buf + alg->size, sizeof(key_buf) - alg->size); } else { memcpy(key_buf, key.start, key.length); - memset(key_buf + key.length, 0, sizeof(key_buf) - key.length); + nxt_explicit_memzero(key_buf + key.length, + sizeof(key_buf) - key.length); } for (i = 0; i < 64; i++) { diff -r c4035cc63370 -r 21858b20db0c njs/njs_date.c --- a/njs/njs_date.c Thu Sep 27 17:37:55 2018 +0300 +++ b/njs/njs_date.c Tue Oct 02 20:28:10 2018 +0300 @@ -81,7 +81,7 @@ njs_date_constructor(njs_vm_t *vm, njs_v time = njs_date_string_parse(&args[1]); } else { - memset(values, 0, 8 * sizeof(int64_t)); + nxt_memzero(values, 8 * sizeof(int64_t)); /* Month. */ values[2] = 1; @@ -165,7 +165,7 @@ njs_date_utc(njs_vm_t *vm, njs_value_t * time = NAN; if (nargs > 2) { - memset(values, 0, 8 * sizeof(int32_t)); + nxt_memzero(values, 8 * sizeof(int32_t)); n = nxt_min(8, nargs); diff -r c4035cc63370 -r 21858b20db0c njs/njs_function.c --- a/njs/njs_function.c Thu Sep 27 17:37:55 2018 +0300 +++ b/njs/njs_function.c Tue Oct 02 20:28:10 2018 +0300 @@ -262,7 +262,7 @@ njs_function_frame_alloc(njs_vm_t *vm, s vm->stack_size += spare_size; } - memset(frame, 0, sizeof(njs_native_frame_t)); + nxt_memzero(frame, sizeof(njs_native_frame_t)); frame->size = chunk_size; frame->free_size = spare_size - size; diff -r c4035cc63370 -r 21858b20db0c njs/njs_shell.c --- a/njs/njs_shell.c Thu Sep 27 17:37:55 2018 +0300 +++ b/njs/njs_shell.c Tue Oct 02 20:28:10 2018 +0300 @@ -130,7 +130,7 @@ main(int argc, char **argv) njs_opts_t opts; njs_vm_opt_t vm_options; - memset(&opts, 0, sizeof(njs_opts_t)); + nxt_memzero(&opts, sizeof(njs_opts_t)); opts.interactive = 1; ret = njs_get_options(&opts, argc, argv); @@ -143,7 +143,7 @@ main(int argc, char **argv) return EXIT_SUCCESS; } - memset(&vm_options, 0, sizeof(njs_vm_opt_t)); + nxt_memzero(&vm_options, sizeof(njs_vm_opt_t)); vm_options.accumulative = 1; vm_options.backtrace = 1; diff -r c4035cc63370 -r 21858b20db0c njs/test/njs_benchmark.c --- a/njs/test/njs_benchmark.c Thu Sep 27 17:37:55 2018 +0300 +++ b/njs/test/njs_benchmark.c Tue Oct 02 20:28:10 2018 +0300 @@ -26,7 +26,7 @@ njs_unit_test_benchmark(nxt_str_t *scrip njs_vm_opt_t options; struct rusage usage; - memset(&options, 0, sizeof(njs_vm_opt_t)); + nxt_memzero(&options, sizeof(njs_vm_opt_t)); vm = NULL; nvm = NULL; diff -r c4035cc63370 -r 21858b20db0c njs/test/njs_interactive_test.c --- a/njs/test/njs_interactive_test.c Thu Sep 27 17:37:55 2018 +0300 +++ b/njs/test/njs_interactive_test.c Tue Oct 02 20:28:10 2018 +0300 @@ -245,7 +245,7 @@ njs_interactive_test(nxt_bool_t verbose) fflush(stdout); } - memset(&options, 0, sizeof(njs_vm_opt_t)); + nxt_memzero(&options, sizeof(njs_vm_opt_t)); options.accumulative = 1; options.backtrace = 1; diff -r c4035cc63370 -r 21858b20db0c njs/test/njs_unit_test.c --- a/njs/test/njs_unit_test.c Thu Sep 27 17:37:55 2018 +0300 +++ b/njs/test/njs_unit_test.c Tue Oct 02 20:28:10 2018 +0300 @@ -10233,7 +10233,7 @@ njs_unit_test(nxt_bool_t disassemble, nx fflush(stdout); } - memset(&options, 0, sizeof(njs_vm_opt_t)); + nxt_memzero(&options, sizeof(njs_vm_opt_t)); vm = njs_vm_create(&options); if (vm == NULL) { @@ -10380,7 +10380,7 @@ njs_api_test(nxt_bool_t disassemble, nxt rc = NXT_ERROR; vm = NULL; - memset(&options, 0, sizeof(njs_vm_opt_t)); + nxt_memzero(&options, sizeof(njs_vm_opt_t)); for (i = 0; i < nxt_nitems(njs_api_test); i++) { test = &njs_api_test[i]; diff -r c4035cc63370 -r 21858b20db0c nxt/auto/configure --- a/nxt/auto/configure Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/auto/configure Tue Oct 02 20:28:10 2018 +0300 @@ -54,6 +54,7 @@ END . ${NXT_AUTO}time . ${NXT_AUTO}memalign . ${NXT_AUTO}getrandom +. ${NXT_AUTO}explicit_bzero . ${NXT_AUTO}pcre . ${NXT_AUTO}editline . ${NXT_AUTO}expect diff -r c4035cc63370 -r 21858b20db0c nxt/auto/explicit_bzero --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/nxt/auto/explicit_bzero Tue Oct 02 20:28:10 2018 +0300 @@ -0,0 +1,40 @@ + +# Copyright (C) Igor Sysoev +# Copyright (C) NGINX, Inc. + + +# Linux (glibc and musl from 1.1.20), OpenBSD, FreeBSD and NetBSD. + +nxt_feature="explicit_bzero()" +nxt_feature_name=NXT_HAVE_EXPLICIT_BZERO +nxt_feature_run=yes +nxt_feature_incs= +nxt_feature_libs= +nxt_feature_test="#include + #include + + int main(void) { + int r; + + explicit_bzero(&r, sizeof(r)); + return 0; + }" +. ${NXT_AUTO}feature + + +if [ $nxt_found = no ]; then + + # NetBSD has explicit_memset instead. + + nxt_feature="explicit_memset()" + nxt_feature_name=NXT_HAVE_EXPLICIT_MEMSET + nxt_feature_test="#include + + int main(void) { + int r; + + explicit_memset(&r, 0, sizeof(r)); + return 0; + }" + . ${NXT_AUTO}feature +fi diff -r c4035cc63370 -r 21858b20db0c nxt/nxt_array.c --- a/nxt/nxt_array.c Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/nxt_array.c Tue Oct 02 20:28:10 2018 +0300 @@ -9,6 +9,7 @@ #include #include #include +#include #include @@ -140,7 +141,7 @@ nxt_array_zero_add(nxt_array_t *array, c item = nxt_array_add(array, proto, pool); if (nxt_fast_path(item != NULL)) { - memset(item, 0, array->item_size); + nxt_memzero(item, array->item_size); } return item; diff -r c4035cc63370 -r 21858b20db0c nxt/nxt_lvlhsh.c --- a/nxt/nxt_lvlhsh.c Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/nxt_lvlhsh.c Tue Oct 02 20:28:10 2018 +0300 @@ -465,7 +465,7 @@ nxt_lvlhsh_convert_bucket_to_level(nxt_l return NXT_ERROR; } - memset(lvl, 0, size * (sizeof(void *))); + nxt_memzero(lvl, size * (sizeof(void *))); level = lvl; shift = 0; diff -r c4035cc63370 -r 21858b20db0c nxt/nxt_lvlhsh.h --- a/nxt/nxt_lvlhsh.h Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/nxt_lvlhsh.h Tue Oct 02 20:28:10 2018 +0300 @@ -176,7 +176,7 @@ typedef struct { #define nxt_lvlhsh_each_init(lhe, _proto) \ do { \ - memset(lhe, 0, sizeof(nxt_lvlhsh_each_t)); \ + nxt_memzero(lhe, sizeof(nxt_lvlhsh_each_t)); \ (lhe)->proto = _proto; \ } while (0) diff -r c4035cc63370 -r 21858b20db0c nxt/nxt_md5.c --- a/nxt/nxt_md5.c Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/nxt_md5.c Tue Oct 02 20:28:10 2018 +0300 @@ -10,6 +10,7 @@ #include #include #include +#include #include @@ -72,13 +73,13 @@ nxt_md5_final(u_char result[16], nxt_md5 free = 64 - used; if (free < 8) { - memset(&ctx->buffer[used], 0, free); + nxt_memzero(&ctx->buffer[used], free); (void) nxt_md5_body(ctx, ctx->buffer, 64); used = 0; free = 64; } - memset(&ctx->buffer[used], 0, free - 8); + nxt_memzero(&ctx->buffer[used], free - 8); ctx->bytes <<= 3; ctx->buffer[56] = (u_char) ctx->bytes; @@ -109,7 +110,7 @@ nxt_md5_final(u_char result[16], nxt_md5 result[14] = (u_char) (ctx->d >> 16); result[15] = (u_char) (ctx->d >> 24); - memset(ctx, 0, sizeof(*ctx)); + nxt_memzero(ctx, sizeof(*ctx)); } diff -r c4035cc63370 -r 21858b20db0c nxt/nxt_mem_cache_pool.c --- a/nxt/nxt_mem_cache_pool.c Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/nxt_mem_cache_pool.c Tue Oct 02 20:28:10 2018 +0300 @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -322,7 +323,7 @@ nxt_mem_cache_zalloc(nxt_mem_cache_pool_ p = nxt_mem_cache_alloc(pool, size); if (nxt_fast_path(p != NULL)) { - memset(p, 0, size); + nxt_memzero(p, size); } return p; @@ -368,7 +369,7 @@ nxt_mem_cache_zalign(nxt_mem_cache_pool_ p = nxt_mem_cache_align(pool, alignment, size); if (nxt_fast_path(p != NULL)) { - memset(p, 0, size); + nxt_memzero(p, size); } return p; diff -r c4035cc63370 -r 21858b20db0c nxt/nxt_sha1.c --- a/nxt/nxt_sha1.c Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/nxt_sha1.c Tue Oct 02 20:28:10 2018 +0300 @@ -11,6 +11,7 @@ #include #include #include +#include #include @@ -74,13 +75,13 @@ nxt_sha1_final(u_char result[20], nxt_sh free = 64 - used; if (free < 8) { - memset(&ctx->buffer[used], 0, free); + nxt_memzero(&ctx->buffer[used], free); (void) nxt_sha1_body(ctx, ctx->buffer, 64); used = 0; free = 64; } - memset(&ctx->buffer[used], 0, free - 8); + nxt_memzero(&ctx->buffer[used], free - 8); ctx->bytes <<= 3; ctx->buffer[56] = (u_char) (ctx->bytes >> 56); @@ -115,7 +116,7 @@ nxt_sha1_final(u_char result[20], nxt_sh result[18] = (u_char) (ctx->e >> 8); result[19] = (u_char) ctx->e; - memset(ctx, 0, sizeof(*ctx)); + nxt_memzero(ctx, sizeof(*ctx)); } diff -r c4035cc63370 -r 21858b20db0c nxt/nxt_sha2.c --- a/nxt/nxt_sha2.c Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/nxt_sha2.c Tue Oct 02 20:28:10 2018 +0300 @@ -11,6 +11,7 @@ #include #include #include +#include #include @@ -77,13 +78,13 @@ nxt_sha2_final(u_char result[32], nxt_sh free = 64 - used; if (free < 8) { - memset(&ctx->buffer[used], 0, free); + nxt_memzero(&ctx->buffer[used], free); (void) nxt_sha2_body(ctx, ctx->buffer, 64); used = 0; free = 64; } - memset(&ctx->buffer[used], 0, free - 8); + nxt_memzero(&ctx->buffer[used], free - 8); ctx->bytes <<= 3; ctx->buffer[56] = (u_char) (ctx->bytes >> 56); @@ -130,7 +131,7 @@ nxt_sha2_final(u_char result[32], nxt_sh result[30] = (u_char) (ctx->h >> 8); result[31] = (u_char) ctx->h; - memset(ctx, 0, sizeof(*ctx)); + nxt_memzero(ctx, sizeof(*ctx)); } diff -r c4035cc63370 -r 21858b20db0c nxt/nxt_string.h --- a/nxt/nxt_string.h Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/nxt_string.h Tue Oct 02 20:28:10 2018 +0300 @@ -44,6 +44,21 @@ nxt_upper_case(u_char c) #define nxt_cpymem(dst, src, n) (((u_char *) memcpy(dst, src, n)) + (n)) +#define nxt_memzero(buf, length) (void) (memset(buf, 0, length)) + + +#if (NXT_HAVE_EXPLICIT_BZERO) +#define nxt_explicit_memzero(buf, length) \ + (void) (explicit_bzero(buf, length)) +#elif (NXT_HAVE_EXPLICIT_MEMSET) +#define nxt_explicit_memzero(buf, length) \ + (void) (explicit_memset(buf, 0, length)) +#else +#define nxt_explicit_memzero(buf, length) \ + nxt_memzero(buf, length) +#endif + + #define nxt_strstr_eq(s1, s2) \ (((s1)->length == (s2)->length) \ && (memcmp((s1)->start, (s2)->start, (s1)->length) == 0)) diff -r c4035cc63370 -r 21858b20db0c nxt/test/lvlhsh_unit_test.c --- a/nxt/test/lvlhsh_unit_test.c Thu Sep 27 17:37:55 2018 +0300 +++ b/nxt/test/lvlhsh_unit_test.c Tue Oct 02 20:28:10 2018 +0300 @@ -146,7 +146,7 @@ lvlhsh_zalloc(void *mem, size_t size) p = nxt_malloc(size); if (p != NULL) { - memset(p, 0, size); + nxt_memzero(p, size); } return p; @@ -216,7 +216,7 @@ lvlhsh_unit_test(nxt_uint_t n) printf("lvlhsh unit test started: %ld items\n", (long) n); - memset(&lh, 0, sizeof(nxt_lvlhsh_t)); + nxt_memzero(&lh, sizeof(nxt_lvlhsh_t)); key = 0; for (i = 0; i < n; i++) { From xeioex at nginx.com Tue Oct 2 17:35:12 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 02 Oct 2018 17:35:12 +0000 Subject: [njs] Improved time zone tests for Date object. Message-ID: details: http://hg.nginx.org/njs/rev/3528e41d10d4 branches: changeset: 617:3528e41d10d4 user: Dmitry Volyntsev date: Tue Oct 02 20:34:30 2018 +0300 description: Improved time zone tests for Date object. On some platforms, for example Alpine Linux, non-UTC timezones are unavailable. The fix is to use UTC timezone by default, and use a separate test suite for the original timezone if it is available. diffstat: njs/test/njs_unit_test.c | 184 +++++++++++++++++++++++++++++++++++++--------- 1 files changed, 147 insertions(+), 37 deletions(-) diffs (348 lines): diff -r 891285348d58 -r 3528e41d10d4 njs/test/njs_unit_test.c --- a/njs/test/njs_unit_test.c Tue Oct 02 20:33:14 2018 +0300 +++ b/njs/test/njs_unit_test.c Tue Oct 02 20:34:30 2018 +0300 @@ -7327,34 +7327,34 @@ static njs_unit_test_t njs_test[] = nxt_string("Invalid Date NaN") }, { nxt_string("var d = new Date(1); d = d + ''; d.slice(0, 33)"), - nxt_string("Thu Jan 01 1970 12:45:00 GMT+1245") }, + nxt_string("Thu Jan 01 1970 00:00:00 GMT+0000") }, { nxt_string("var d = new Date(1308895200000); d.getTime()"), nxt_string("1308895200000") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.getTime()"), - nxt_string("1308895200000") }, + nxt_string("1308941100000") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.valueOf()"), - nxt_string("1308895200000") }, + nxt_string("1308941100000") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45);" "d.toString().slice(0, 33)"), - nxt_string("Fri Jun 24 2011 18:45:00 GMT+1245") }, + nxt_string("Fri Jun 24 2011 18:45:00 GMT+0000") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.toDateString()"), nxt_string("Fri Jun 24 2011") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45);" "d.toTimeString().slice(0, 17)"), - nxt_string("18:45:00 GMT+1245") }, + nxt_string("18:45:00 GMT+0000") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.toUTCString()"), - nxt_string("Fri Jun 24 2011 06:00:00 GMT") }, + nxt_string("Fri Jun 24 2011 18:45:00 GMT") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45, 12, 625);" "d.toISOString()"), - nxt_string("2011-06-24T06:00:12.625Z") }, + nxt_string("2011-06-24T18:45:12.625Z") }, #if 0 /* These tests fail on Solaris: gmtime_r() returns off by one day. */ @@ -7523,13 +7523,13 @@ static njs_unit_test_t njs_test[] = nxt_string("18") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.getUTCHours()"), - nxt_string("6") }, + nxt_string("18") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.getMinutes()"), nxt_string("45") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.getUTCMinutes()"), - nxt_string("0") }, + nxt_string("45") }, { nxt_string("var d = new Date(2011, 5, 24, 18, 45, 12);" "d.getSeconds()"), @@ -7549,7 +7549,7 @@ static njs_unit_test_t njs_test[] = { nxt_string("var d = new Date(2011, 5, 24, 18, 45, 12, 625);" "d.getTimezoneOffset()"), - nxt_string("-765") }, + nxt_string("0") }, { nxt_string("var d = new Date(); d.setTime(1308895200000); d.getTime()"), nxt_string("1308895200000") }, @@ -7568,15 +7568,15 @@ static njs_unit_test_t njs_test[] = { nxt_string("var d = new Date(1308895323625); d.setMinutes(3, 2, 5003);" "d.getTime()"), - nxt_string("1308892687003") }, + nxt_string("1308895387003") }, { nxt_string("var d = new Date(1308895323625); d.setMinutes(3, 2);" "d.getTime()"), - nxt_string("1308892682625") }, + nxt_string("1308895382625") }, { nxt_string("var d = new Date(1308895323625); d.setMinutes(3);" "d.getTime()"), - nxt_string("1308892683625") }, + nxt_string("1308895383625") }, { nxt_string("var d = new Date(1308895323625); d.setUTCMinutes(3, 2, 5003);" "d.getTime()"), @@ -7592,19 +7592,19 @@ static njs_unit_test_t njs_test[] = { nxt_string("var d = new Date(1308895323625); d.setHours(20, 3, 2, 5003);" "d.getTime()"), - nxt_string("1308899887003") }, + nxt_string("1308945787003") }, { nxt_string("var d = new Date(1308895323625); d.setHours(20, 3, 2);" "d.getTime()"), - nxt_string("1308899882625") }, + nxt_string("1308945782625") }, { nxt_string("var d = new Date(1308895323625); d.setHours(20, 3);" "d.getTime()"), - nxt_string("1308899883625") }, + nxt_string("1308945783625") }, { nxt_string("var d = new Date(1308895323625); d.setHours(20);" "d.getTime()"), - nxt_string("1308902523625") }, + nxt_string("1308945723625") }, { nxt_string("var d = new Date(1308895323625);" "d.setUTCHours(20, 3, 2, 5003); d.getTime()"), @@ -7632,7 +7632,7 @@ static njs_unit_test_t njs_test[] = { nxt_string("var d = new Date(1308895323625); d.setMonth(2, 10);" "d.getTime()"), - nxt_string("1299733323625") }, + nxt_string("1299736923625") }, { nxt_string("var d = new Date(1308895323625); d.setUTCMonth(2, 10);" "d.getTime()"), @@ -7640,7 +7640,7 @@ static njs_unit_test_t njs_test[] = { nxt_string("var d = new Date(1308895323625); d.setMonth(2);" "d.getTime()"), - nxt_string("1300942923625") }, + nxt_string("1300946523625") }, { nxt_string("var d = new Date(1308895323625); d.setUTCMonth(2);" "d.getTime()"), @@ -7648,7 +7648,7 @@ static njs_unit_test_t njs_test[] = { nxt_string("var d = new Date(1308895323625); d.setFullYear(2010, 2, 10);" "d.getTime()"), - nxt_string("1268197323625") }, + nxt_string("1268200923625") }, { nxt_string("var d = new Date(1308895323625);" "d.setUTCFullYear(2010, 2, 10); d.getTime()"), @@ -7656,7 +7656,7 @@ static njs_unit_test_t njs_test[] = { nxt_string("var d = new Date(1308895323625); d.setFullYear(2010, 2);" "d.getTime()"), - nxt_string("1269406923625") }, + nxt_string("1269410523625") }, { nxt_string("var d = new Date(1308895323625); d.setUTCFullYear(2010, 2);" "d.getTime()"), @@ -7672,7 +7672,7 @@ static njs_unit_test_t njs_test[] = { nxt_string("var d = new Date(2011, 5, 24, 18, 45, 12, 625);" "d.toJSON(1)"), - nxt_string("2011-06-24T06:00:12.625Z") }, + nxt_string("2011-06-24T18:45:12.625Z") }, { nxt_string("var o = { toISOString: function() { return 'OK' } };" "Date.prototype.toJSON.call(o, 1)"), @@ -9819,6 +9819,92 @@ static njs_unit_test_t njs_test[] = }; +static njs_unit_test_t njs_tz_test[] = +{ + { nxt_string("var d = new Date(1); d = d + ''; d.slice(0, 33)"), + nxt_string("Thu Jan 01 1970 12:45:00 GMT+1245") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.getTime()"), + nxt_string("1308895200000") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.valueOf()"), + nxt_string("1308895200000") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45);" + "d.toString().slice(0, 33)"), + nxt_string("Fri Jun 24 2011 18:45:00 GMT+1245") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45);" + "d.toTimeString().slice(0, 17)"), + nxt_string("18:45:00 GMT+1245") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.toUTCString()"), + nxt_string("Fri Jun 24 2011 06:00:00 GMT") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45, 12, 625);" + "d.toISOString()"), + nxt_string("2011-06-24T06:00:12.625Z") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.getUTCHours()"), + nxt_string("6") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45); d.getUTCMinutes()"), + nxt_string("0") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45, 12, 625);" + "d.getTimezoneOffset()"), + nxt_string("-765") }, + + { nxt_string("var d = new Date(1308895323625); d.setMinutes(3, 2, 5003);" + "d.getTime()"), + nxt_string("1308892687003") }, + + { nxt_string("var d = new Date(1308895323625); d.setMinutes(3, 2);" + "d.getTime()"), + nxt_string("1308892682625") }, + + { nxt_string("var d = new Date(1308895323625); d.setMinutes(3);" + "d.getTime()"), + nxt_string("1308892683625") }, + + { nxt_string("var d = new Date(1308895323625); d.setHours(20, 3, 2, 5003);" + "d.getTime()"), + nxt_string("1308899887003") }, + + { nxt_string("var d = new Date(1308895323625); d.setHours(20, 3, 2);" + "d.getTime()"), + nxt_string("1308899882625") }, + + { nxt_string("var d = new Date(1308895323625); d.setHours(20, 3);" + "d.getTime()"), + nxt_string("1308899883625") }, + + { nxt_string("var d = new Date(1308895323625); d.setHours(20);" + "d.getTime()"), + nxt_string("1308902523625") }, + + { nxt_string("var d = new Date(1308895323625); d.setMonth(2, 10);" + "d.getTime()"), + nxt_string("1299733323625") }, + + { nxt_string("var d = new Date(1308895323625); d.setMonth(2);" + "d.getTime()"), + nxt_string("1300942923625") }, + + { nxt_string("var d = new Date(1308895323625); d.setFullYear(2010, 2, 10);" + "d.getTime()"), + nxt_string("1268197323625") }, + + { nxt_string("var d = new Date(1308895323625); d.setFullYear(2010, 2);" + "d.getTime()"), + nxt_string("1269406923625") }, + + { nxt_string("var d = new Date(2011, 5, 24, 18, 45, 12, 625);" + "d.toJSON(1)"), + nxt_string("2011-06-24T06:00:12.625Z") }, +}; + + typedef struct { nxt_str_t uri; uint32_t a; @@ -10202,7 +10288,8 @@ njs_externals_init(njs_vm_t *vm) static nxt_int_t -njs_unit_test(nxt_bool_t disassemble, nxt_bool_t verbose) +njs_unit_test(njs_unit_test_t tests[], size_t num, nxt_bool_t disassemble, + nxt_bool_t verbose) { u_char *start; njs_vm_t *vm, *nvm; @@ -10212,20 +10299,12 @@ njs_unit_test(nxt_bool_t disassemble, nx nxt_bool_t success; njs_vm_opt_t options; - /* - * Chatham Islands NZ-CHAT time zone. - * Standard time: UTC+12:45, Daylight Saving time: UTC+13:45. - */ - (void) putenv((char *) "TZ=Pacific/Chatham"); - tzset(); - - vm = NULL; nvm = NULL; rc = NXT_ERROR; - for (i = 0; i < nxt_nitems(njs_test); i++) { + for (i = 0; i < num; i++) { if (verbose) { printf("\"%.*s\"\n", @@ -10310,10 +10389,6 @@ done: njs_vm_destroy(vm); } - if (rc == NXT_OK) { - printf("njs unit tests passed\n"); - } - return rc; } @@ -10414,6 +10489,10 @@ done: int nxt_cdecl main(int argc, char **argv) { + size_t size; + u_char buf[16]; + time_t clock; + struct tm tm; nxt_int_t ret; nxt_bool_t disassemble, verbose; @@ -10436,10 +10515,41 @@ main(int argc, char **argv) } } - ret = njs_unit_test(disassemble, verbose); + (void) putenv((char *) "TZ=UTC"); + tzset(); + + ret = njs_unit_test(njs_test, nxt_nitems(njs_test), disassemble, verbose); if (ret != NXT_OK) { return ret; } + printf("njs unit tests passed\n"); + + /* + * Chatham Islands NZ-CHAT time zone. + * Standard time: UTC+12:45, Daylight Saving time: UTC+13:45. + */ + (void) putenv((char *) "TZ=Pacific/Chatham"); + tzset(); + + clock = 0; + localtime_r(&clock, &tm); + + size = strftime((char *) buf, sizeof(buf), "%z", &tm); + + if (memcmp(buf, "+1245", size) == 0) { + ret = njs_unit_test(njs_tz_test, nxt_nitems(njs_tz_test), disassemble, + verbose); + if (ret != NXT_OK) { + return ret; + } + + printf("njs timezone tests passed\n"); + + } else { + printf("njs timezone tests skipped, timezone is unavailable\n"); + } + + return njs_api_test(disassemble, verbose); } From xeioex at nginx.com Wed Oct 3 12:56:58 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 03 Oct 2018 12:56:58 +0000 Subject: [njs] Removed useless casting to void for explicit_bzero(). Message-ID: details: http://hg.nginx.org/njs/rev/26ebe2b71708 branches: changeset: 618:26ebe2b71708 user: Dmitry Volyntsev date: Wed Oct 03 15:36:36 2018 +0300 description: Removed useless casting to void for explicit_bzero(). diffstat: nxt/nxt_string.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 3528e41d10d4 -r 26ebe2b71708 nxt/nxt_string.h --- a/nxt/nxt_string.h Tue Oct 02 20:34:30 2018 +0300 +++ b/nxt/nxt_string.h Wed Oct 03 15:36:36 2018 +0300 @@ -52,7 +52,7 @@ nxt_upper_case(u_char c) #if (NXT_HAVE_EXPLICIT_BZERO) #define nxt_explicit_memzero(buf, length) \ - (void) (explicit_bzero(buf, length)) + explicit_bzero(buf, length) #elif (NXT_HAVE_EXPLICIT_MEMSET) #define nxt_explicit_memzero(buf, length) \ (void) (explicit_memset(buf, 0, length)) From xeioex at nginx.com Wed Oct 3 12:56:59 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 03 Oct 2018 12:56:59 +0000 Subject: [njs] Fixed default implementation for nxt_explicit_memzero(). Message-ID: details: http://hg.nginx.org/njs/rev/b769c1037a57 branches: changeset: 619:b769c1037a57 user: Dmitry Volyntsev date: Wed Oct 03 15:38:19 2018 +0300 description: Fixed default implementation for nxt_explicit_memzero(). Thanks to David CARLIER. diffstat: nxt/nxt_string.h | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-) diffs (22 lines): diff -r 26ebe2b71708 -r b769c1037a57 nxt/nxt_string.h --- a/nxt/nxt_string.h Wed Oct 03 15:36:36 2018 +0300 +++ b/nxt/nxt_string.h Wed Oct 03 15:38:19 2018 +0300 @@ -57,8 +57,16 @@ nxt_upper_case(u_char c) #define nxt_explicit_memzero(buf, length) \ (void) (explicit_memset(buf, 0, length)) #else -#define nxt_explicit_memzero(buf, length) \ - nxt_memzero(buf, length) +nxt_inline void +nxt_explicit_memzero(u_char *buf, size_t length) +{ + volatile u_char *p = (volatile u_char *) buf; + + while (length != 0) { + *p++ = 0; + length--; + } +} #endif From vl at nginx.com Wed Oct 3 15:09:11 2018 From: vl at nginx.com (Vladimir Homutov) Date: Wed, 03 Oct 2018 15:09:11 +0000 Subject: [nginx] Version bump. Message-ID: details: http://hg.nginx.org/nginx/rev/aa1c9f846567 branches: changeset: 7370:aa1c9f846567 user: Vladimir Homutov date: Wed Oct 03 17:02:44 2018 +0300 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r 64721be763b6 -r aa1c9f846567 src/core/nginx.h --- a/src/core/nginx.h Tue Oct 02 18:13:52 2018 +0300 +++ b/src/core/nginx.h Wed Oct 03 17:02:44 2018 +0300 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1015005 -#define NGINX_VERSION "1.15.5" +#define nginx_version 1015006 +#define NGINX_VERSION "1.15.6" #define NGINX_VER "nginx/" NGINX_VERSION #ifdef NGX_BUILD From vl at nginx.com Wed Oct 3 15:09:12 2018 From: vl at nginx.com (Vladimir Homutov) Date: Wed, 03 Oct 2018 15:09:12 +0000 Subject: [nginx] Upstream: proxy_socket_keepalive and friends. Message-ID: details: http://hg.nginx.org/nginx/rev/8b68d50090e4 branches: changeset: 7371:8b68d50090e4 user: Vladimir Homutov date: Wed Oct 03 14:08:51 2018 +0300 description: Upstream: proxy_socket_keepalive and friends. The directives enable the use of the SO_KEEPALIVE option on upstream connections. By default, the value is left unchanged. diffstat: src/event/ngx_event_connect.c | 14 +++++++++++++- src/event/ngx_event_connect.h | 1 + src/http/modules/ngx_http_fastcgi_module.c | 11 +++++++++++ src/http/modules/ngx_http_grpc_module.c | 11 +++++++++++ src/http/modules/ngx_http_memcached_module.c | 11 +++++++++++ src/http/modules/ngx_http_proxy_module.c | 11 +++++++++++ src/http/modules/ngx_http_scgi_module.c | 11 +++++++++++ src/http/modules/ngx_http_uwsgi_module.c | 11 +++++++++++ src/http/ngx_http_upstream.c | 4 ++++ src/http/ngx_http_upstream.h | 1 + src/stream/ngx_stream_proxy_module.c | 16 ++++++++++++++++ 11 files changed, 101 insertions(+), 1 deletions(-) diffs (331 lines): diff -r aa1c9f846567 -r 8b68d50090e4 src/event/ngx_event_connect.c --- a/src/event/ngx_event_connect.c Wed Oct 03 17:02:44 2018 +0300 +++ b/src/event/ngx_event_connect.c Wed Oct 03 14:08:51 2018 +0300 @@ -20,7 +20,7 @@ static ngx_int_t ngx_event_connect_set_t ngx_int_t ngx_event_connect_peer(ngx_peer_connection_t *pc) { - int rc, type; + int rc, type, value; #if (NGX_HAVE_IP_BIND_ADDRESS_NO_PORT || NGX_LINUX) in_port_t port; #endif @@ -73,6 +73,18 @@ ngx_event_connect_peer(ngx_peer_connecti } } + if (pc->so_keepalive) { + value = 1; + + if (setsockopt(s, SOL_SOCKET, SO_KEEPALIVE, + (const void *) &value, sizeof(int)) + == -1) + { + ngx_log_error(NGX_LOG_ALERT, pc->log, ngx_socket_errno, + "setsockopt(SO_KEEPALIVE) failed, ignored"); + } + } + if (ngx_nonblocking(s) == -1) { ngx_log_error(NGX_LOG_ALERT, pc->log, ngx_socket_errno, ngx_nonblocking_n " failed"); diff -r aa1c9f846567 -r 8b68d50090e4 src/event/ngx_event_connect.h --- a/src/event/ngx_event_connect.h Wed Oct 03 17:02:44 2018 +0300 +++ b/src/event/ngx_event_connect.h Wed Oct 03 14:08:51 2018 +0300 @@ -62,6 +62,7 @@ struct ngx_peer_connection_s { unsigned cached:1; unsigned transparent:1; + unsigned so_keepalive:1; /* ngx_connection_log_error_e */ unsigned log_error:2; diff -r aa1c9f846567 -r 8b68d50090e4 src/http/modules/ngx_http_fastcgi_module.c --- a/src/http/modules/ngx_http_fastcgi_module.c Wed Oct 03 17:02:44 2018 +0300 +++ b/src/http/modules/ngx_http_fastcgi_module.c Wed Oct 03 14:08:51 2018 +0300 @@ -286,6 +286,13 @@ static ngx_command_t ngx_http_fastcgi_c offsetof(ngx_http_fastcgi_loc_conf_t, upstream.local), NULL }, + { ngx_string("fastcgi_socket_keepalive"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_fastcgi_loc_conf_t, upstream.socket_keepalive), + NULL }, + { ngx_string("fastcgi_connect_timeout"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_msec_slot, @@ -2721,6 +2728,7 @@ ngx_http_fastcgi_create_loc_conf(ngx_con conf->upstream.force_ranges = NGX_CONF_UNSET; conf->upstream.local = NGX_CONF_UNSET_PTR; + conf->upstream.socket_keepalive = NGX_CONF_UNSET; conf->upstream.connect_timeout = NGX_CONF_UNSET_MSEC; conf->upstream.send_timeout = NGX_CONF_UNSET_MSEC; @@ -2824,6 +2832,9 @@ ngx_http_fastcgi_merge_loc_conf(ngx_conf ngx_conf_merge_ptr_value(conf->upstream.local, prev->upstream.local, NULL); + ngx_conf_merge_value(conf->upstream.socket_keepalive, + prev->upstream.socket_keepalive, 0); + ngx_conf_merge_msec_value(conf->upstream.connect_timeout, prev->upstream.connect_timeout, 60000); diff -r aa1c9f846567 -r 8b68d50090e4 src/http/modules/ngx_http_grpc_module.c --- a/src/http/modules/ngx_http_grpc_module.c Wed Oct 03 17:02:44 2018 +0300 +++ b/src/http/modules/ngx_http_grpc_module.c Wed Oct 03 14:08:51 2018 +0300 @@ -248,6 +248,13 @@ static ngx_command_t ngx_http_grpc_comm offsetof(ngx_http_grpc_loc_conf_t, upstream.local), NULL }, + { ngx_string("grpc_socket_keepalive"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_grpc_loc_conf_t, upstream.socket_keepalive), + NULL }, + { ngx_string("grpc_connect_timeout"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_msec_slot, @@ -4150,6 +4157,7 @@ ngx_http_grpc_create_loc_conf(ngx_conf_t */ conf->upstream.local = NGX_CONF_UNSET_PTR; + conf->upstream.socket_keepalive = NGX_CONF_UNSET; conf->upstream.next_upstream_tries = NGX_CONF_UNSET_UINT; conf->upstream.connect_timeout = NGX_CONF_UNSET_MSEC; conf->upstream.send_timeout = NGX_CONF_UNSET_MSEC; @@ -4205,6 +4213,9 @@ ngx_http_grpc_merge_loc_conf(ngx_conf_t ngx_conf_merge_ptr_value(conf->upstream.local, prev->upstream.local, NULL); + ngx_conf_merge_value(conf->upstream.socket_keepalive, + prev->upstream.socket_keepalive, 0); + ngx_conf_merge_uint_value(conf->upstream.next_upstream_tries, prev->upstream.next_upstream_tries, 0); diff -r aa1c9f846567 -r 8b68d50090e4 src/http/modules/ngx_http_memcached_module.c --- a/src/http/modules/ngx_http_memcached_module.c Wed Oct 03 17:02:44 2018 +0300 +++ b/src/http/modules/ngx_http_memcached_module.c Wed Oct 03 14:08:51 2018 +0300 @@ -67,6 +67,13 @@ static ngx_command_t ngx_http_memcached offsetof(ngx_http_memcached_loc_conf_t, upstream.local), NULL }, + { ngx_string("memcached_socket_keepalive"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_memcached_loc_conf_t, upstream.socket_keepalive), + NULL }, + { ngx_string("memcached_connect_timeout"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_msec_slot, @@ -595,6 +602,7 @@ ngx_http_memcached_create_loc_conf(ngx_c */ conf->upstream.local = NGX_CONF_UNSET_PTR; + conf->upstream.socket_keepalive = NGX_CONF_UNSET; conf->upstream.next_upstream_tries = NGX_CONF_UNSET_UINT; conf->upstream.connect_timeout = NGX_CONF_UNSET_MSEC; conf->upstream.send_timeout = NGX_CONF_UNSET_MSEC; @@ -634,6 +642,9 @@ ngx_http_memcached_merge_loc_conf(ngx_co ngx_conf_merge_ptr_value(conf->upstream.local, prev->upstream.local, NULL); + ngx_conf_merge_value(conf->upstream.socket_keepalive, + prev->upstream.socket_keepalive, 0); + ngx_conf_merge_uint_value(conf->upstream.next_upstream_tries, prev->upstream.next_upstream_tries, 0); diff -r aa1c9f846567 -r 8b68d50090e4 src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c Wed Oct 03 17:02:44 2018 +0300 +++ b/src/http/modules/ngx_http_proxy_module.c Wed Oct 03 14:08:51 2018 +0300 @@ -324,6 +324,13 @@ static ngx_command_t ngx_http_proxy_com offsetof(ngx_http_proxy_loc_conf_t, upstream.local), NULL }, + { ngx_string("proxy_socket_keepalive"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_proxy_loc_conf_t, upstream.socket_keepalive), + NULL }, + { ngx_string("proxy_connect_timeout"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_msec_slot, @@ -2833,6 +2840,7 @@ ngx_http_proxy_create_loc_conf(ngx_conf_ conf->upstream.force_ranges = NGX_CONF_UNSET; conf->upstream.local = NGX_CONF_UNSET_PTR; + conf->upstream.socket_keepalive = NGX_CONF_UNSET; conf->upstream.connect_timeout = NGX_CONF_UNSET_MSEC; conf->upstream.send_timeout = NGX_CONF_UNSET_MSEC; @@ -2953,6 +2961,9 @@ ngx_http_proxy_merge_loc_conf(ngx_conf_t ngx_conf_merge_ptr_value(conf->upstream.local, prev->upstream.local, NULL); + ngx_conf_merge_value(conf->upstream.socket_keepalive, + prev->upstream.socket_keepalive, 0); + ngx_conf_merge_msec_value(conf->upstream.connect_timeout, prev->upstream.connect_timeout, 60000); diff -r aa1c9f846567 -r 8b68d50090e4 src/http/modules/ngx_http_scgi_module.c --- a/src/http/modules/ngx_http_scgi_module.c Wed Oct 03 17:02:44 2018 +0300 +++ b/src/http/modules/ngx_http_scgi_module.c Wed Oct 03 14:08:51 2018 +0300 @@ -143,6 +143,13 @@ static ngx_command_t ngx_http_scgi_comma offsetof(ngx_http_scgi_loc_conf_t, upstream.local), NULL }, + { ngx_string("scgi_socket_keepalive"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_scgi_loc_conf_t, upstream.socket_keepalive), + NULL }, + { ngx_string("scgi_connect_timeout"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_msec_slot, @@ -1200,6 +1207,7 @@ ngx_http_scgi_create_loc_conf(ngx_conf_t conf->upstream.force_ranges = NGX_CONF_UNSET; conf->upstream.local = NGX_CONF_UNSET_PTR; + conf->upstream.socket_keepalive = NGX_CONF_UNSET; conf->upstream.connect_timeout = NGX_CONF_UNSET_MSEC; conf->upstream.send_timeout = NGX_CONF_UNSET_MSEC; @@ -1298,6 +1306,9 @@ ngx_http_scgi_merge_loc_conf(ngx_conf_t ngx_conf_merge_ptr_value(conf->upstream.local, prev->upstream.local, NULL); + ngx_conf_merge_value(conf->upstream.socket_keepalive, + prev->upstream.socket_keepalive, 0); + ngx_conf_merge_msec_value(conf->upstream.connect_timeout, prev->upstream.connect_timeout, 60000); diff -r aa1c9f846567 -r 8b68d50090e4 src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c Wed Oct 03 17:02:44 2018 +0300 +++ b/src/http/modules/ngx_http_uwsgi_module.c Wed Oct 03 14:08:51 2018 +0300 @@ -204,6 +204,13 @@ static ngx_command_t ngx_http_uwsgi_comm offsetof(ngx_http_uwsgi_loc_conf_t, upstream.local), NULL }, + { ngx_string("uwsgi_socket_keepalive"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, upstream.socket_keepalive), + NULL }, + { ngx_string("uwsgi_connect_timeout"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_msec_slot, @@ -1413,6 +1420,7 @@ ngx_http_uwsgi_create_loc_conf(ngx_conf_ conf->upstream.force_ranges = NGX_CONF_UNSET; conf->upstream.local = NGX_CONF_UNSET_PTR; + conf->upstream.socket_keepalive = NGX_CONF_UNSET; conf->upstream.connect_timeout = NGX_CONF_UNSET_MSEC; conf->upstream.send_timeout = NGX_CONF_UNSET_MSEC; @@ -1519,6 +1527,9 @@ ngx_http_uwsgi_merge_loc_conf(ngx_conf_t ngx_conf_merge_ptr_value(conf->upstream.local, prev->upstream.local, NULL); + ngx_conf_merge_value(conf->upstream.socket_keepalive, + prev->upstream.socket_keepalive, 0); + ngx_conf_merge_msec_value(conf->upstream.connect_timeout, prev->upstream.connect_timeout, 60000); diff -r aa1c9f846567 -r 8b68d50090e4 src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c Wed Oct 03 17:02:44 2018 +0300 +++ b/src/http/ngx_http_upstream.c Wed Oct 03 14:08:51 2018 +0300 @@ -628,6 +628,10 @@ ngx_http_upstream_init_request(ngx_http_ return; } + if (u->conf->socket_keepalive) { + u->peer.so_keepalive = 1; + } + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); u->output.alignment = clcf->directio_alignment; diff -r aa1c9f846567 -r 8b68d50090e4 src/http/ngx_http_upstream.h --- a/src/http/ngx_http_upstream.h Wed Oct 03 17:02:44 2018 +0300 +++ b/src/http/ngx_http_upstream.h Wed Oct 03 14:08:51 2018 +0300 @@ -188,6 +188,7 @@ typedef struct { ngx_array_t *pass_headers; ngx_http_upstream_local_t *local; + ngx_flag_t socket_keepalive; #if (NGX_HTTP_CACHE) ngx_shm_zone_t *cache_zone; diff -r aa1c9f846567 -r 8b68d50090e4 src/stream/ngx_stream_proxy_module.c --- a/src/stream/ngx_stream_proxy_module.c Wed Oct 03 17:02:44 2018 +0300 +++ b/src/stream/ngx_stream_proxy_module.c Wed Oct 03 14:08:51 2018 +0300 @@ -31,6 +31,7 @@ typedef struct { ngx_flag_t next_upstream; ngx_flag_t proxy_protocol; ngx_stream_upstream_local_t *local; + ngx_flag_t socket_keepalive; #if (NGX_STREAM_SSL) ngx_flag_t ssl_enable; @@ -136,6 +137,13 @@ static ngx_command_t ngx_stream_proxy_c 0, NULL }, + { ngx_string("proxy_socket_keepalive"), + NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_STREAM_SRV_CONF_OFFSET, + offsetof(ngx_stream_proxy_srv_conf_t, socket_keepalive), + NULL }, + { ngx_string("proxy_connect_timeout"), NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_msec_slot, @@ -388,6 +396,10 @@ ngx_stream_proxy_handler(ngx_stream_sess return; } + if (pscf->socket_keepalive) { + u->peer.so_keepalive = 1; + } + u->peer.type = c->type; u->start_sec = ngx_time(); @@ -1898,6 +1910,7 @@ ngx_stream_proxy_create_srv_conf(ngx_con conf->next_upstream = NGX_CONF_UNSET; conf->proxy_protocol = NGX_CONF_UNSET; conf->local = NGX_CONF_UNSET_PTR; + conf->socket_keepalive = NGX_CONF_UNSET; #if (NGX_STREAM_SSL) conf->ssl_enable = NGX_CONF_UNSET; @@ -1948,6 +1961,9 @@ ngx_stream_proxy_merge_srv_conf(ngx_conf ngx_conf_merge_ptr_value(conf->local, prev->local, NULL); + ngx_conf_merge_value(conf->socket_keepalive, + prev->socket_keepalive, 0); + #if (NGX_STREAM_SSL) ngx_conf_merge_value(conf->ssl_enable, prev->ssl_enable, 0); From arut at nginx.com Thu Oct 4 09:32:26 2018 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 4 Oct 2018 12:32:26 +0300 Subject: cache: move open to thread pool In-Reply-To: <20180913181023.GW56558@mdounin.ru> References: <20180808181653.GX56558@mdounin.ru> <20180810113946.GG56558@mdounin.ru> <20180903160903.GI56558@mdounin.ru> <20180913181023.GW56558@mdounin.ru> Message-ID: <20181004093226.GD62311@Romans-MacBook-Air.local> Hi, On Thu, Sep 13, 2018 at 09:10:23PM +0300, Maxim Dounin wrote: > Hello! > > On Tue, Sep 04, 2018 at 04:58:05PM -0700, Ka-Hing Cheung via nginx-devel wrote: > > > On Mon, Sep 3, 2018 at 9:09 AM, Maxim Dounin wrote: [..] Here's another approach to thread open. This time it's 4 patches: - #1 a small open file cache refactoring - #2 thread open in open file cache - #3 thread open in http static module - #4 thread open in http file cache -- Roman Arutyunyan -------------- next part -------------- # HG changeset patch # User Roman Arutyunyan # Date 1538143636 -10800 # Fri Sep 28 17:07:16 2018 +0300 # Node ID 1ebbfd3e3cd814e4428561b814a5684c69668ba4 # Parent cd4fa2fab8d8f1a2d03841249230bed76d318502 Prepared open file cache for async open. Specifically, several open file cache internal functions now receive pool instead of log and return file descriptor in the ngx_open_file_info_t structure instead of returning it directtly. diff --git a/src/core/ngx_open_file_cache.c b/src/core/ngx_open_file_cache.c --- a/src/core/ngx_open_file_cache.c +++ b/src/core/ngx_open_file_cache.c @@ -30,13 +30,13 @@ static ngx_int_t ngx_file_o_path_info(ng ngx_log_t *log); #endif #endif -static ngx_fd_t ngx_open_file_wrapper(ngx_str_t *name, +static ngx_int_t ngx_open_file_wrapper(ngx_str_t *name, ngx_open_file_info_t *of, ngx_int_t mode, ngx_int_t create, - ngx_int_t access, ngx_log_t *log); + ngx_int_t access, ngx_pool_t *pool); static ngx_int_t ngx_file_info_wrapper(ngx_str_t *name, - ngx_open_file_info_t *of, ngx_file_info_t *fi, ngx_log_t *log); + ngx_open_file_info_t *of, ngx_file_info_t *fi, ngx_pool_t *pool); static ngx_int_t ngx_open_and_stat_file(ngx_str_t *name, - ngx_open_file_info_t *of, ngx_log_t *log); + ngx_open_file_info_t *of, ngx_pool_t *pool); static void ngx_open_file_add_event(ngx_open_file_cache_t *cache, ngx_cached_open_file_t *file, ngx_open_file_info_t *of, ngx_log_t *log); static void ngx_open_file_cleanup(void *data); @@ -160,10 +160,9 @@ ngx_open_cached_file(ngx_open_file_cache if (of->test_only) { - if (ngx_file_info_wrapper(name, of, &fi, pool->log) - == NGX_FILE_ERROR) - { - return NGX_ERROR; + rc = ngx_file_info_wrapper(name, of, &fi, pool); + if (rc != NGX_OK) { + return rc; } of->uniq = ngx_file_uniq(&fi); @@ -183,7 +182,7 @@ ngx_open_cached_file(ngx_open_file_cache return NGX_ERROR; } - rc = ngx_open_and_stat_file(name, of, pool->log); + rc = ngx_open_and_stat_file(name, of, pool); if (rc == NGX_OK && !of->is_dir) { cln->handler = ngx_pool_cleanup_file; @@ -218,7 +217,7 @@ ngx_open_cached_file(ngx_open_file_cache /* file was not used often enough to keep open */ - rc = ngx_open_and_stat_file(name, of, pool->log); + rc = ngx_open_and_stat_file(name, of, pool); if (rc != NGX_OK && (of->err == 0 || !of->errors)) { goto failed; @@ -286,7 +285,7 @@ ngx_open_cached_file(ngx_open_file_cache of->fd = file->fd; of->uniq = file->uniq; - rc = ngx_open_and_stat_file(name, of, pool->log); + rc = ngx_open_and_stat_file(name, of, pool); if (rc != NGX_OK && (of->err == 0 || !of->errors)) { goto failed; @@ -351,7 +350,7 @@ ngx_open_cached_file(ngx_open_file_cache /* not found */ - rc = ngx_open_and_stat_file(name, of, pool->log); + rc = ngx_open_and_stat_file(name, of, pool); if (rc != NGX_OK && (of->err == 0 || !of->errors)) { goto failed; @@ -610,40 +609,38 @@ ngx_file_o_path_info(ngx_fd_t fd, ngx_fi #endif /* NGX_HAVE_OPENAT */ -static ngx_fd_t +static ngx_int_t ngx_open_file_wrapper(ngx_str_t *name, ngx_open_file_info_t *of, - ngx_int_t mode, ngx_int_t create, ngx_int_t access, ngx_log_t *log) + ngx_int_t mode, ngx_int_t create, ngx_int_t access, ngx_pool_t *pool) { - ngx_fd_t fd; - #if !(NGX_HAVE_OPENAT) - fd = ngx_open_file(name->data, mode, create, access); + of->fd = ngx_open_file(name->data, mode, create, access); - if (fd == NGX_INVALID_FILE) { + if (of->fd == NGX_INVALID_FILE) { of->err = ngx_errno; of->failed = ngx_open_file_n; - return NGX_INVALID_FILE; + return NGX_ERROR; } - return fd; + return NGX_OK; #else - u_char *p, *cp, *end; - ngx_fd_t at_fd; - ngx_str_t at_name; + u_char *p, *cp, *end; + ngx_fd_t fd, at_fd; + ngx_str_t at_name; if (of->disable_symlinks == NGX_DISABLE_SYMLINKS_OFF) { - fd = ngx_open_file(name->data, mode, create, access); + of->fd = ngx_open_file(name->data, mode, create, access); - if (fd == NGX_INVALID_FILE) { + if (of->fd == NGX_INVALID_FILE) { of->err = ngx_errno; of->failed = ngx_open_file_n; - return NGX_INVALID_FILE; + return NGX_ERROR; } - return fd; + return NGX_OK; } p = name->data; @@ -665,7 +662,7 @@ ngx_open_file_wrapper(ngx_str_t *name, n if (at_fd == NGX_INVALID_FILE) { of->err = ngx_errno; of->failed = ngx_open_file_n; - return NGX_INVALID_FILE; + return NGX_ERROR; } at_name.len = of->disable_symlinks_from; @@ -680,7 +677,7 @@ ngx_open_file_wrapper(ngx_str_t *name, n if (at_fd == NGX_INVALID_FILE) { of->err = ngx_errno; of->failed = ngx_openat_file_n; - return NGX_INVALID_FILE; + return NGX_ERROR; } at_name.len = 1; @@ -706,7 +703,7 @@ ngx_open_file_wrapper(ngx_str_t *name, n if (of->disable_symlinks == NGX_DISABLE_SYMLINKS_NOTOWNER) { fd = ngx_openat_file_owner(at_fd, p, NGX_FILE_SEARCH|NGX_FILE_NONBLOCK, - NGX_FILE_OPEN, 0, log); + NGX_FILE_OPEN, 0, pool->log); } else { fd = ngx_openat_file(at_fd, p, @@ -723,7 +720,7 @@ ngx_open_file_wrapper(ngx_str_t *name, n } if (at_fd != NGX_AT_FDCWD && ngx_close_file(at_fd) == NGX_FILE_ERROR) { - ngx_log_error(NGX_LOG_ALERT, log, ngx_errno, + ngx_log_error(NGX_LOG_ALERT, pool->log, ngx_errno, ngx_close_file_n " \"%V\" failed", &at_name); } @@ -752,7 +749,7 @@ ngx_open_file_wrapper(ngx_str_t *name, n if (of->disable_symlinks == NGX_DISABLE_SYMLINKS_NOTOWNER && !(create & (NGX_FILE_CREATE_OR_OPEN|NGX_FILE_TRUNCATE))) { - fd = ngx_openat_file_owner(at_fd, p, mode, create, access, log); + fd = ngx_openat_file_owner(at_fd, p, mode, create, access, pool->log); } else { fd = ngx_openat_file(at_fd, p, mode|NGX_FILE_NOFOLLOW, create, access); @@ -768,18 +765,20 @@ done: failed: if (at_fd != NGX_AT_FDCWD && ngx_close_file(at_fd) == NGX_FILE_ERROR) { - ngx_log_error(NGX_LOG_ALERT, log, ngx_errno, + ngx_log_error(NGX_LOG_ALERT, pool->log, ngx_errno, ngx_close_file_n " \"%V\" failed", &at_name); } - return fd; + of->fd = fd; + + return fd == NGX_INVALID_FILE ? NGX_ERROR : NGX_OK; #endif } static ngx_int_t ngx_file_info_wrapper(ngx_str_t *name, ngx_open_file_info_t *of, - ngx_file_info_t *fi, ngx_log_t *log) + ngx_file_info_t *fi, ngx_pool_t *pool) { ngx_int_t rc; @@ -790,15 +789,13 @@ ngx_file_info_wrapper(ngx_str_t *name, n if (rc == NGX_FILE_ERROR) { of->err = ngx_errno; of->failed = ngx_file_info_n; - return NGX_FILE_ERROR; + return NGX_ERROR; } - return rc; + return NGX_OK; #else - ngx_fd_t fd; - if (of->disable_symlinks == NGX_DISABLE_SYMLINKS_OFF) { rc = ngx_file_info(name->data, fi); @@ -806,48 +803,55 @@ ngx_file_info_wrapper(ngx_str_t *name, n if (rc == NGX_FILE_ERROR) { of->err = ngx_errno; of->failed = ngx_file_info_n; - return NGX_FILE_ERROR; + return NGX_ERROR; } + return NGX_OK; + } + + rc = ngx_open_file_wrapper(name, of, NGX_FILE_RDONLY|NGX_FILE_NONBLOCK, + NGX_FILE_OPEN, 0, pool); + if (rc != NGX_OK) { return rc; } - fd = ngx_open_file_wrapper(name, of, NGX_FILE_RDONLY|NGX_FILE_NONBLOCK, - NGX_FILE_OPEN, 0, log); - - if (fd == NGX_INVALID_FILE) { - return NGX_FILE_ERROR; - } - - rc = ngx_fd_info(fd, fi); + rc = ngx_fd_info(of->fd, fi); if (rc == NGX_FILE_ERROR) { of->err = ngx_errno; of->failed = ngx_fd_info_n; } - if (ngx_close_file(fd) == NGX_FILE_ERROR) { - ngx_log_error(NGX_LOG_ALERT, log, ngx_errno, + if (ngx_close_file(of->fd) == NGX_FILE_ERROR) { + ngx_log_error(NGX_LOG_ALERT, pool->log, ngx_errno, ngx_close_file_n " \"%V\" failed", name); } - return rc; + of->fd = NGX_INVALID_FILE; + + return rc == NGX_FILE_ERROR ? NGX_ERROR : NGX_OK; #endif } static ngx_int_t ngx_open_and_stat_file(ngx_str_t *name, ngx_open_file_info_t *of, - ngx_log_t *log) + ngx_pool_t *pool) { - ngx_fd_t fd; + ngx_int_t rc; ngx_file_info_t fi; if (of->fd != NGX_INVALID_FILE) { - if (ngx_file_info_wrapper(name, of, &fi, log) == NGX_FILE_ERROR) { + rc = ngx_file_info_wrapper(name, of, &fi, pool); + + if (rc == NGX_AGAIN) { + return NGX_AGAIN; + } + + if (rc != NGX_OK) { of->fd = NGX_INVALID_FILE; - return NGX_ERROR; + return rc; } if (of->uniq == ngx_file_uniq(&fi)) { @@ -856,9 +860,15 @@ ngx_open_and_stat_file(ngx_str_t *name, } else if (of->test_dir) { - if (ngx_file_info_wrapper(name, of, &fi, log) == NGX_FILE_ERROR) { + rc = ngx_file_info_wrapper(name, of, &fi, pool); + + if (rc == NGX_AGAIN) { + return NGX_AGAIN; + } + + if (rc != NGX_OK) { of->fd = NGX_INVALID_FILE; - return NGX_ERROR; + return rc; } if (ngx_is_dir(&fi)) { @@ -873,26 +883,25 @@ ngx_open_and_stat_file(ngx_str_t *name, * This flag has no effect on a regular files. */ - fd = ngx_open_file_wrapper(name, of, NGX_FILE_RDONLY|NGX_FILE_NONBLOCK, - NGX_FILE_OPEN, 0, log); + rc = ngx_open_file_wrapper(name, of, NGX_FILE_RDONLY|NGX_FILE_NONBLOCK, + NGX_FILE_OPEN, 0, pool); } else { - fd = ngx_open_file_wrapper(name, of, NGX_FILE_APPEND, + rc = ngx_open_file_wrapper(name, of, NGX_FILE_APPEND, NGX_FILE_CREATE_OR_OPEN, - NGX_FILE_DEFAULT_ACCESS, log); + NGX_FILE_DEFAULT_ACCESS, pool); } - if (fd == NGX_INVALID_FILE) { - of->fd = NGX_INVALID_FILE; - return NGX_ERROR; + if (rc != NGX_OK) { + return rc; } - if (ngx_fd_info(fd, &fi) == NGX_FILE_ERROR) { - ngx_log_error(NGX_LOG_CRIT, log, ngx_errno, + if (ngx_fd_info(of->fd, &fi) == NGX_FILE_ERROR) { + ngx_log_error(NGX_LOG_CRIT, pool->log, ngx_errno, ngx_fd_info_n " \"%V\" failed", name); - if (ngx_close_file(fd) == NGX_FILE_ERROR) { - ngx_log_error(NGX_LOG_ALERT, log, ngx_errno, + if (ngx_close_file(of->fd) == NGX_FILE_ERROR) { + ngx_log_error(NGX_LOG_ALERT, pool->log, ngx_errno, ngx_close_file_n " \"%V\" failed", name); } @@ -902,26 +911,24 @@ ngx_open_and_stat_file(ngx_str_t *name, } if (ngx_is_dir(&fi)) { - if (ngx_close_file(fd) == NGX_FILE_ERROR) { - ngx_log_error(NGX_LOG_ALERT, log, ngx_errno, + if (ngx_close_file(of->fd) == NGX_FILE_ERROR) { + ngx_log_error(NGX_LOG_ALERT, pool->log, ngx_errno, ngx_close_file_n " \"%V\" failed", name); } of->fd = NGX_INVALID_FILE; } else { - of->fd = fd; - if (of->read_ahead && ngx_file_size(&fi) > NGX_MIN_READ_AHEAD) { - if (ngx_read_ahead(fd, of->read_ahead) == NGX_ERROR) { - ngx_log_error(NGX_LOG_ALERT, log, ngx_errno, + if (ngx_read_ahead(of->fd, of->read_ahead) == NGX_ERROR) { + ngx_log_error(NGX_LOG_ALERT, pool->log, ngx_errno, ngx_read_ahead_n " \"%V\" failed", name); } } if (of->directio <= ngx_file_size(&fi)) { - if (ngx_directio_on(fd) == NGX_FILE_ERROR) { - ngx_log_error(NGX_LOG_ALERT, log, ngx_errno, + if (ngx_directio_on(of->fd) == NGX_FILE_ERROR) { + ngx_log_error(NGX_LOG_ALERT, pool->log, ngx_errno, ngx_directio_on_n " \"%V\" failed", name); } else { -------------- next part -------------- # HG changeset patch # User Roman Arutyunyan # Date 1538644508 -10800 # Thu Oct 04 12:15:08 2018 +0300 # Node ID 82f2dea6f3c6e3edf78968ab9f623d7b92e074fd # Parent 1ebbfd3e3cd814e4428561b814a5684c69668ba4 Threaded open support in open file cache. diff --git a/src/core/ngx_file.h b/src/core/ngx_file.h --- a/src/core/ngx_file.h +++ b/src/core/ngx_file.h @@ -17,6 +17,7 @@ struct ngx_file_s { ngx_fd_t fd; ngx_str_t name; ngx_file_info_t info; + ngx_err_t err; off_t offset; off_t sys_offset; diff --git a/src/core/ngx_open_file_cache.c b/src/core/ngx_open_file_cache.c --- a/src/core/ngx_open_file_cache.c +++ b/src/core/ngx_open_file_cache.c @@ -177,14 +177,21 @@ ngx_open_cached_file(ngx_open_file_cache return NGX_OK; } - cln = ngx_pool_cleanup_add(pool, sizeof(ngx_pool_cleanup_file_t)); - if (cln == NULL) { - return NGX_ERROR; - } - rc = ngx_open_and_stat_file(name, of, pool); if (rc == NGX_OK && !of->is_dir) { + cln = ngx_pool_cleanup_add(pool, sizeof(ngx_pool_cleanup_file_t)); + + if (cln == NULL) { + if (ngx_close_file(of->fd) == NGX_FILE_ERROR) { + ngx_log_error(NGX_LOG_ALERT, pool->log, ngx_errno, + ngx_close_file_n " \"%s\" failed", + name->data); + } + + return NGX_ERROR; + } + cln->handler = ngx_pool_cleanup_file; clnf = cln->data; @@ -196,6 +203,11 @@ ngx_open_cached_file(ngx_open_file_cache return rc; } +#if (NGX_THREADS) + /* disable thread open if cache is enabled */ + of->thread_handler = NULL; +#endif + cln = ngx_pool_cleanup_add(pool, sizeof(ngx_open_file_cache_cleanup_t)); if (cln == NULL) { return NGX_ERROR; @@ -613,6 +625,40 @@ static ngx_int_t ngx_open_file_wrapper(ngx_str_t *name, ngx_open_file_info_t *of, ngx_int_t mode, ngx_int_t create, ngx_int_t access, ngx_pool_t *pool) { +#if (NGX_THREADS) + + if (of->thread_handler) { + ngx_int_t rc; + ngx_file_t file; + + ngx_memzero(&file, sizeof(ngx_file_t)); + + file.log = pool->log; + file.fd = NGX_INVALID_FILE; + file.thread_handler = of->thread_handler; + file.thread_ctx = of->thread_ctx; + file.thread_task = of->thread_task; + + rc = ngx_thread_open(&file, name->data, mode, create, access, pool); + + if (rc == NGX_AGAIN) { + of->thread_task = file.thread_task; + return NGX_AGAIN; + } + + if (rc != NGX_OK) { + of->err = file.err; + of->failed = ngx_open_file_n; + return NGX_ERROR; + } + + of->fd = file.fd; + + return NGX_OK; + } + +#endif + #if !(NGX_HAVE_OPENAT) of->fd = ngx_open_file(name->data, mode, create, access); diff --git a/src/core/ngx_open_file_cache.h b/src/core/ngx_open_file_cache.h --- a/src/core/ngx_open_file_cache.h +++ b/src/core/ngx_open_file_cache.h @@ -32,6 +32,13 @@ typedef struct { ngx_uint_t min_uses; +#if (NGX_THREADS || NGX_COMPAT) + ngx_int_t (*thread_handler)(ngx_thread_task_t *task, + ngx_file_t *file); + void *thread_ctx; + ngx_thread_task_t *thread_task; +#endif + #if (NGX_HAVE_OPENAT) size_t disable_symlinks_from; unsigned disable_symlinks:2; diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -420,6 +420,13 @@ static ngx_command_t ngx_http_core_comm offsetof(ngx_http_core_loc_conf_t, aio_write), NULL }, + { ngx_string("aio_open"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_core_loc_conf_t, aio_open), + NULL }, + { ngx_string("read_ahead"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_size_slot, @@ -3380,6 +3387,7 @@ ngx_http_core_create_loc_conf(ngx_conf_t clcf->subrequest_output_buffer_size = NGX_CONF_UNSET_SIZE; clcf->aio = NGX_CONF_UNSET; clcf->aio_write = NGX_CONF_UNSET; + clcf->aio_open = NGX_CONF_UNSET; #if (NGX_THREADS) clcf->thread_pool = NGX_CONF_UNSET_PTR; clcf->thread_pool_value = NGX_CONF_UNSET_PTR; @@ -3605,6 +3613,7 @@ ngx_http_core_merge_loc_conf(ngx_conf_t (size_t) ngx_pagesize); ngx_conf_merge_value(conf->aio, prev->aio, NGX_HTTP_AIO_OFF); ngx_conf_merge_value(conf->aio_write, prev->aio_write, 0); + ngx_conf_merge_value(conf->aio_open, prev->aio_open, 0); #if (NGX_THREADS) ngx_conf_merge_ptr_value(conf->thread_pool, prev->thread_pool, NULL); ngx_conf_merge_ptr_value(conf->thread_pool_value, prev->thread_pool_value, diff --git a/src/http/ngx_http_core_module.h b/src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h +++ b/src/http/ngx_http_core_module.h @@ -382,6 +382,7 @@ struct ngx_http_core_loc_conf_s { ngx_flag_t sendfile; /* sendfile */ ngx_flag_t aio; /* aio */ ngx_flag_t aio_write; /* aio_write */ + ngx_flag_t aio_open; /* aio_open */ ngx_flag_t tcp_nopush; /* tcp_nopush */ ngx_flag_t tcp_nodelay; /* tcp_nodelay */ ngx_flag_t reset_timedout_connection; /* reset_timedout_connection */ diff --git a/src/os/unix/ngx_files.c b/src/os/unix/ngx_files.c --- a/src/os/unix/ngx_files.c +++ b/src/os/unix/ngx_files.c @@ -11,6 +11,7 @@ #if (NGX_THREADS) #include +static void ngx_thread_open_handler(void *data, ngx_log_t *log); static void ngx_thread_read_handler(void *data, ngx_log_t *log); static void ngx_thread_write_chain_to_file_handler(void *data, ngx_log_t *log); #endif @@ -77,20 +78,112 @@ ngx_read_file(ngx_file_t *file, u_char * #if (NGX_THREADS) +typedef enum { + NGX_THREAD_FILE_OPEN = 1, + NGX_THREAD_FILE_READ, + NGX_THREAD_FILE_WRITE +} ngx_thread_file_op_e; + + typedef struct { ngx_fd_t fd; - ngx_uint_t write; /* unsigned write:1; */ + u_char *name; + ngx_uint_t op; /* ngx_thread_file_op_e */ u_char *buf; size_t size; ngx_chain_t *chain; off_t offset; + ngx_int_t mode; + ngx_int_t create; + ngx_int_t access; + size_t nbytes; ngx_err_t err; } ngx_thread_file_ctx_t; +ngx_int_t +ngx_thread_open(ngx_file_t *file, u_char *name, ngx_int_t mode, + ngx_int_t create, ngx_int_t access, ngx_pool_t *pool) +{ + ngx_thread_task_t *task; + ngx_thread_file_ctx_t *ctx; + + ngx_log_debug1(NGX_LOG_DEBUG_CORE, file->log, 0, + "thread open: \"%s\"", name); + + task = file->thread_task; + + if (task == NULL) { + task = ngx_thread_task_alloc(pool, sizeof(ngx_thread_file_ctx_t)); + if (task == NULL) { + return NGX_ERROR; + } + + file->thread_task = task; + } + + ctx = task->ctx; + + if (task->event.complete) { + task->event.complete = 0; + + if (ctx->op != NGX_THREAD_FILE_OPEN) { + ngx_log_error(NGX_LOG_ALERT, file->log, 0, + "invalid thread operation, open expected"); + return NGX_ERROR; + } + + if (ctx->err) { + file->err = ctx->err; + return NGX_ERROR; + } + + file->fd = ctx->fd; + + return NGX_OK; + } + + task->handler = ngx_thread_open_handler; + + ctx->op = NGX_THREAD_FILE_OPEN; + + ctx->name = name; + ctx->mode = mode; + ctx->create = create; + ctx->access = access; + + if (file->thread_handler(task, file) != NGX_OK) { + return NGX_ERROR; + } + + return NGX_AGAIN; +} + + +static void +ngx_thread_open_handler(void *data, ngx_log_t *log) +{ + ngx_thread_file_ctx_t *ctx = data; + + ngx_fd_t fd; + + ngx_log_debug0(NGX_LOG_DEBUG_CORE, log, 0, "thread open handler"); + + fd = ngx_open_file(ctx->name, ctx->mode, ctx->create, ctx->access); + + if (fd == NGX_INVALID_FILE) { + ctx->err = ngx_errno; + + } else { + ctx->fd = fd; + ctx->err = 0; + } +} + + ssize_t ngx_thread_read(ngx_file_t *file, u_char *buf, size_t size, off_t offset, ngx_pool_t *pool) @@ -118,9 +211,9 @@ ngx_thread_read(ngx_file_t *file, u_char if (task->event.complete) { task->event.complete = 0; - if (ctx->write) { + if (ctx->op != NGX_THREAD_FILE_READ) { ngx_log_error(NGX_LOG_ALERT, file->log, 0, - "invalid thread call, read instead of write"); + "invalid thread operation, read expected"); return NGX_ERROR; } @@ -135,7 +228,7 @@ ngx_thread_read(ngx_file_t *file, u_char task->handler = ngx_thread_read_handler; - ctx->write = 0; + ctx->op = NGX_THREAD_FILE_READ; ctx->fd = file->fd; ctx->buf = buf; @@ -501,9 +594,9 @@ ngx_thread_write_chain_to_file(ngx_file_ if (task->event.complete) { task->event.complete = 0; - if (!ctx->write) { + if (ctx->op != NGX_THREAD_FILE_WRITE) { ngx_log_error(NGX_LOG_ALERT, file->log, 0, - "invalid thread call, write instead of read"); + "invalid thread operation, write expected"); return NGX_ERROR; } @@ -519,7 +612,7 @@ ngx_thread_write_chain_to_file(ngx_file_ task->handler = ngx_thread_write_chain_to_file_handler; - ctx->write = 1; + ctx->op = NGX_THREAD_FILE_WRITE; ctx->fd = file->fd; ctx->chain = cl; diff --git a/src/os/unix/ngx_files.h b/src/os/unix/ngx_files.h --- a/src/os/unix/ngx_files.h +++ b/src/os/unix/ngx_files.h @@ -385,6 +385,8 @@ extern ngx_uint_t ngx_file_aio; #endif #if (NGX_THREADS) +ngx_int_t ngx_thread_open(ngx_file_t *file, u_char *name, + ngx_int_t mode, ngx_int_t create, ngx_int_t access, ngx_pool_t *pool); ssize_t ngx_thread_read(ngx_file_t *file, u_char *buf, size_t size, off_t offset, ngx_pool_t *pool); ssize_t ngx_thread_write_chain_to_file(ngx_file_t *file, ngx_chain_t *cl, -------------- next part -------------- # HG changeset patch # User Roman Arutyunyan # Date 1538403503 -10800 # Mon Oct 01 17:18:23 2018 +0300 # Node ID 68e2acd8c4cf81cdbe57c0a83bbafe43072cb9b5 # Parent 82f2dea6f3c6e3edf78968ab9f623d7b92e074fd Static: support for threaded open. diff --git a/src/http/modules/ngx_http_static_module.c b/src/http/modules/ngx_http_static_module.c --- a/src/http/modules/ngx_http_static_module.c +++ b/src/http/modules/ngx_http_static_module.c @@ -11,7 +11,21 @@ static ngx_int_t ngx_http_static_handler(ngx_http_request_t *r); +static ngx_int_t ngx_http_static_send(ngx_http_request_t *r); static ngx_int_t ngx_http_static_init(ngx_conf_t *cf); +#if (NGX_THREADS) +static void ngx_http_static_write_event_handler(ngx_http_request_t *r); +static ngx_int_t ngx_http_static_thread_handler(ngx_thread_task_t *task, + ngx_file_t *file); +static void ngx_http_static_thread_event_handler(ngx_event_t *ev); +#endif + + +typedef struct { + u_char *last; + ngx_str_t path; + ngx_open_file_info_t of; +} ngx_http_static_ctx_t; static ngx_http_module_t ngx_http_static_module_ctx = { @@ -48,15 +62,9 @@ ngx_module_t ngx_http_static_module = { static ngx_int_t ngx_http_static_handler(ngx_http_request_t *r) { - u_char *last, *location; - size_t root, len; - ngx_str_t path; + size_t root; ngx_int_t rc; - ngx_uint_t level; - ngx_log_t *log; - ngx_buf_t *b; - ngx_chain_t out; - ngx_open_file_info_t of; + ngx_http_static_ctx_t *ctx; ngx_http_core_loc_conf_t *clcf; if (!(r->method & (NGX_HTTP_GET|NGX_HTTP_HEAD|NGX_HTTP_POST))) { @@ -67,42 +75,94 @@ ngx_http_static_handler(ngx_http_request return NGX_DECLINED; } - log = r->connection->log; + ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_static_ctx_t)); + if (ctx == NULL) { + return NGX_ERROR; + } + + ngx_http_set_ctx(r, ctx, ngx_http_static_module); /* * ngx_http_map_uri_to_path() allocates memory for terminating '\0' * so we do not need to reserve memory for '/' for possible redirect */ - last = ngx_http_map_uri_to_path(r, &path, &root, 0); - if (last == NULL) { + ctx->last = ngx_http_map_uri_to_path(r, &ctx->path, &root, 0); + if (ctx->last == NULL) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } - path.len = last - path.data; + ctx->path.len = ctx->last - ctx->path.data; - ngx_log_debug1(NGX_LOG_DEBUG_HTTP, log, 0, - "http filename: \"%s\"", path.data); + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http filename: \"%s\"", ctx->path.data); clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); - ngx_memzero(&of, sizeof(ngx_open_file_info_t)); + ctx->of.read_ahead = clcf->read_ahead; + ctx->of.directio = clcf->directio; + ctx->of.valid = clcf->open_file_cache_valid; + ctx->of.min_uses = clcf->open_file_cache_min_uses; + ctx->of.errors = clcf->open_file_cache_errors; + ctx->of.events = clcf->open_file_cache_events; - of.read_ahead = clcf->read_ahead; - of.directio = clcf->directio; - of.valid = clcf->open_file_cache_valid; - of.min_uses = clcf->open_file_cache_min_uses; - of.errors = clcf->open_file_cache_errors; - of.events = clcf->open_file_cache_events; +#if (NGX_THREADS) + if (clcf->aio == NGX_HTTP_AIO_THREADS && clcf->aio_open) { + ctx->of.thread_handler = ngx_http_static_thread_handler; + ctx->of.thread_ctx = r; + } +#endif - if (ngx_http_set_disable_symlinks(r, clcf, &path, &of) != NGX_OK) { + if (ngx_http_set_disable_symlinks(r, clcf, &ctx->path, &ctx->of) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } - if (ngx_open_cached_file(clcf->open_file_cache, &path, &of, r->pool) - != NGX_OK) - { - switch (of.err) { + rc = ngx_http_static_send(r); + +#if (NGX_THREADS) + if (rc == NGX_DONE) { + r->main->count++; + r->write_event_handler = ngx_http_static_write_event_handler; + } +#endif + + return rc; +} + + +static ngx_int_t +ngx_http_static_send(ngx_http_request_t *r) +{ + u_char *location; + size_t len; + ngx_log_t *log; + ngx_int_t rc; + ngx_uint_t level; + ngx_buf_t *b; + ngx_chain_t out; + ngx_http_static_ctx_t *ctx; + ngx_http_core_loc_conf_t *clcf; + + ctx = ngx_http_get_module_ctx(r, ngx_http_static_module); + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http static send: \"%s\"", ctx->path.data); + + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); + + log = r->connection->log; + + rc = ngx_open_cached_file(clcf->open_file_cache, &ctx->path, &ctx->of, + r->pool); + +#if (NGX_THREADS) + if (rc == NGX_AGAIN) { + return NGX_DONE; + } +#endif + + if (rc != NGX_OK) { + switch (ctx->of.err) { case 0: return NGX_HTTP_INTERNAL_SERVER_ERROR; @@ -133,8 +193,8 @@ ngx_http_static_handler(ngx_http_request } if (rc != NGX_HTTP_NOT_FOUND || clcf->log_not_found) { - ngx_log_error(level, log, of.err, - "%s \"%s\" failed", of.failed, path.data); + ngx_log_error(level, log, ctx->of.err, + "%s \"%s\" failed", ctx->of.failed, ctx->path.data); } return rc; @@ -142,9 +202,10 @@ ngx_http_static_handler(ngx_http_request r->root_tested = !r->error_page; - ngx_log_debug1(NGX_LOG_DEBUG_HTTP, log, 0, "http static fd: %d", of.fd); + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, log, 0, "http static fd: %d", + ctx->of.fd); - if (of.is_dir) { + if (ctx->of.is_dir) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, log, 0, "http dir"); @@ -158,9 +219,9 @@ ngx_http_static_handler(ngx_http_request len = r->uri.len + 1; if (!clcf->alias && clcf->root_lengths == NULL && r->args.len == 0) { - location = path.data + clcf->root.len; + location = ctx->path.data + clcf->root.len; - *last = '/'; + *ctx->last = '/'; } else { if (r->args.len) { @@ -173,13 +234,13 @@ ngx_http_static_handler(ngx_http_request return NGX_HTTP_INTERNAL_SERVER_ERROR; } - last = ngx_copy(location, r->uri.data, r->uri.len); + ctx->last = ngx_copy(location, r->uri.data, r->uri.len); - *last = '/'; + *ctx->last = '/'; if (r->args.len) { - *++last = '?'; - ngx_memcpy(++last, r->args.data, r->args.len); + *++ctx->last = '?'; + ngx_memcpy(++ctx->last, r->args.data, r->args.len); } } @@ -193,9 +254,9 @@ ngx_http_static_handler(ngx_http_request #if !(NGX_WIN32) /* the not regular files are probably Unix specific */ - if (!of.is_file) { + if (!ctx->of.is_file) { ngx_log_error(NGX_LOG_CRIT, log, 0, - "\"%s\" is not a regular file", path.data); + "\"%s\" is not a regular file", ctx->path.data); return NGX_HTTP_NOT_FOUND; } @@ -215,8 +276,8 @@ ngx_http_static_handler(ngx_http_request log->action = "sending response to client"; r->headers_out.status = NGX_HTTP_OK; - r->headers_out.content_length_n = of.size; - r->headers_out.last_modified_time = of.mtime; + r->headers_out.content_length_n = ctx->of.size; + r->headers_out.last_modified_time = ctx->of.mtime; if (ngx_http_set_etag(r) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; @@ -226,7 +287,7 @@ ngx_http_static_handler(ngx_http_request return NGX_HTTP_INTERNAL_SERVER_ERROR; } - if (r != r->main && of.size == 0) { + if (r != r->main && ctx->of.size == 0) { return ngx_http_send_header(r); } @@ -251,16 +312,16 @@ ngx_http_static_handler(ngx_http_request } b->file_pos = 0; - b->file_last = of.size; + b->file_last = ctx->of.size; b->in_file = b->file_last ? 1: 0; b->last_buf = (r == r->main) ? 1: 0; b->last_in_chain = 1; - b->file->fd = of.fd; - b->file->name = path; + b->file->fd = ctx->of.fd; + b->file->name = ctx->path; b->file->log = log; - b->file->directio = of.is_directio; + b->file->directio = ctx->of.is_directio; out.buf = b; out.next = NULL; @@ -269,6 +330,90 @@ ngx_http_static_handler(ngx_http_request } +#if (NGX_THREADS) + +static void +ngx_http_static_write_event_handler(ngx_http_request_t *r) +{ + ngx_int_t rc; + + rc = ngx_http_static_send(r); + + if (rc != NGX_DONE) { + ngx_http_finalize_request(r, rc); + } +} + + +static ngx_int_t +ngx_http_static_thread_handler(ngx_thread_task_t *task, ngx_file_t *file) +{ + ngx_str_t name; + ngx_thread_pool_t *tp; + ngx_http_request_t *r; + ngx_http_core_loc_conf_t *clcf; + + r = file->thread_ctx; + + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); + + tp = clcf->thread_pool; + + if (tp == NULL) { + if (ngx_http_complex_value(r, clcf->thread_pool_value, &name) + != NGX_OK) + { + return NGX_ERROR; + } + + tp = ngx_thread_pool_get((ngx_cycle_t *) ngx_cycle, &name); + + if (tp == NULL) { + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "thread pool \"%V\" not found", &name); + return NGX_ERROR; + } + } + + task->event.data = r; + task->event.handler = ngx_http_static_thread_event_handler; + + if (ngx_thread_task_post(tp, task) != NGX_OK) { + return NGX_ERROR; + } + + r->main->blocked++; + r->aio = 1; + + return NGX_OK; +} + + +static void +ngx_http_static_thread_event_handler(ngx_event_t *ev) +{ + ngx_connection_t *c; + ngx_http_request_t *r; + + r = ev->data; + c = r->connection; + + ngx_http_set_log_request(c->log, r); + + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, c->log, 0, + "http static thread: \"%V?%V\"", &r->uri, &r->args); + + r->main->blocked--; + r->aio = 0; + + r->write_event_handler(r); + + ngx_http_run_posted_requests(c); +} + +#endif + + static ngx_int_t ngx_http_static_init(ngx_conf_t *cf) { -------------- next part -------------- # HG changeset patch # User Roman Arutyunyan # Date 1538403511 -10800 # Mon Oct 01 17:18:31 2018 +0300 # Node ID 8f502ffdfbc032b30f45e6d4271b0822b01c2558 # Parent 68e2acd8c4cf81cdbe57c0a83bbafe43072cb9b5 Threaded open support in http file cache. diff --git a/src/http/ngx_http_cache.h b/src/http/ngx_http_cache.h --- a/src/http/ngx_http_cache.h +++ b/src/http/ngx_http_cache.h @@ -69,6 +69,7 @@ struct ngx_http_cache_s { u_char key[NGX_HTTP_CACHE_KEY_LEN]; u_char main[NGX_HTTP_CACHE_KEY_LEN]; + ngx_open_file_info_t of; ngx_file_uniq_t uniq; time_t valid_sec; time_t updating_sec; diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c --- a/src/http/ngx_http_file_cache.c +++ b/src/http/ngx_http_file_cache.c @@ -18,6 +18,8 @@ static void ngx_http_file_cache_lock_wai ngx_http_cache_t *c); static ngx_int_t ngx_http_file_cache_read(ngx_http_request_t *r, ngx_http_cache_t *c); +static ngx_int_t ngx_http_file_cache_aio_open(ngx_http_request_t *r, + ngx_http_cache_t *c); static ssize_t ngx_http_file_cache_aio_read(ngx_http_request_t *r, ngx_http_cache_t *c); #if (NGX_HAVE_FILE_AIO) @@ -264,13 +266,11 @@ ngx_http_file_cache_create_key(ngx_http_ ngx_int_t ngx_http_file_cache_open(ngx_http_request_t *r) { - ngx_int_t rc, rv; - ngx_uint_t test; - ngx_http_cache_t *c; - ngx_pool_cleanup_t *cln; - ngx_open_file_info_t of; - ngx_http_file_cache_t *cache; - ngx_http_core_loc_conf_t *clcf; + ngx_int_t rc; + ngx_uint_t test; + ngx_http_cache_t *c; + ngx_pool_cleanup_t *cln; + ngx_http_file_cache_t *cache; c = r->cache; @@ -315,7 +315,6 @@ ngx_http_file_cache_open(ngx_http_reques c->temp_file = 1; test = c->exists ? 1 : 0; - rv = NGX_DECLINED; } else { /* rc == NGX_DECLINED */ @@ -327,11 +326,10 @@ ngx_http_file_cache_open(ngx_http_reques return NGX_HTTP_CACHE_SCARCE; } - rv = NGX_HTTP_CACHE_SCARCE; + c->temp_file = 0; } else { c->temp_file = 1; - rv = NGX_DECLINED; } } @@ -340,62 +338,10 @@ ngx_http_file_cache_open(ngx_http_reques } if (!test) { - goto done; - } - - clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); - - ngx_memzero(&of, sizeof(ngx_open_file_info_t)); - - of.uniq = c->uniq; - of.valid = clcf->open_file_cache_valid; - of.min_uses = clcf->open_file_cache_min_uses; - of.events = clcf->open_file_cache_events; - of.directio = NGX_OPEN_FILE_DIRECTIO_OFF; - of.read_ahead = clcf->read_ahead; - - if (ngx_open_cached_file(clcf->open_file_cache, &c->file.name, &of, r->pool) - != NGX_OK) - { - switch (of.err) { - - case 0: - return NGX_ERROR; - - case NGX_ENOENT: - case NGX_ENOTDIR: - goto done; - - default: - ngx_log_error(NGX_LOG_CRIT, r->connection->log, of.err, - ngx_open_file_n " \"%s\" failed", c->file.name.data); - return NGX_ERROR; - } - } - - ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "http file cache fd: %d", of.fd); - - c->file.fd = of.fd; - c->file.log = r->connection->log; - c->uniq = of.uniq; - c->length = of.size; - c->fs_size = (of.fs_size + cache->bsize - 1) / cache->bsize; - - c->buf = ngx_create_temp_buf(r->pool, c->body_start); - if (c->buf == NULL) { - return NGX_ERROR; - } - - return ngx_http_file_cache_read(r, c); - -done: - - if (rv == NGX_DECLINED) { return ngx_http_file_cache_lock(r, c); } - return rv; + return ngx_http_file_cache_read(r, c); } @@ -405,6 +351,10 @@ ngx_http_file_cache_lock(ngx_http_reques ngx_msec_t now, timer; ngx_http_file_cache_t *cache; + if (!c->temp_file) { + return NGX_HTTP_CACHE_SCARCE; + } + if (!c->lock) { return NGX_DECLINED; } @@ -534,6 +484,12 @@ ngx_http_file_cache_read(ngx_http_reques ngx_http_file_cache_t *cache; ngx_http_file_cache_header_t *h; + rc = ngx_http_file_cache_aio_open(r, c); + + if (rc != NGX_OK) { + return rc; + } + n = ngx_http_file_cache_aio_read(r, c); if (n < 0) { @@ -663,6 +619,88 @@ ngx_http_file_cache_read(ngx_http_reques } +static ngx_int_t +ngx_http_file_cache_aio_open(ngx_http_request_t *r, ngx_http_cache_t *c) +{ + ngx_int_t rc; + ngx_http_file_cache_t *cache; + ngx_http_core_loc_conf_t *clcf; + + if (c->file.fd != NGX_INVALID_FILE) { + return NGX_OK; + } + + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); + + if (!c->reading) { + ngx_memzero(&c->of, sizeof(ngx_open_file_info_t)); + + c->of.uniq = c->uniq; + c->of.valid = clcf->open_file_cache_valid; + c->of.min_uses = clcf->open_file_cache_min_uses; + c->of.events = clcf->open_file_cache_events; + c->of.directio = NGX_OPEN_FILE_DIRECTIO_OFF; + c->of.read_ahead = clcf->read_ahead; + +#if (NGX_THREADS) + if (clcf->aio == NGX_HTTP_AIO_THREADS && clcf->aio_open) { + c->of.thread_task = c->thread_task; + c->of.thread_handler = ngx_http_cache_thread_handler; + c->of.thread_ctx = r; + } +#endif + } + + rc = ngx_open_cached_file(clcf->open_file_cache, &c->file.name, &c->of, + r->pool); + +#if (NGX_THREADS) + + if (rc == NGX_AGAIN) { + c->reading = 1; + return NGX_AGAIN; + } + + c->reading = 0; + +#endif + + if (rc != NGX_OK) { + switch (c->of.err) { + + case NGX_OK: + return NGX_ERROR; + + case NGX_ENOENT: + case NGX_ENOTDIR: + return ngx_http_file_cache_lock(r, c); + + default: + ngx_log_error(NGX_LOG_CRIT, r->connection->log, c->of.err, + ngx_open_file_n " \"%s\" failed", c->file.name.data); + return NGX_ERROR; + } + } + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http file cache fd: %d", c->of.fd); + + cache = c->file_cache; + c->file.fd = c->of.fd; + c->file.log = r->connection->log; + c->uniq = c->of.uniq; + c->length = c->of.size; + c->fs_size = (c->of.fs_size + cache->bsize - 1) / cache->bsize; + + c->buf = ngx_create_temp_buf(r->pool, c->body_start); + if (c->buf == NULL) { + return NGX_ERROR; + } + + return NGX_OK; +} + + static ssize_t ngx_http_file_cache_aio_read(ngx_http_request_t *r, ngx_http_cache_t *c) { @@ -1229,6 +1267,7 @@ ngx_http_file_cache_reopen(ngx_http_requ ngx_shmtx_unlock(&cache->shpool->mutex); c->secondary = 1; + c->file.fd = NGX_INVALID_FILE; c->file.name.len = 0; c->body_start = c->buf->end - c->buf->start; From xeioex at nginx.com Thu Oct 4 17:55:05 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 04 Oct 2018 17:55:05 +0000 Subject: [njs] Stream: fixed counting pending events. Message-ID: details: http://hg.nginx.org/njs/rev/f72e19b699e3 branches: changeset: 621:f72e19b699e3 user: Dmitry Volyntsev date: Thu Oct 04 20:45:40 2018 +0300 description: Stream: fixed counting pending events. Previously, erroneous messages 'pending events' were reported to error log. diffstat: nginx/ngx_stream_js_module.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diffs (20 lines): diff -r 2383666ab775 -r f72e19b699e3 nginx/ngx_stream_js_module.c --- a/nginx/ngx_stream_js_module.c Thu Oct 04 20:43:25 2018 +0300 +++ b/nginx/ngx_stream_js_module.c Thu Oct 04 20:45:40 2018 +0300 @@ -749,6 +749,16 @@ ngx_stream_js_cleanup_ctx(void *data) { ngx_stream_js_ctx_t *ctx = data; + if (ctx->upload_event != NULL) { + njs_vm_del_event(ctx->vm, ctx->upload_event); + ctx->upload_event = NULL; + } + + if (ctx->download_event != NULL) { + njs_vm_del_event(ctx->vm, ctx->download_event); + ctx->download_event = NULL; + } + if (njs_vm_pending(ctx->vm)) { ngx_log_error(NGX_LOG_ERR, ctx->log, 0, "pending events"); } From xeioex at nginx.com Thu Oct 4 17:55:05 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 04 Oct 2018 17:55:05 +0000 Subject: [njs] Stream: fixed s.off(). Message-ID: details: http://hg.nginx.org/njs/rev/2383666ab775 branches: changeset: 620:2383666ab775 user: Dmitry Volyntsev date: Thu Oct 04 20:43:25 2018 +0300 description: Stream: fixed s.off(). Previously, s.off() did not delete the event in njs vm using njs_vm_del_event(). diffstat: nginx/ngx_stream_js_module.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diffs (12 lines): diff -r b769c1037a57 -r 2383666ab775 nginx/ngx_stream_js_module.c --- a/nginx/ngx_stream_js_module.c Wed Oct 03 15:38:19 2018 +0300 +++ b/nginx/ngx_stream_js_module.c Thu Oct 04 20:43:25 2018 +0300 @@ -1081,6 +1081,8 @@ ngx_stream_js_ext_off(njs_vm_t *vm, njs_ return NJS_ERROR; } + njs_vm_del_event(vm, *event); + *event = NULL; return NJS_OK; From logan at nginx.lp-programming.com Mon Oct 8 02:00:14 2018 From: logan at nginx.lp-programming.com (Logan Perkins) Date: Sun, 7 Oct 2018 19:00:14 -0700 Subject: Adding support for transparent reverse proxies Message-ID: So, I'm looking to add support to nginx for transparent reverse proxies via UNIX sockets, and am hoping for some advice on how to do it with minimal impact on the rest of the server (both performance, and code complexity).? For clarity, a transparent reverse proxy takes an incoming (TCP) connection and copies the file descriptor to an 'upstream' backend, via a UNIX socket, with the sendmsg system call.? The determination of which upstream server should receive the file descriptor is made via calling recv on the connection with the MSG_PEEK flag, and then inspecting either the HTTP header, or the TLS header to determine the intended hostname.? The upstream backend catches the file descriptor via recvmsg, and then proceeds as though it were directly listening on the TCP interface. Transparent proxies have a number of performance and convenience advantages over the conventional HTTP proxy protocol.? On the performance front, once the reverse proxy passes the file descriptor to the upstream backend, it can immediately close its copy of the file handle.? This accomplishes several things.? First, there is no need to tell the upstream host anything about the file handle (like peer address or port), as upstream can use the usual socket functions to obtain that information, or set any socket options needed.? Second, there is no need to maintain a connection between the proxy and upstream, so the number of open file handles per connection is reduced by 1.? Third, there is no need to copy or buffer data between the TCP connection and the upstream server, which means large files can be sent directly via the sendfile system call. On the complexity front, beyond adding one indirection step in obtaining the file descriptor, the upstream backend has no increased complexity relative to running directly on the TCP port.? There is no need to read the peer address from the proxy.? Socket options can be set directly on the file descriptor, rather than relaying requests to set them through the proxy.? Calling shutdown on the file descriptor will shutdown the connection to the client machine, no delay, no ability for the proxy to fail to flush the pending data or close the connection.? Additionally, the reverse proxy is greatly simplified, taking less than 100 LoC, most of which are simply to parse the TLS headers. Anyway, after looking at the development_guide for nginx, and poking through the source code, I see a couple of possible ways to implement this.? The simplest way is to only accept a single file descriptor per incoming connection on the UNIX socket, shutting down the incoming connection after receiving one file descriptor.? This can be done inside ngx_event_accept, by retrieving the file descriptor, shutting down the accepted connection (c->fd), and setting c->fd to the new file descriptor.? Unfortunately, this requires the reverse proxy to be reliably fast at passing the incoming file descriptor, but is trivially simple for testing.? I think a better solution would be a new ls->handler, which runs when the socket is ready for reading, and then runs the existing ls->handler (ngx_http_init_connection) once the file descriptor is fetched and the existing socket is closed.? The more intrusive, but technically superior way to implement running behind a transparent reverse proxy, is to reuse the socket connection from the reverse proxy for an unlimited number of file descriptors.? This would involve a new handler for ngx_connection_t, which adds the connection to the list of connections watched by poll, kqueue or similar.? When the connection is ready for reading, recvmsg is used to fetch the file descriptor(s), which are then initialized like a normal http connection.? The UNIX socket connection would then persist either until the peer disconnects, or nginx shuts down. I have implemented the first method, as a proof of concept, but I have several questions before trying to implement the second method.? First, should this use a new nginx listener, rather than simply a setting on the existing listener?? I suspect the answer is 'yes', since I think it needs its own handler, which either runs before or instead of ngx_http_init_connection. Second, is there any reason the UNIX socket connections can't get put in the same pool as the TCP connections?? Third, should this be implemented in its own file, similarly to how http/2 is separated out? Fourth, what should the config file syntax be?? In my proof of concept version, I just added a flag after the 'listen unix:path', but I could see an advantage to defining the expected file descriptors separate from the UNIX socket. From paul at mrarm.io Fri Oct 12 19:54:04 2018 From: paul at mrarm.io (Paul Pawlowski) Date: Fri, 12 Oct 2018 21:54:04 +0200 Subject: [PATCH] Add client_body_temp_access configuration directive Message-ID: <67cc676dbfaf56332bb6.1539374044@desktop> # HG changeset patch # User Paul Pawlowski # Date 1539371172 -7200 # Fri Oct 12 21:06:12 2018 +0200 # Node ID 67cc676dbfaf56332bb6c61e635c0073c1e49907 # Parent 8b68d50090e4f134a35da60146fefd5e63770759 Add client_body_temp_access configuration directive Adds client_body_temp_access configuration directive, which sets the unix permissions of the temporary files holding client request bodies. This makes it possible for a process running as another user to access the files stored using client_body_temp_path and client_body_in_file_only. This is useful when using the mentioned directives in order to have nginx store the request body to file and forward the file path to another server running on the same machine as another user. diff -r 8b68d50090e4 -r 67cc676dbfaf contrib/vim/syntax/nginx.vim --- a/contrib/vim/syntax/nginx.vim Wed Oct 03 14:08:51 2018 +0300 +++ b/contrib/vim/syntax/nginx.vim Fri Oct 12 21:06:12 2018 +0200 @@ -155,6 +155,7 @@ syn keyword ngxDirective contained chunked_transfer_encoding syn keyword ngxDirective contained client_body_buffer_size syn keyword ngxDirective contained client_body_in_file_only +syn keyword ngxDirective contained client_body_temp_access syn keyword ngxDirective contained client_body_in_single_buffer syn keyword ngxDirective contained client_body_temp_path syn keyword ngxDirective contained client_body_timeout diff -r 8b68d50090e4 -r 67cc676dbfaf src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_core_module.c Fri Oct 12 21:06:12 2018 +0200 @@ -370,6 +370,13 @@ offsetof(ngx_http_core_loc_conf_t, client_body_temp_path), NULL }, + { ngx_string("client_body_temp_access"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE123, + ngx_conf_set_access_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_core_loc_conf_t, client_body_temp_access), + NULL }, + { ngx_string("client_body_in_file_only"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_enum_slot, @@ -3373,6 +3380,7 @@ clcf->if_modified_since = NGX_CONF_UNSET_UINT; clcf->max_ranges = NGX_CONF_UNSET_UINT; clcf->client_body_in_file_only = NGX_CONF_UNSET_UINT; + clcf->client_body_temp_access = NGX_CONF_UNSET_UINT; clcf->client_body_in_single_buffer = NGX_CONF_UNSET; clcf->internal = NGX_CONF_UNSET; clcf->sendfile = NGX_CONF_UNSET; @@ -3594,6 +3602,8 @@ ngx_conf_merge_uint_value(conf->client_body_in_file_only, prev->client_body_in_file_only, NGX_HTTP_REQUEST_BODY_FILE_OFF); + ngx_conf_merge_uint_value(conf->client_body_temp_access, + prev->client_body_temp_access, 0); ngx_conf_merge_value(conf->client_body_in_single_buffer, prev->client_body_in_single_buffer, 0); ngx_conf_merge_value(conf->internal, prev->internal, 0); diff -r 8b68d50090e4 -r 67cc676dbfaf src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_core_module.h Fri Oct 12 21:06:12 2018 +0200 @@ -375,6 +375,7 @@ ngx_uint_t if_modified_since; /* if_modified_since */ ngx_uint_t max_ranges; /* max_ranges */ ngx_uint_t client_body_in_file_only; /* client_body_in_file_only */ + ngx_uint_t client_body_temp_access; /* client_body_temp_access */ ngx_flag_t client_body_in_single_buffer; /* client_body_in_singe_buffer */ diff -r 8b68d50090e4 -r 67cc676dbfaf src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_request_body.c Fri Oct 12 21:06:12 2018 +0200 @@ -450,6 +450,7 @@ tf->file.fd = NGX_INVALID_FILE; tf->file.log = r->connection->log; tf->path = clcf->client_body_temp_path; + tf->access = clcf->client_body_temp_access; tf->pool = r->pool; tf->warn = "a client request body is buffered to a temporary file"; tf->log_level = r->request_body_file_log_level; From mdounin at mdounin.ru Mon Oct 15 14:27:02 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 15 Oct 2018 17:27:02 +0300 Subject: [PATCH] Add client_body_temp_access configuration directive In-Reply-To: <67cc676dbfaf56332bb6.1539374044@desktop> References: <67cc676dbfaf56332bb6.1539374044@desktop> Message-ID: <20181015142702.GC56558@mdounin.ru> Hello! On Fri, Oct 12, 2018 at 09:54:04PM +0200, Paul Pawlowski via nginx-devel wrote: > # HG changeset patch > # User Paul Pawlowski > # Date 1539371172 -7200 > # Fri Oct 12 21:06:12 2018 +0200 > # Node ID 67cc676dbfaf56332bb6c61e635c0073c1e49907 > # Parent 8b68d50090e4f134a35da60146fefd5e63770759 > Add client_body_temp_access configuration directive Style: there should be a trailing dot and a reference to ticket you've created, and please use past tense. Also, I think that "client_body_access" might be a better name: Added client_body_access directive (ticket #1653). > Adds client_body_temp_access configuration directive, which sets the unix permissions of the temporary files holding client request bodies. > > This makes it possible for a process running as another user to access the files stored using client_body_temp_path and client_body_in_file_only. > This is useful when using the mentioned directives in order to have nginx store the request body to file and forward the file path to another server running on the same machine as another user. Style: please wrap lines at 80 characters or less. > > diff -r 8b68d50090e4 -r 67cc676dbfaf contrib/vim/syntax/nginx.vim > --- a/contrib/vim/syntax/nginx.vim Wed Oct 03 14:08:51 2018 +0300 > +++ b/contrib/vim/syntax/nginx.vim Fri Oct 12 21:06:12 2018 +0200 > @@ -155,6 +155,7 @@ > syn keyword ngxDirective contained chunked_transfer_encoding > syn keyword ngxDirective contained client_body_buffer_size > syn keyword ngxDirective contained client_body_in_file_only > +syn keyword ngxDirective contained client_body_temp_access > syn keyword ngxDirective contained client_body_in_single_buffer > syn keyword ngxDirective contained client_body_temp_path > syn keyword ngxDirective contained client_body_timeout This list is expected to be sorted alphabetically. > diff -r 8b68d50090e4 -r 67cc676dbfaf src/http/ngx_http_core_module.c > --- a/src/http/ngx_http_core_module.c Wed Oct 03 14:08:51 2018 +0300 > +++ b/src/http/ngx_http_core_module.c Fri Oct 12 21:06:12 2018 +0200 > @@ -370,6 +370,13 @@ > offsetof(ngx_http_core_loc_conf_t, client_body_temp_path), > NULL }, > > + { ngx_string("client_body_temp_access"), > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE123, > + ngx_conf_set_access_slot, > + NGX_HTTP_LOC_CONF_OFFSET, > + offsetof(ngx_http_core_loc_conf_t, client_body_temp_access), > + NULL }, > + > { ngx_string("client_body_in_file_only"), > NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, > ngx_conf_set_enum_slot, > @@ -3373,6 +3380,7 @@ > clcf->if_modified_since = NGX_CONF_UNSET_UINT; > clcf->max_ranges = NGX_CONF_UNSET_UINT; > clcf->client_body_in_file_only = NGX_CONF_UNSET_UINT; > + clcf->client_body_temp_access = NGX_CONF_UNSET_UINT; > clcf->client_body_in_single_buffer = NGX_CONF_UNSET; > clcf->internal = NGX_CONF_UNSET; > clcf->sendfile = NGX_CONF_UNSET; > @@ -3594,6 +3602,8 @@ > ngx_conf_merge_uint_value(conf->client_body_in_file_only, > prev->client_body_in_file_only, > NGX_HTTP_REQUEST_BODY_FILE_OFF); > + ngx_conf_merge_uint_value(conf->client_body_temp_access, > + prev->client_body_temp_access, 0); Explicitly using 0600 as the default should be a better option. > ngx_conf_merge_value(conf->client_body_in_single_buffer, > prev->client_body_in_single_buffer, 0); > ngx_conf_merge_value(conf->internal, prev->internal, 0); > diff -r 8b68d50090e4 -r 67cc676dbfaf src/http/ngx_http_core_module.h > --- a/src/http/ngx_http_core_module.h Wed Oct 03 14:08:51 2018 +0300 > +++ b/src/http/ngx_http_core_module.h Fri Oct 12 21:06:12 2018 +0200 > @@ -375,6 +375,7 @@ > ngx_uint_t if_modified_since; /* if_modified_since */ > ngx_uint_t max_ranges; /* max_ranges */ > ngx_uint_t client_body_in_file_only; /* client_body_in_file_only */ > + ngx_uint_t client_body_temp_access; /* client_body_temp_access */ > > ngx_flag_t client_body_in_single_buffer; > /* client_body_in_singe_buffer */ > diff -r 8b68d50090e4 -r 67cc676dbfaf src/http/ngx_http_request_body.c > --- a/src/http/ngx_http_request_body.c Wed Oct 03 14:08:51 2018 +0300 > +++ b/src/http/ngx_http_request_body.c Fri Oct 12 21:06:12 2018 +0200 > @@ -450,6 +450,7 @@ > tf->file.fd = NGX_INVALID_FILE; > tf->file.log = r->connection->log; > tf->path = clcf->client_body_temp_path; > + tf->access = clcf->client_body_temp_access; > tf->pool = r->pool; > tf->warn = "a client request body is buffered to a temporary file"; > tf->log_level = r->request_body_file_log_level; The following block: if (r->request_body_file_group_access) { tf->access = 0660; } raises the question on how it is expected to interact with r->request_body_file_group_access. I suspect that r->request_body_file_group_access is not currently needed and can be removed altogether, but this needs an additional investigation. Another question to consider is how this is expected to interact with directories created for various temp file levels. These directories, much like the client_body_temp_path diretory itself, are unconditionally created with 0700 access, and will prevent access to temporary files from non-nginx user as well. These can be pre-created with appropriate permissions - but as long as it is a recommended approach, this probably needs to be mentioned in the commit log. -- Maxim Dounin http://mdounin.ru/ From paul at mrarm.io Mon Oct 15 18:14:56 2018 From: paul at mrarm.io (Paul) Date: Mon, 15 Oct 2018 20:14:56 +0200 Subject: [PATCH] Add client_body_temp_access configuration directive In-Reply-To: <20181015142702.GC56558@mdounin.ru> References: <67cc676dbfaf56332bb6.1539374044@desktop> <20181015142702.GC56558@mdounin.ru> Message-ID: <0045675415f152dfd405f6883205ab1c@mrarm.io> Hello and thank you for the code review, On 2018-10-15 16:27, Maxim Dounin wrote: > Hello! > > On Fri, Oct 12, 2018 at 09:54:04PM +0200, Paul Pawlowski via > nginx-devel wrote: > >> # HG changeset patch >> # User Paul Pawlowski >> # Date 1539371172 -7200 >> # Fri Oct 12 21:06:12 2018 +0200 >> # Node ID 67cc676dbfaf56332bb6c61e635c0073c1e49907 >> # Parent 8b68d50090e4f134a35da60146fefd5e63770759 >> Add client_body_temp_access configuration directive > > Style: there should be a trailing dot and a reference to ticket > you've created, and please use past tense. Also, I think that > "client_body_access" might be a better name: > > Added client_body_access directive (ticket #1653). > >> Adds client_body_temp_access configuration directive, which sets the >> unix permissions of the temporary files holding client request bodies. >> >> This makes it possible for a process running as another user to access >> the files stored using client_body_temp_path and >> client_body_in_file_only. >> This is useful when using the mentioned directives in order to have >> nginx store the request body to file and forward the file path to >> another server running on the same machine as another user. > > Style: please wrap lines at 80 characters or less. > I've addressed the style concerns, as well as changed the name to the suggested one. >> >> diff -r 8b68d50090e4 -r 67cc676dbfaf contrib/vim/syntax/nginx.vim >> --- a/contrib/vim/syntax/nginx.vim Wed Oct 03 14:08:51 2018 +0300 >> +++ b/contrib/vim/syntax/nginx.vim Fri Oct 12 21:06:12 2018 +0200 >> @@ -155,6 +155,7 @@ >> syn keyword ngxDirective contained chunked_transfer_encoding >> syn keyword ngxDirective contained client_body_buffer_size >> syn keyword ngxDirective contained client_body_in_file_only >> +syn keyword ngxDirective contained client_body_temp_access >> syn keyword ngxDirective contained client_body_in_single_buffer >> syn keyword ngxDirective contained client_body_temp_path >> syn keyword ngxDirective contained client_body_timeout > > This list is expected to be sorted alphabetically. > >> diff -r 8b68d50090e4 -r 67cc676dbfaf src/http/ngx_http_core_module.c >> --- a/src/http/ngx_http_core_module.c Wed Oct 03 14:08:51 2018 +0300 >> +++ b/src/http/ngx_http_core_module.c Fri Oct 12 21:06:12 2018 +0200 >> @@ -370,6 +370,13 @@ >> offsetof(ngx_http_core_loc_conf_t, client_body_temp_path), >> NULL }, >> >> + { ngx_string("client_body_temp_access"), >> + >> NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE123, >> + ngx_conf_set_access_slot, >> + NGX_HTTP_LOC_CONF_OFFSET, >> + offsetof(ngx_http_core_loc_conf_t, client_body_temp_access), >> + NULL }, >> + >> { ngx_string("client_body_in_file_only"), >> >> NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, >> ngx_conf_set_enum_slot, >> @@ -3373,6 +3380,7 @@ >> clcf->if_modified_since = NGX_CONF_UNSET_UINT; >> clcf->max_ranges = NGX_CONF_UNSET_UINT; >> clcf->client_body_in_file_only = NGX_CONF_UNSET_UINT; >> + clcf->client_body_temp_access = NGX_CONF_UNSET_UINT; >> clcf->client_body_in_single_buffer = NGX_CONF_UNSET; >> clcf->internal = NGX_CONF_UNSET; >> clcf->sendfile = NGX_CONF_UNSET; >> @@ -3594,6 +3602,8 @@ >> ngx_conf_merge_uint_value(conf->client_body_in_file_only, >> prev->client_body_in_file_only, >> NGX_HTTP_REQUEST_BODY_FILE_OFF); >> + ngx_conf_merge_uint_value(conf->client_body_temp_access, >> + prev->client_body_temp_access, 0); > > Explicitly using 0600 as the default should be a better option. I have changed the default value to 0600. > >> ngx_conf_merge_value(conf->client_body_in_single_buffer, >> prev->client_body_in_single_buffer, 0); >> ngx_conf_merge_value(conf->internal, prev->internal, 0); >> diff -r 8b68d50090e4 -r 67cc676dbfaf src/http/ngx_http_core_module.h >> --- a/src/http/ngx_http_core_module.h Wed Oct 03 14:08:51 2018 +0300 >> +++ b/src/http/ngx_http_core_module.h Fri Oct 12 21:06:12 2018 +0200 >> @@ -375,6 +375,7 @@ >> ngx_uint_t if_modified_since; /* if_modified_since */ >> ngx_uint_t max_ranges; /* max_ranges */ >> ngx_uint_t client_body_in_file_only; /* >> client_body_in_file_only */ >> + ngx_uint_t client_body_temp_access; /* client_body_temp_access >> */ >> >> ngx_flag_t client_body_in_single_buffer; >> /* >> client_body_in_singe_buffer */ >> diff -r 8b68d50090e4 -r 67cc676dbfaf src/http/ngx_http_request_body.c >> --- a/src/http/ngx_http_request_body.c Wed Oct 03 14:08:51 2018 +0300 >> +++ b/src/http/ngx_http_request_body.c Fri Oct 12 21:06:12 2018 +0200 >> @@ -450,6 +450,7 @@ >> tf->file.fd = NGX_INVALID_FILE; >> tf->file.log = r->connection->log; >> tf->path = clcf->client_body_temp_path; >> + tf->access = clcf->client_body_temp_access; >> tf->pool = r->pool; >> tf->warn = "a client request body is buffered to a temporary >> file"; >> tf->log_level = r->request_body_file_log_level; > > The following block: > > if (r->request_body_file_group_access) { > tf->access = 0660; > } > > raises the question on how it is expected to interact with > r->request_body_file_group_access. I suspect that > r->request_body_file_group_access is not currently needed and can > be removed altogether, but this needs an additional investigation. I've checked where is it used - the only place is the ngx_http_dav_module, where it is used in order to set the permissions for the temporary uploaded file in the PUT handler - however, as the file is later moved for storage using ngx_ext_rename_file function (in ngx_http_dav_put_handler) with permissions taken from the configuration file (with a default value of 0600) I do not believe that the temporary file permission matter at that stage. The usage by third party modules also seem limited: I've tried to do a GitHub search for it: https://github.com/search?q=request_body_file_group_access+-filename%3Angx_http_request_body.c+-filename%3Angx_http_request.h+-filename%3Angx_http_dav_module.c&type=Code which results in 1,123 code results. However most of these are simply slightly altered versions of ngx_http_request_body.c, and if we exclude the top three results we end up with 264 results: https://github.com/search?q=request_body_file_group_access+-filename%3Angx_http_request_body.c+-filename%3Angx_http_request.h+-filename%3Angx_http_dav_module.c+-filename%3Angx_http_lua_req_body.c+-filename%3Angx_http_waf_filter_module.c+-filename%3Angx_http_spdy.c&type=Code with most of them still largely being nginx codebase - however a few usages can still be found including: - https://github.com/oscar810429/graphicsmagick_nginx_module/blob/6f171579ba7638f62f885d3d8e0687d7977f2f33/ngx_http_graphicsmagick_module.c [I don't believe it is needed here - the file is not forwarded to other users I believe after taking a quick look at the code] - https://github.com/heapsource/BlackLinks/blob/8173ba44270d65cbe50aa2865eab91af3d44f3d4/nginx-hello/ngx_http_hello_world_module.c - https://github.com/alacner/nginx_lua_module/blob/e194c26f96e50504540fd5830283e1c01670d170/src/ngx_http_lua_module.c - https://github.com/AntonRiab/ngx_pgcopy/blob/ca32432024d0677754faddbe4d8b45c5d896c79b/ngx_http_pgcopy_module.c Even though I believe they should be replaceable, is this something I should look into more? I'm making the deletion of the request_body_file_group_access a separate commit, as this touches code not directly related to the core added functionality. > > Another question to consider is how this is expected to interact > with directories created for various temp file levels. These > directories, much like the client_body_temp_path diretory itself, > are unconditionally created with 0700 access, and will prevent > access to temporary files from non-nginx user as well. These can > be pre-created with appropriate permissions - but as long as it is > a recommended approach, this probably needs to be mentioned in the > commit log. This is an issue I have completely missed while writing this patch. I have checked how do the other nginx module solve this particular issue and figured out that probably the best option would be to use the specified file permissions as the ones to use (using the ngx_dir_access helper function), and this is what I've implemented. I'm unfortunately unsure what is the proper way to E-Mail the patches in a response mail so I'm simply attaching them to the E-Mail below (there are two separate commits below): # HG changeset patch # User Paul Pawlowski # Date 1539621300 -7200 # Mon Oct 15 18:35:00 2018 +0200 # Node ID c94787463c982dab370b695e2f3ddcbc7655ead5 # Parent 8b68d50090e4f134a35da60146fefd5e63770759 Added client_body_access configuration directive (ticket #1651). Added client_body_access configuration directive, which sets the unix permissions of the temporary files and directories holding client request bodies. This makes it possible for a process running as another user to access the files stored using client_body_temp_path and client_body_in_file_only. This is useful when using the mentioned directives in order to have nginx store the request body to file and forward the file path to another server running on the same machine as another user. diff -r 8b68d50090e4 -r c94787463c98 contrib/vim/syntax/nginx.vim --- a/contrib/vim/syntax/nginx.vim Wed Oct 03 14:08:51 2018 +0300 +++ b/contrib/vim/syntax/nginx.vim Mon Oct 15 18:35:00 2018 +0200 @@ -153,6 +153,7 @@ syn keyword ngxDirective contained charset syn keyword ngxDirective contained charset_types syn keyword ngxDirective contained chunked_transfer_encoding +syn keyword ngxDirective contained client_body_access syn keyword ngxDirective contained client_body_buffer_size syn keyword ngxDirective contained client_body_in_file_only syn keyword ngxDirective contained client_body_in_single_buffer diff -r 8b68d50090e4 -r c94787463c98 src/core/ngx_file.c --- a/src/core/ngx_file.c Wed Oct 03 14:08:51 2018 +0300 +++ b/src/core/ngx_file.c Mon Oct 15 18:35:00 2018 +0200 @@ -146,10 +146,17 @@ uint32_t n; ngx_err_t err; ngx_str_t name; + ngx_uint_t path_access; ngx_uint_t prefix; ngx_pool_cleanup_t *cln; ngx_pool_cleanup_file_t *clnf; + if (access != 0) { + path_access = ngx_dir_access(access); + } else { + path_access = 0700; + } + if (file->name.len) { name = file->name; levels = 0; @@ -230,7 +237,7 @@ return NGX_ERROR; } - if (ngx_create_path(file, path) == NGX_ERROR) { + if (ngx_create_path(file, path, path_access) == NGX_ERROR) { return NGX_ERROR; } } @@ -263,7 +270,7 @@ ngx_int_t -ngx_create_path(ngx_file_t *file, ngx_path_t *path) +ngx_create_path(ngx_file_t *file, ngx_path_t *path, ngx_uint_t access) { size_t pos; ngx_err_t err; @@ -283,7 +290,7 @@ ngx_log_debug1(NGX_LOG_DEBUG_CORE, file->log, 0, "temp file: \"%s\"", file->name.data); - if (ngx_create_dir(file->name.data, 0700) == NGX_FILE_ERROR) { + if (ngx_create_dir(file->name.data, access) == NGX_FILE_ERROR) { err = ngx_errno; if (err != NGX_EEXIST) { ngx_log_error(NGX_LOG_CRIT, file->log, err, diff -r 8b68d50090e4 -r c94787463c98 src/core/ngx_file.h --- a/src/core/ngx_file.h Wed Oct 03 14:08:51 2018 +0300 +++ b/src/core/ngx_file.h Mon Oct 15 18:35:00 2018 +0200 @@ -140,7 +140,8 @@ ngx_pool_t *pool, ngx_uint_t persistent, ngx_uint_t clean, ngx_uint_t access); void ngx_create_hashed_filename(ngx_path_t *path, u_char *file, size_t len); -ngx_int_t ngx_create_path(ngx_file_t *file, ngx_path_t *path); +ngx_int_t ngx_create_path(ngx_file_t *file, ngx_path_t *path, + ngx_uint_t access); ngx_err_t ngx_create_full_path(u_char *dir, ngx_uint_t access); ngx_int_t ngx_add_path(ngx_conf_t *cf, ngx_path_t **slot); ngx_int_t ngx_create_paths(ngx_cycle_t *cycle, ngx_uid_t user); diff -r 8b68d50090e4 -r c94787463c98 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_core_module.c Mon Oct 15 18:35:00 2018 +0200 @@ -370,6 +370,13 @@ offsetof(ngx_http_core_loc_conf_t, client_body_temp_path), NULL }, + { ngx_string("client_body_access"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE123, + ngx_conf_set_access_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_core_loc_conf_t, client_body_access), + NULL }, + { ngx_string("client_body_in_file_only"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_conf_set_enum_slot, @@ -3373,6 +3380,7 @@ clcf->if_modified_since = NGX_CONF_UNSET_UINT; clcf->max_ranges = NGX_CONF_UNSET_UINT; clcf->client_body_in_file_only = NGX_CONF_UNSET_UINT; + clcf->client_body_access = NGX_CONF_UNSET_UINT; clcf->client_body_in_single_buffer = NGX_CONF_UNSET; clcf->internal = NGX_CONF_UNSET; clcf->sendfile = NGX_CONF_UNSET; @@ -3594,6 +3602,8 @@ ngx_conf_merge_uint_value(conf->client_body_in_file_only, prev->client_body_in_file_only, NGX_HTTP_REQUEST_BODY_FILE_OFF); + ngx_conf_merge_uint_value(conf->client_body_access, + prev->client_body_access, 0600); ngx_conf_merge_value(conf->client_body_in_single_buffer, prev->client_body_in_single_buffer, 0); ngx_conf_merge_value(conf->internal, prev->internal, 0); diff -r 8b68d50090e4 -r c94787463c98 src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_core_module.h Mon Oct 15 18:35:00 2018 +0200 @@ -375,6 +375,7 @@ ngx_uint_t if_modified_since; /* if_modified_since */ ngx_uint_t max_ranges; /* max_ranges */ ngx_uint_t client_body_in_file_only; /* client_body_in_file_only */ + ngx_uint_t client_body_access; /* client_body_access */ ngx_flag_t client_body_in_single_buffer; /* client_body_in_singe_buffer */ diff -r 8b68d50090e4 -r c94787463c98 src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_request_body.c Mon Oct 15 18:35:00 2018 +0200 @@ -450,6 +450,7 @@ tf->file.fd = NGX_INVALID_FILE; tf->file.log = r->connection->log; tf->path = clcf->client_body_temp_path; + tf->access = clcf->client_body_access; tf->pool = r->pool; tf->warn = "a client request body is buffered to a temporary file"; tf->log_level = r->request_body_file_log_level; # HG changeset patch # User Paul Pawlowski # Date 1539623192 -7200 # Mon Oct 15 19:06:32 2018 +0200 # Node ID e6f5c3e20646abe8708db185e5acdd861c13ce35 # Parent c94787463c982dab370b695e2f3ddcbc7655ead5 Removed request_body_file_group_access from ngx_http_request_s. request_body_file_group_access's only usage is in the ngx_http_dav_module. However the temporary file created by that module after a successful upload is moved into another directory, and the file permissions are changed to user specified ones. Therefore, it doesn't matter what the initial file permissions were and the code can be simplified. diff -r c94787463c98 -r e6f5c3e20646 src/http/modules/ngx_http_dav_module.c --- a/src/http/modules/ngx_http_dav_module.c Mon Oct 15 18:35:00 2018 +0200 +++ b/src/http/modules/ngx_http_dav_module.c Mon Oct 15 19:06:32 2018 +0200 @@ -170,7 +170,6 @@ r->request_body_in_file_only = 1; r->request_body_in_persistent_file = 1; r->request_body_in_clean_file = 1; - r->request_body_file_group_access = 1; r->request_body_file_log_level = 0; rc = ngx_http_read_client_request_body(r, ngx_http_dav_put_handler); diff -r c94787463c98 -r e6f5c3e20646 src/http/ngx_http_request.h --- a/src/http/ngx_http_request.h Mon Oct 15 18:35:00 2018 +0200 +++ b/src/http/ngx_http_request.h Mon Oct 15 19:06:32 2018 +0200 @@ -482,7 +482,6 @@ unsigned request_body_in_file_only:1; unsigned request_body_in_persistent_file:1; unsigned request_body_in_clean_file:1; - unsigned request_body_file_group_access:1; unsigned request_body_file_log_level:3; unsigned request_body_no_buffering:1; diff -r c94787463c98 -r e6f5c3e20646 src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c Mon Oct 15 18:35:00 2018 +0200 +++ b/src/http/ngx_http_request_body.c Mon Oct 15 19:06:32 2018 +0200 @@ -457,10 +457,6 @@ tf->persistent = r->request_body_in_persistent_file; tf->clean = r->request_body_in_clean_file; - if (r->request_body_file_group_access) { - tf->access = 0660; - } - rb->temp_file = tf; if (rb->bufs == NULL) { From miranovy at gmail.com Wed Oct 17 10:49:13 2018 From: miranovy at gmail.com (Miroslav Novy) Date: Wed, 17 Oct 2018 12:49:13 +0200 Subject: limit_rate_after support variables In-Reply-To: References: <20180827112847.GE71000@lo0.su> <20180827160021.GG71000@lo0.su> <20180829114230.GC43480@lo0.su> Message-ID: Hello, I prepare patch on actual sources. Settings limit_rate and limit_rate_after works good. Please make code review, our testing and merge to main branche. Thank you Miroslav Nov? Example of configration: location / { root /var/www/default/; index index.html index.htm; set $my_limit_rate 4k; set $my_limit_rate_after 4m; limit_rate $my_limit_rate; limit_rate_after $my_limit_rate_after; access_by_lua_block { ngx.var.my_limit_rate = '2k' ngx.var.my_limit_rate_after = '10m' } } # HG changeset patch # User Miroslav Nov? # Date 1539773045 0 # Wed Oct 17 10:44:05 2018 +0000 # Node ID 0de0d409a946b9f33284c036fdf3dcdaec0853c2 # Parent 8b68d50090e4f134a35da60146fefd5e63770759 limit_rate and limit_rate_after support variables diff -r 8b68d50090e4 -r 0de0d409a946 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_core_module.c Wed Oct 17 10:44:05 2018 +0000 @@ -479,18 +479,18 @@ { ngx_string("limit_rate"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF |NGX_CONF_TAKE1, - ngx_conf_set_size_slot, + ngx_http_set_complex_value_slot, NGX_HTTP_LOC_CONF_OFFSET, offsetof(ngx_http_core_loc_conf_t, limit_rate), - NULL }, + &ngx_http_complex_value_size_p }, { ngx_string("limit_rate_after"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF |NGX_CONF_TAKE1, - ngx_conf_set_size_slot, + ngx_http_set_complex_value_slot, NGX_HTTP_LOC_CONF_OFFSET, offsetof(ngx_http_core_loc_conf_t, limit_rate_after), - NULL }, + &ngx_http_complex_value_size_p }, { ngx_string("keepalive_timeout"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE12, @@ -1281,10 +1281,6 @@ r->connection->tcp_nopush = NGX_TCP_NOPUSH_DISABLED; } - if (r->limit_rate == 0) { - r->limit_rate = clcf->limit_rate; - } - if (clcf->handler) { r->content_handler = clcf->handler; } @@ -3362,6 +3358,8 @@ * clcf->exact_match = 0; * clcf->auto_redirect = 0; * clcf->alias = 0; + * clcf->limit_rate = NULL; + * clcf->limit_rate_after = NULL; * clcf->gzip_proxied = 0; * clcf->keepalive_disable = 0; */ @@ -3392,8 +3390,6 @@ clcf->send_timeout = NGX_CONF_UNSET_MSEC; clcf->send_lowat = NGX_CONF_UNSET_SIZE; clcf->postpone_output = NGX_CONF_UNSET_SIZE; - clcf->limit_rate = NGX_CONF_UNSET_SIZE; - clcf->limit_rate_after = NGX_CONF_UNSET_SIZE; clcf->keepalive_timeout = NGX_CONF_UNSET_MSEC; clcf->keepalive_header = NGX_CONF_UNSET; clcf->keepalive_requests = NGX_CONF_UNSET_UINT; @@ -3581,6 +3577,14 @@ ngx_conf_merge_msec_value(conf->client_body_timeout, prev->client_body_timeout, 60000); + if (conf->limit_rate == NULL) { + conf->limit_rate = prev->limit_rate; + } + + if (conf->limit_rate_after == NULL) { + conf->limit_rate_after = prev->limit_rate_after; + } + ngx_conf_merge_bitmask_value(conf->keepalive_disable, prev->keepalive_disable, (NGX_CONF_BITMASK_SET @@ -3622,9 +3626,7 @@ ngx_conf_merge_size_value(conf->send_lowat, prev->send_lowat, 0); ngx_conf_merge_size_value(conf->postpone_output, prev->postpone_output, 1460); - ngx_conf_merge_size_value(conf->limit_rate, prev->limit_rate, 0); - ngx_conf_merge_size_value(conf->limit_rate_after, prev->limit_rate_after, - 0); + ngx_conf_merge_msec_value(conf->keepalive_timeout, prev->keepalive_timeout, 75000); ngx_conf_merge_sec_value(conf->keepalive_header, diff -r 8b68d50090e4 -r 0de0d409a946 src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_core_module.h Wed Oct 17 10:44:05 2018 +0000 @@ -350,8 +350,9 @@ size_t client_body_buffer_size; /* client_body_buffer_size */ size_t send_lowat; /* send_lowat */ size_t postpone_output; /* postpone_output */ - size_t limit_rate; /* limit_rate */ - size_t limit_rate_after; /* limit_rate_after */ + ngx_http_complex_value_t *limit_rate; /* limit_rate */ + ngx_http_complex_value_t *limit_rate_after; + /* limit_rate_after */ size_t sendfile_max_chunk; /* sendfile_max_chunk */ size_t read_ahead; /* read_ahead */ size_t subrequest_output_buffer_size; diff -r 8b68d50090e4 -r 0de0d409a946 src/http/ngx_http_script.c --- a/src/http/ngx_http_script.c Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_script.c Wed Oct 17 10:44:05 2018 +0000 @@ -9,7 +9,8 @@ #include #include - +static char *ngx_http_complex_value_set_size(ngx_conf_t *cf, void *post, + void *data); static ngx_int_t ngx_http_script_init_arrays(ngx_http_script_compile_t *sc); static ngx_int_t ngx_http_script_done(ngx_http_script_compile_t *sc); static ngx_int_t ngx_http_script_add_copy_code(ngx_http_script_compile_t *sc, @@ -31,6 +32,8 @@ static uintptr_t ngx_http_script_exit_code = (uintptr_t) NULL; +ngx_conf_post_handler_pt ngx_http_complex_value_size_p = + ngx_http_complex_value_set_size; void ngx_http_script_flush_complex_value(ngx_http_request_t *r, @@ -103,6 +106,23 @@ return NGX_OK; } +ngx_int_t +ngx_http_complex_value_size(ngx_http_request_t *r, + ngx_http_complex_value_t *val, ngx_str_t *value, ssize_t *size) +{ + if (val->lengths == NULL) { + *size = val->u.size; + return NGX_OK; + } + + if (ngx_http_complex_value(r, val, value) != NGX_OK) { + return NGX_ERROR; + } + + *size = ngx_parse_size(value); + + return NGX_OK; +} ngx_int_t ngx_http_compile_complex_value(ngx_http_compile_complex_value_t *ccv) @@ -214,6 +234,7 @@ char *p = conf; ngx_str_t *value; + ngx_conf_post_t *post; ngx_http_complex_value_t **cv; ngx_http_compile_complex_value_t ccv; @@ -240,6 +261,29 @@ return NGX_CONF_ERROR; } + if (cmd->post) { + post = cmd->post; + return post->post_handler(cf, post, *cv); + } + + return NGX_CONF_OK; +} + + +static char * +ngx_http_complex_value_set_size(ngx_conf_t *cf, void *post, void *data) +{ + ngx_http_complex_value_t *cv = data; + + if (cv->lengths) { + return NGX_CONF_OK; + } + + cv->u.size = ngx_parse_size(&cv->value); + if (cv->u.size == (size_t) NGX_ERROR) { + return "invalid value"; + } + return NGX_CONF_OK; } diff -r 8b68d50090e4 -r 0de0d409a946 src/http/ngx_http_script.h --- a/src/http/ngx_http_script.h Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_script.h Wed Oct 17 10:44:05 2018 +0000 @@ -68,6 +68,10 @@ ngx_uint_t *flushes; void *lengths; void *values; + + union { + size_t size; + } u; } ngx_http_complex_value_t; @@ -205,6 +209,8 @@ void ngx_http_script_flush_complex_value(ngx_http_request_t *r, ngx_http_complex_value_t *val); +ngx_int_t ngx_http_complex_value_size(ngx_http_request_t *r, + ngx_http_complex_value_t *val, ngx_str_t *value, ssize_t *size); ngx_int_t ngx_http_complex_value(ngx_http_request_t *r, ngx_http_complex_value_t *val, ngx_str_t *value); ngx_int_t ngx_http_compile_complex_value(ngx_http_compile_complex_value_t *ccv); @@ -253,5 +259,6 @@ void ngx_http_script_var_code(ngx_http_script_engine_t *e); void ngx_http_script_nop_code(ngx_http_script_engine_t *e); +extern ngx_conf_post_handler_pt ngx_http_complex_value_size_p; #endif /* _NGX_HTTP_SCRIPT_H_INCLUDED_ */ diff -r 8b68d50090e4 -r 0de0d409a946 src/http/ngx_http_write_filter_module.c --- a/src/http/ngx_http_write_filter_module.c Wed Oct 03 14:08:51 2018 +0300 +++ b/src/http/ngx_http_write_filter_module.c Wed Oct 17 10:44:05 2018 +0000 @@ -48,6 +48,8 @@ ngx_http_write_filter(ngx_http_request_t *r, ngx_chain_t *in) { off_t size, sent, nsent, limit; + ssize_t limit_rate, limit_rate_after; + ngx_str_t val; ngx_uint_t last, flush, sync; ngx_msec_t delay; ngx_chain_t *cl, *ln, **ll, *chain; @@ -218,9 +220,38 @@ return NGX_ERROR; } + if (r->limit_rate == 0 + && clcf->limit_rate + && ngx_http_complex_value_size(r, clcf->limit_rate, &val, + &limit_rate) + == NGX_OK) + { + if (limit_rate_after != NGX_ERROR) { + r->limit_rate = limit_rate; + + } else if (val.len) { + ngx_log_error(NGX_LOG_ERR, c->log, 0, + "invalid \"limit_rate\" value \"%V\"", + &val); + } + } + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0, "http limit rate \"%z\"", r->limit_rate); + if (r->limit_rate) { - if (r->limit_rate_after == 0) { - r->limit_rate_after = clcf->limit_rate_after; + if (r->limit_rate_after == 0 + && clcf->limit_rate_after + && ngx_http_complex_value_size(r, clcf->limit_rate_after, &val, + &limit_rate_after) + == NGX_OK) + { + if (limit_rate_after != NGX_ERROR) { + r->limit_rate_after = limit_rate_after; + } else if (val.len) { + ngx_log_error(NGX_LOG_ERR, c->log, 0, + "invalid \"limit_rate_after\" value \"%V\"", + &val); + } } limit = (off_t) r->limit_rate * (ngx_time() - r->start_sec + 1) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ourwaiting at gmail.com Fri Oct 19 03:33:57 2018 From: ourwaiting at gmail.com (HENG) Date: Fri, 19 Oct 2018 11:33:57 +0800 Subject: Nginx NJS fs.writeFileSync is atomic writing and race condition prevention ? Message-ID: Hello: I am new to Nginx NJS, and I want to write a website with NJS. I want to write a simple JSON database with NJS fs.writeFileSync ,just like Node.js LowDB. but I have no idea . Does NJS fs.writeFileSync is atomic writing and race condition prevention ? If NJS fs.writeFileSync is NOT atomic writing, it can NOT be treate as a normal database file write. Thank you ! -- -------------------------------------------------------------------- Heng --------------------------------------------------------------------- -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Fri Oct 19 11:27:27 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 19 Oct 2018 14:27:27 +0300 Subject: Nginx NJS fs.writeFileSync is atomic writing and race condition prevention ? In-Reply-To: References: Message-ID: On 19.10.2018 06:33, HENG wrote: > Hello: > > I am new to Nginx NJS, and I want to write a website with NJS. > > I want to write a simple JSON database with NJS fs.writeFileSync ,just > like Node.js LowDB. > > but I have no idea . Does?NJS fs.writeFileSync is atomic writing and > race condition prevention ? http://hg.nginx.org/njs/file/tip/njs/njs_fs.c#l649 It simply opens the file, makes several write syscalls into it, and closes the file. So, it is not atomic. Probably we need here something like https://www.npmjs.com/package/write-file-atomic > > If?NJS fs.writeFileSync is NOT atomic writing, it can NOT be treate as a > normal database file write. > > Thank you ! > > -- > -------------------------------------------------------------------- > Heng > --------------------------------------------------------------------- > -- > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > From ourwaiting at gmail.com Fri Oct 19 16:58:35 2018 From: ourwaiting at gmail.com (HENG) Date: Sat, 20 Oct 2018 00:58:35 +0800 Subject: Nginx NJS fs.writeFileSync is atomic writing and race condition prevention ? In-Reply-To: References: Message-ID: Thanks your reply. When I deep into NJS source you ponit to, I found something about GCC write file flag with O_APPEND I google Linux GCC write file O_APPEND to learn something. In fact NJS fs.writeFileSync() with O_APPEND, it is atomic write file if it's buffer just write all in one time. Even mutil process to write one file use O_APPEND. But, if write buffer can NOT write all in one time, in mutil process, it is NOT atomic. I found Nginx Log write is use O_APPEND to write atomic, even Nginx has a lot of workers. And I read the source code by your link `write-file-atomic`, it is hard to port to NJS. The reason is that NJS fs without fs.rename, and Nodejs write file atomic use temp file rename and write queue to make it. Thanks. HENG ?2018?10?20??? ??12:55??? > Thanks your reply. > > When I deep into NJS source you ponit to, I found something about GCC > write file flag with O_APPEND > > I google Linux GCC write file O_APPEND to learn something. > > In fact NJS fs.writeFileSync() with O_APPEND, it is atomic write file if > it's buffer just write all in one time. Even mutil process to write one > file use O_APPEND. > > But, if write buffer can NOT write all in one time, in mutil process, it > is NOT atomic. > > I found Nginx Log write is use O_APPEND to write atomic, even Nginx has a > lot of workers. > > And > > I read the source code by your link `write-file-atomic`, it is hard to > port to NJS. The reason is that NJS fs without fs.rename, and Nodejs write > file atomic use temp file rename and write queue to make it. > > Thanks. > > > -- > -------------------------------------------------------------------- > Heng > --------------------------------------------------------------------- > -- > -- -------------------------------------------------------------------- Heng --------------------------------------------------------------------- -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbart at nginx.com Fri Oct 19 17:03:32 2018 From: vbart at nginx.com (Valentin V. Bartenev) Date: Fri, 19 Oct 2018 20:03:32 +0300 Subject: Nginx NJS fs.writeFileSync is atomic writing and race condition prevention ? In-Reply-To: References: Message-ID: <2273312.NR0LRtDZKo@vbart-laptop> On Friday, 19 October 2018 06:33:57 MSK HENG wrote: > Hello: > > I am new to Nginx NJS, and I want to write a website with NJS. > > I want to write a simple JSON database with NJS fs.writeFileSync ,just like > Node.js LowDB. > > but I have no idea . Does NJS fs.writeFileSync is atomic writing and race > condition prevention ? > > If NJS fs.writeFileSync is NOT atomic writing, it can NOT be treate as a > normal database file write. > > Thank you ! > The fs.writeFileSync function calls a simple write(), which is atomic with regular files. But at the first glance to work with file like a database using multiple nginx worker processes it's not enough to have atomic writing function. You need to have atomic the whole sequence of operations: read file -> update data -> write file. Since currently there's no locking functionality in njs, the only way to achieve this is to use only one worker process. wbr, Valentin V. Bartenev From xeioex at nginx.com Fri Oct 19 18:30:53 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 19 Oct 2018 18:30:53 +0000 Subject: [njs] delete operator refactored. Message-ID: details: http://hg.nginx.org/njs/rev/da92ae8bcae3 branches: changeset: 624:da92ae8bcae3 user: Dmitry Volyntsev date: Fri Oct 19 20:54:59 2018 +0300 description: delete operator refactored. diffstat: njs/njs_json.c | 9 ++++++++- njs/njs_object.c | 4 ++-- njs/njs_vm.c | 23 +++++++++++++++++------ njs/test/njs_unit_test.c | 22 ++++++++++++++++++++++ 4 files changed, 49 insertions(+), 9 deletions(-) diffs (154 lines): diff -r 9d7b9cc03569 -r da92ae8bcae3 njs/njs_json.c --- a/njs/njs_json.c Fri Oct 19 20:52:57 2018 +0300 +++ b/njs/njs_json.c Fri Oct 19 20:54:59 2018 +0300 @@ -934,6 +934,12 @@ njs_json_parse_continuation(njs_vm_t *vm } prop = lhq.value; + + if (prop->type == NJS_WHITEOUT) { + state->index++; + break; + } + state->prop_value = &prop->value; if (njs_json_is_non_empty(&prop->value)) { @@ -1233,6 +1239,7 @@ njs_json_stringify_continuation(njs_vm_t if (!prop->enumerable || njs_is_void(&prop->value) + || !njs_is_valid(&prop->value) || njs_is_function(&prop->value)) { break; @@ -2382,7 +2389,7 @@ njs_vm_value_dump(njs_vm_t *vm, nxt_str_ prop = lhq.value; val = &prop->value; - if (!prop->enumerable) { + if (prop->type == NJS_WHITEOUT || !prop->enumerable) { break; } } diff -r 9d7b9cc03569 -r da92ae8bcae3 njs/njs_object.c --- a/njs/njs_object.c Fri Oct 19 20:52:57 2018 +0300 +++ b/njs/njs_object.c Fri Oct 19 20:54:59 2018 +0300 @@ -922,7 +922,7 @@ njs_object_keys_array(njs_vm_t *vm, cons break; } - if (prop->enumerable) { + if (prop->type != NJS_WHITEOUT && prop->enumerable) { keys_length++; } } @@ -954,7 +954,7 @@ njs_object_keys_array(njs_vm_t *vm, cons break; } - if (prop->enumerable) { + if (prop->type != NJS_WHITEOUT && prop->enumerable) { njs_string_copy(&keys->start[n++], &prop->name); } } diff -r 9d7b9cc03569 -r da92ae8bcae3 njs/njs_vm.c --- a/njs/njs_vm.c Fri Oct 19 20:52:57 2018 +0300 +++ b/njs/njs_vm.c Fri Oct 19 20:54:59 2018 +0300 @@ -552,6 +552,19 @@ njs_vmcode_property_set(njs_vm_t *vm, nj return NXT_ERROR; } + if (nxt_slow_path(pq.lhq.value != NULL)) { + prop = pq.lhq.value; + + if (nxt_slow_path(prop->type == NJS_WHITEOUT)) { + /* Previously deleted property. */ + prop->type = NJS_PROPERTY; + prop->enumerable = 1; + prop->configurable = 1; + prop->writable = 1; + break; + } + } + prop = njs_object_prop_alloc(vm, &pq.value, &njs_value_void, 1); if (nxt_slow_path(prop == NULL)) { return NXT_ERROR; @@ -695,11 +708,9 @@ njs_vmcode_property_delete(njs_vm_t *vm, return NXT_ERROR; } - pq.lhq.pool = vm->mem_cache_pool; - - (void) nxt_lvlhsh_delete(&object->data.u.object->hash, &pq.lhq); - - njs_release(vm, property); + /* GC: release value. */ + prop->type = NJS_WHITEOUT; + njs_set_invalid(&prop->value); retval = &njs_value_true; @@ -811,7 +822,7 @@ njs_vmcode_property_next(njs_vm_t *vm, n break; } - if (prop->enumerable) { + if (prop->type != NJS_WHITEOUT && prop->enumerable) { *retval = prop->name; return code->offset; diff -r 9d7b9cc03569 -r da92ae8bcae3 njs/test/njs_unit_test.c --- a/njs/test/njs_unit_test.c Fri Oct 19 20:52:57 2018 +0300 +++ b/njs/test/njs_unit_test.c Fri Oct 19 20:54:59 2018 +0300 @@ -2112,6 +2112,14 @@ static njs_unit_test_t njs_test[] = "for (var p in o) {s += p}; s"), nxt_string("y") }, + { nxt_string("var o = {a:1, b:2}; var arr = []; " + "for (var a in o) {arr.push(a)}; arr"), + nxt_string("a,b") }, + + { nxt_string("var o = {a:1, b:2}; var arr = []; delete o.a; " + "for (var a in o) {arr.push(a)}; arr"), + nxt_string("b") }, + /* switch. */ { nxt_string("switch"), @@ -2636,6 +2644,9 @@ static njs_unit_test_t njs_test[] = { nxt_string("delete --[][1]"), nxt_string("true") }, + { nxt_string("var a = [1,2]; delete a.length"), + nxt_string("false") }, + { nxt_string("var a = [1,2,3]; a.x = 10; delete a[1]"), nxt_string("true") }, @@ -2654,6 +2665,14 @@ static njs_unit_test_t njs_test[] = { nxt_string("Math.E = 1"), nxt_string("TypeError: Cannot assign to read-only property 'E' of object") }, + { nxt_string("var o = { 'a': 1, 'b': 2 }; var i; " + "for (i in o) { delete o.a; delete o.b; }; njs.dump(o)"), + nxt_string("{}") }, + + { nxt_string("var o = {}; Object.defineProperty(o, 'a', {value:1, configurable:1}); " + "delete o.a; o.a=2; o.a"), + nxt_string("2") }, + { nxt_string("var a = {}; 1 in a"), nxt_string("false") }, @@ -6894,6 +6913,9 @@ static njs_unit_test_t njs_test[] = { nxt_string("var a = [,6,,3]; a.one = 7; Object.keys(a)"), nxt_string("1,3,one") }, + { nxt_string("var o = {a:1,b:2}; delete o.a; Object.keys(o)"), + nxt_string("b") }, + { nxt_string("Object.keys()"), nxt_string("TypeError: cannot convert void argument to object") }, From xeioex at nginx.com Fri Oct 19 18:30:53 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 19 Oct 2018 18:30:53 +0000 Subject: [njs] Fixed Object.prototype.hasOwnProperty() for non-object properties. Message-ID: details: http://hg.nginx.org/njs/rev/7e76c67af046 branches: changeset: 625:7e76c67af046 user: Dmitry Volyntsev date: Fri Oct 19 20:55:23 2018 +0300 description: Fixed Object.prototype.hasOwnProperty() for non-object properties. This fixes #9 issue on Github. diffstat: njs/njs_object.c | 67 +++++++++++++++++++++-------------------------- njs/test/njs_unit_test.c | 24 +++++++++++++++++ 2 files changed, 54 insertions(+), 37 deletions(-) diffs (120 lines): diff -r da92ae8bcae3 -r 7e76c67af046 njs/njs_object.c --- a/njs/njs_object.c Fri Oct 19 20:54:59 2018 +0300 +++ b/njs/njs_object.c Fri Oct 19 20:55:23 2018 +0300 @@ -2079,45 +2079,38 @@ static njs_ret_t njs_object_prototype_has_own_property(njs_vm_t *vm, njs_value_t *args, nxt_uint_t nargs, njs_index_t unused) { - uint32_t index; - nxt_int_t ret; - njs_array_t *array; - const njs_value_t *value, *prop, *retval; - nxt_lvlhsh_query_t lhq; - - retval = &njs_value_false; - value = &args[0]; - - if (njs_is_object(value)) { - - prop = njs_arg(args, nargs, 1); - - if (njs_is_array(value)) { - array = value->data.u.array; - index = njs_string_to_index(prop); - - if (index < array->length && njs_is_valid(&array->start[index])) { - retval = &njs_value_true; - goto done; - } - } - - njs_string_get(prop, &lhq.key); - lhq.key_hash = nxt_djb_hash(lhq.key.start, lhq.key.length); - lhq.proto = &njs_object_hash_proto; - - ret = nxt_lvlhsh_find(&value->data.u.object->hash, &lhq); - - if (ret == NXT_OK) { - retval = &njs_value_true; - } + nxt_int_t ret; + const njs_value_t *value, *property; + njs_property_query_t pq; + + value = njs_arg(args, nargs, 0); + + if (njs_is_null_or_void(value)) { + njs_type_error(vm, "cannot convert %s argument to object", + njs_type_string(value->type)); + return NXT_ERROR; } -done: - - vm->retval = *retval; - - return NXT_OK; + property = njs_arg(args, nargs, 1); + + njs_property_query_init(&pq, NJS_PROPERTY_QUERY_GET, 1); + + ret = njs_property_query(vm, &pq, (njs_value_t *) value, property); + + switch (ret) { + case NXT_OK: + vm->retval = njs_value_true; + return NXT_OK; + + case NXT_DECLINED: + vm->retval = njs_value_false; + return NXT_OK; + + case NJS_TRAP: + case NXT_ERROR: + default: + return ret; + } } diff -r da92ae8bcae3 -r 7e76c67af046 njs/test/njs_unit_test.c --- a/njs/test/njs_unit_test.c Fri Oct 19 20:54:59 2018 +0300 +++ b/njs/test/njs_unit_test.c Fri Oct 19 20:55:23 2018 +0300 @@ -3671,6 +3671,15 @@ static njs_unit_test_t njs_test[] = { nxt_string("'abc'.length"), nxt_string("3") }, + { nxt_string("''.hasOwnProperty('length')"), + nxt_string("true") }, + + { nxt_string("'abc'.hasOwnProperty('length')"), + nxt_string("true") }, + + { nxt_string("(new String('abc')).hasOwnProperty('length')"), + nxt_string("true") }, + { nxt_string("'abc'.toUTF8().length"), nxt_string("3") }, @@ -7103,6 +7112,21 @@ static njs_unit_test_t njs_test[] = { nxt_string("[,].hasOwnProperty()"), nxt_string("false") }, + { nxt_string("[1,2].hasOwnProperty('len')"), + nxt_string("false") }, + + { nxt_string("[].hasOwnProperty('length')"), + nxt_string("true") }, + + { nxt_string("[1,2].hasOwnProperty('length')"), + nxt_string("true") }, + + { nxt_string("(new Array()).hasOwnProperty('length')"), + nxt_string("true") }, + + { nxt_string("(new Array(10)).hasOwnProperty('length')"), + nxt_string("true") }, + { nxt_string("Object.valueOf.hasOwnProperty()"), nxt_string("false") }, From xeioex at nginx.com Fri Oct 19 18:30:54 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 19 Oct 2018 18:30:54 +0000 Subject: [njs] Added njs_primitive_value_to_number(). Message-ID: details: http://hg.nginx.org/njs/rev/bd9f53d9de27 branches: changeset: 626:bd9f53d9de27 user: Dmitry Volyntsev date: Fri Oct 19 20:55:30 2018 +0300 description: Added njs_primitive_value_to_number(). diffstat: njs/njs_number.c | 18 ++++++++++++++++++ njs/njs_number.h | 2 ++ 2 files changed, 20 insertions(+), 0 deletions(-) diffs (40 lines): diff -r 7e76c67af046 -r bd9f53d9de27 njs/njs_number.c --- a/njs/njs_number.c Fri Oct 19 20:55:23 2018 +0300 +++ b/njs/njs_number.c Fri Oct 19 20:55:30 2018 +0300 @@ -64,6 +64,24 @@ njs_value_to_index(const njs_value_t *va double +njs_primitive_value_to_number(const njs_value_t *value) +{ + if (nxt_fast_path(njs_is_numeric(value))) { + return value->data.u.number; + } + + return njs_string_to_number(value, 1); +} + + +uint32_t +njs_primitive_value_to_integer(const njs_value_t *value) +{ + return njs_number_to_integer(njs_primitive_value_to_number(value)); +} + + +double njs_number_dec_parse(const u_char **start, const u_char *end) { return nxt_strtod(start, end); diff -r 7e76c67af046 -r bd9f53d9de27 njs/njs_number.h --- a/njs/njs_number.h Fri Oct 19 20:55:23 2018 +0300 +++ b/njs/njs_number.h Fri Oct 19 20:55:30 2018 +0300 @@ -13,6 +13,8 @@ uint32_t njs_value_to_index(const njs_value_t *value); +double njs_primitive_value_to_number(const njs_value_t *value); +uint32_t njs_primitive_value_to_integer(const njs_value_t *value); double njs_number_dec_parse(const u_char **start, const u_char *end); uint64_t njs_number_oct_parse(const u_char **start, const u_char *end); uint64_t njs_number_bin_parse(const u_char **start, const u_char *end); From xeioex at nginx.com Fri Oct 19 18:30:53 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 19 Oct 2018 18:30:53 +0000 Subject: [njs] Object property quering is refactored. Message-ID: details: http://hg.nginx.org/njs/rev/9d7b9cc03569 branches: changeset: 623:9d7b9cc03569 user: Dmitry Volyntsev date: Fri Oct 19 20:52:57 2018 +0300 description: Object property quering is refactored. njs_property_query() is rectified and unified 1) returns only property descriptors. Special return codes NJS_PRIMITIVE_VALUE, NJS_STRING_VALUE, NJS_ARRAY_VALUE and NJS_EXTERNAL_VALUE are replaced with a temporary property descriptor of type NJS_PROPERTY_REF or NJS_PROPERTY_HANDLER. If NJS_PROPERTY_REF is set reference to a value is contained in prop->value.data.u.value. 2) NJS_PROPERTY_HANDLER properties returned as is. 3) njs_property_query_t.own can be used to query for an object's OwnProperty. 4) NJS_PROPERTY_QUERY_IN is removed. The aim is to implement with it [[GetOwnProperty]] and [[GetProperty]] methods from specification. Which are used extensively in many places of the ECMAScript spec. njs_value_property() is introduced which corresponds to [[Get]] method from specification. This fixes #32 and #34 issues on Github. diffstat: njs/njs_extern.h | 3 + njs/njs_object.c | 620 ++++++++++++++++++++++++++++----------- njs/njs_object.h | 24 +- njs/njs_vm.c | 668 ++++++++++++++++-------------------------- njs/njs_vm.h | 24 +- njs/test/njs_expect_test.exp | 2 +- njs/test/njs_unit_test.c | 527 +++++++++++++++++++++++++++++++-- 7 files changed, 1228 insertions(+), 640 deletions(-) diffs (truncated from 2473 to 1000 lines): diff -r 6b06b00139a1 -r 9d7b9cc03569 njs/njs_extern.h --- a/njs/njs_extern.h Sat Oct 06 17:52:40 2018 +0300 +++ b/njs/njs_extern.h Fri Oct 19 20:52:57 2018 +0300 @@ -11,6 +11,9 @@ #define njs_extern_object(vm, ext) \ (*(void **) nxt_array_item((vm)->external_objects, (ext)->external.index)) +#define njs_extern_index(vm, idx) \ + (*(void **) nxt_array_item((vm)->external_objects, idx)) + struct njs_extern_s { /* A hash of inclusive njs_extern_t. */ diff -r 6b06b00139a1 -r 9d7b9cc03569 njs/njs_object.c --- a/njs/njs_object.c Sat Oct 06 17:52:40 2018 +0300 +++ b/njs/njs_object.c Fri Oct 19 20:52:57 2018 +0300 @@ -10,9 +10,18 @@ static nxt_int_t njs_object_hash_test(nxt_lvlhsh_query_t *lhq, void *data); static njs_ret_t njs_object_property_query(njs_vm_t *vm, - njs_property_query_t *pq, njs_value_t *value, njs_object_t *object); + njs_property_query_t *pq, njs_object_t *object, + const njs_value_t *property); static njs_ret_t njs_array_property_query(njs_vm_t *vm, + njs_property_query_t *pq, njs_array_t *array, uint32_t index); +static njs_ret_t njs_string_property_query(njs_vm_t *vm, njs_property_query_t *pq, njs_value_t *object, uint32_t index); +static njs_ret_t njs_external_property_query(njs_vm_t *vm, + njs_property_query_t *pq, njs_value_t *object); +static njs_ret_t njs_external_property_set(njs_vm_t *vm, njs_value_t *value, + njs_value_t *setval, njs_value_t *retval); +static njs_ret_t njs_external_property_delete(njs_vm_t *vm, njs_value_t *value, + njs_value_t *setval, njs_value_t *retval); static njs_ret_t njs_object_query_prop_handler(njs_property_query_t *pq, njs_object_t *object); static njs_ret_t njs_define_property(njs_vm_t *vm, njs_object_t *object, @@ -232,28 +241,39 @@ njs_object_property(njs_vm_t *vm, const /* + * ES5.1, 8.12.1: [[GetOwnProperty]], [[GetProperty]]. * The njs_property_query() returns values * NXT_OK property has been found in object, + * retval of type njs_object_prop_t * is in pq->lhq.value. + * in NJS_PROPERTY_QUERY_GET + * prop->type is NJS_PROPERTY, NJS_METHOD or NJS_PROPERTY_HANDLER. + * in NJS_PROPERTY_QUERY_SET, NJS_PROPERTY_QUERY_DELETE + * prop->type is NJS_PROPERTY, NJS_PROPERTY_REF, NJS_METHOD or + * NJS_PROPERTY_HANDLER. * NXT_DECLINED property was not found in object, - * NJS_PRIMITIVE_VALUE property operation was applied to a numeric - * or boolean value, - * NJS_STRING_VALUE property operation was applied to a string, - * NJS_ARRAY_VALUE object is array, - * NJS_EXTERNAL_VALUE object is external entity, + * if pq->lhq.value != NULL it contains retval of type + * njs_object_prop_t * where prop->type is NJS_WHITEOUT * NJS_TRAP the property trap must be called, * NXT_ERROR exception has been thrown. + * + * TODO: + * Object.create([1,2]).length + * Object.defineProperty([1,2], '1', {configurable:false}) */ njs_ret_t njs_property_query(njs_vm_t *vm, njs_property_query_t *pq, njs_value_t *object, - njs_value_t *property) + const njs_value_t *property) { - uint32_t index; - uint32_t (*hash)(const void *, size_t); - njs_ret_t ret; - njs_object_t *obj; - njs_function_t *function; - const njs_extern_t *ext_proto; + uint32_t index; + uint32_t (*hash)(const void *, size_t); + njs_ret_t ret; + njs_object_t *obj; + njs_function_t *function; + + if (nxt_slow_path(!njs_is_primitive(property))) { + return njs_trap(vm, NJS_TRAP_PROPERTY); + } hash = nxt_djb_hash; @@ -261,34 +281,47 @@ njs_property_query(njs_vm_t *vm, njs_pro case NJS_BOOLEAN: case NJS_NUMBER: - if (pq->query != NJS_PROPERTY_QUERY_GET) { - return NJS_PRIMITIVE_VALUE; - } - index = njs_primitive_prototype_index(object->type); obj = &vm->prototypes[index].object; break; case NJS_STRING: - if (pq->query == NJS_PROPERTY_QUERY_DELETE) { - return NXT_DECLINED; + if (nxt_fast_path(!njs_is_null_or_void_or_boolean(property))) { + index = njs_value_to_index(property); + + if (nxt_fast_path(index < NJS_STRING_MAX_LENGTH)) { + return njs_string_property_query(vm, pq, object, index); + } } obj = &vm->prototypes[NJS_PROTOTYPE_STRING].object; break; + case NJS_OBJECT_STRING: + if (nxt_fast_path(!njs_is_null_or_void_or_boolean(property))) { + index = njs_value_to_index(property); + + if (nxt_fast_path(index < NJS_STRING_MAX_LENGTH)) { + ret = njs_string_property_query(vm, pq, + &object->data.u.object_value->value, + index); + + if (nxt_fast_path(ret != NXT_DECLINED)) { + return ret; + } + } + } + + obj = object->data.u.object; + break; + case NJS_ARRAY: if (nxt_fast_path(!njs_is_null_or_void_or_boolean(property))) { - - if (nxt_fast_path(njs_is_primitive(property))) { - index = njs_value_to_index(property); - - if (nxt_fast_path(index < NJS_ARRAY_MAX_LENGTH)) { - return njs_array_property_query(vm, pq, object, index); - } - - } else { - return njs_trap(vm, NJS_TRAP_PROPERTY); + index = njs_value_to_index(property); + + if (nxt_fast_path(index < NJS_ARRAY_MAX_LENGTH)) { + return njs_array_property_query(vm, pq, object->data.u.array, + index); } } @@ -297,7 +330,6 @@ njs_property_query(njs_vm_t *vm, njs_pro case NJS_OBJECT: case NJS_OBJECT_BOOLEAN: case NJS_OBJECT_NUMBER: - case NJS_OBJECT_STRING: case NJS_REGEXP: case NJS_DATE: case NJS_OBJECT_ERROR: @@ -322,28 +354,19 @@ njs_property_query(njs_vm_t *vm, njs_pro break; case NJS_EXTERNAL: - ext_proto = object->external.proto; - - if (ext_proto->type == NJS_EXTERN_CASELESS_OBJECT) { - hash = nxt_djb_hash_lowcase; - } - obj = NULL; break; case NJS_VOID: case NJS_NULL: default: - if (nxt_fast_path(njs_is_primitive(property))) { - - ret = njs_primitive_value_to_string(vm, &pq->value, property); - - if (nxt_fast_path(ret == NXT_OK)) { - njs_string_get(&pq->value, &pq->lhq.key); - njs_type_error(vm, "cannot get property '%.*s' of undefined", - (int) pq->lhq.key.length, pq->lhq.key.start); - return NXT_ERROR; - } + ret = njs_primitive_value_to_string(vm, &pq->value, property); + + if (nxt_fast_path(ret == NXT_OK)) { + njs_string_get(&pq->value, &pq->lhq.key); + njs_type_error(vm, "cannot get property '%.*s' of undefined", + (int) pq->lhq.key.length, pq->lhq.key.start); + return NXT_ERROR; } njs_type_error(vm, "cannot get property 'unknown' of undefined"); @@ -351,37 +374,34 @@ njs_property_query(njs_vm_t *vm, njs_pro return NXT_ERROR; } - if (nxt_fast_path(njs_is_primitive(property))) { - - ret = njs_primitive_value_to_string(vm, &pq->value, property); - - if (nxt_fast_path(ret == NXT_OK)) { - - njs_string_get(&pq->value, &pq->lhq.key); - pq->lhq.key_hash = hash(pq->lhq.key.start, pq->lhq.key.length); - - if (obj == NULL) { - pq->lhq.proto = &njs_extern_hash_proto; - - return NJS_EXTERNAL_VALUE; - } - - return njs_object_property_query(vm, pq, object, obj); + ret = njs_primitive_value_to_string(vm, &pq->value, property); + + if (nxt_fast_path(ret == NXT_OK)) { + + njs_string_get(&pq->value, &pq->lhq.key); + pq->lhq.key_hash = hash(pq->lhq.key.start, pq->lhq.key.length); + + if (obj == NULL) { + return njs_external_property_query(vm, pq, object); } - return ret; + return njs_object_property_query(vm, pq, obj, property); } - return njs_trap(vm, NJS_TRAP_PROPERTY); + return ret; } njs_ret_t njs_object_property_query(njs_vm_t *vm, njs_property_query_t *pq, - njs_value_t *value, njs_object_t *object) + njs_object_t *object, const njs_value_t *property) { - njs_ret_t ret; - njs_object_prop_t *prop; + uint32_t index; + njs_ret_t ret; + njs_array_t *array; + njs_object_t *proto; + njs_object_prop_t *prop; + njs_object_value_t *ov; pq->lhq.proto = &njs_object_hash_proto; @@ -392,83 +412,83 @@ njs_object_property_query(njs_vm_t *vm, } } + proto = object; + do { - pq->prototype = object; - - ret = nxt_lvlhsh_find(&object->hash, &pq->lhq); - - if (ret == NXT_OK) { - prop = pq->lhq.value; - - if (prop->type != NJS_WHITEOUT) { - pq->shared = 0; - - return ret; + pq->prototype = proto; + + /* length and other shared properties should be Own property */ + + if (nxt_fast_path(!pq->own || proto == object)) { + ret = nxt_lvlhsh_find(&proto->hash, &pq->lhq); + + if (ret == NXT_OK) { + prop = pq->lhq.value; + + if (prop->type != NJS_WHITEOUT) { + pq->shared = 0; + + return ret; + } + + goto next; } - goto next; + if (proto != object && !njs_is_null_or_void_or_boolean(property)) { + switch (proto->type) { + case NJS_ARRAY: + index = njs_value_to_index(property); + if (nxt_fast_path(index < NJS_ARRAY_MAX_LENGTH)) { + array = (njs_array_t *) proto; + return njs_array_property_query(vm, pq, array, index); + } + + break; + + case NJS_OBJECT_STRING: + index = njs_value_to_index(property); + if (nxt_fast_path(index < NJS_STRING_MAX_LENGTH)) { + ov = (njs_object_value_t *) proto; + return njs_string_property_query(vm, pq, &ov->value, + index); + } + + default: + break; + } + } } - if (pq->query > NJS_PROPERTY_QUERY_IN) { - /* NXT_DECLINED */ - return ret; - } - - ret = nxt_lvlhsh_find(&object->shared_hash, &pq->lhq); + ret = nxt_lvlhsh_find(&proto->shared_hash, &pq->lhq); if (ret == NXT_OK) { pq->shared = 1; - if (pq->query == NJS_PROPERTY_QUERY_GET) { - prop = pq->lhq.value; - - if (prop->type == NJS_PROPERTY_HANDLER) { - pq->scratch = *prop; - prop = &pq->scratch; - ret = prop->value.data.u.prop_handler(vm, value, NULL, - &prop->value); - - if (nxt_fast_path(ret == NXT_OK)) { - prop->type = NJS_PROPERTY; - pq->lhq.value = prop; - } - } - } - return ret; } - if (pq->query > NJS_PROPERTY_QUERY_IN) { - /* NXT_DECLINED */ - return ret; + if (pq->query > NJS_PROPERTY_QUERY_GET) { + return NXT_DECLINED; } - next: - - object = object->__proto__; - - } while (object != NULL); - - if (njs_is_string(value)) { - return NJS_STRING_VALUE; - } - - /* NXT_DECLINED */ - - return ret; +next: + + proto = proto->__proto__; + + } while (proto != NULL); + + return NXT_DECLINED; } static njs_ret_t njs_array_property_query(njs_vm_t *vm, njs_property_query_t *pq, - njs_value_t *object, uint32_t index) + njs_array_t *array, uint32_t index) { - uint32_t size; - njs_ret_t ret; - njs_value_t *value; - njs_array_t *array; - - array = object->data.u.array; + uint32_t size; + njs_ret_t ret; + njs_value_t *value; + njs_object_prop_t *prop; if (index >= array->length) { if (pq->query != NJS_PROPERTY_QUERY_SET) { @@ -493,9 +513,200 @@ njs_array_property_query(njs_vm_t *vm, n array->length = index + 1; } - pq->lhq.value = &array->start[index]; - - return NJS_ARRAY_VALUE; + prop = &pq->scratch; + + if (pq->query == NJS_PROPERTY_QUERY_GET) { + if (!njs_is_valid(&array->start[index])) { + return NXT_DECLINED; + } + + prop->value = array->start[index]; + prop->type = NJS_PROPERTY; + + } else { + prop->value.data.u.value = &array->start[index]; + prop->type = NJS_PROPERTY_REF; + } + + prop->configurable = 1; + prop->enumerable = 1; + prop->writable = 1; + + pq->lhq.value = prop; + + return NXT_OK; +} + + +static njs_ret_t +njs_string_property_query(njs_vm_t *vm, njs_property_query_t *pq, + njs_value_t *object, uint32_t index) +{ + njs_slice_prop_t slice; + njs_object_prop_t *prop; + njs_string_prop_t string; + + prop = &pq->scratch; + + slice.start = index; + slice.length = 1; + slice.string_length = njs_string_prop(&string, object); + + if (slice.start < slice.string_length) { + /* + * A single codepoint string fits in retval + * so the function cannot fail. + */ + (void) njs_string_slice(vm, &prop->value, &string, &slice); + prop->type = NJS_PROPERTY; + prop->configurable = 0; + prop->enumerable = 1; + prop->writable = 0; + + pq->lhq.value = prop; + + if (pq->query != NJS_PROPERTY_QUERY_GET) { + /* pq->lhq.key is used by njs_vmcode_property_set for TypeError */ + njs_uint32_to_string(&pq->value, index); + njs_string_get(&pq->value, &pq->lhq.key); + } + + return NXT_OK; + } + + return NXT_DECLINED; +} + + +static njs_ret_t +njs_external_property_query(njs_vm_t *vm, njs_property_query_t *pq, + njs_value_t *object) +{ + void *obj; + njs_ret_t ret; + uintptr_t data; + njs_object_prop_t *prop; + const njs_extern_t *ext_proto; + + prop = &pq->scratch; + + prop->type = NJS_PROPERTY; + prop->configurable = 0; + prop->enumerable = 1; + prop->writable = 0; + + ext_proto = object->external.proto; + + pq->lhq.proto = &njs_extern_hash_proto; + ret = nxt_lvlhsh_find(&ext_proto->hash, &pq->lhq); + + if (ret == NXT_OK) { + ext_proto = pq->lhq.value; + + prop->value.type = NJS_EXTERNAL; + prop->value.data.truth = 1; + prop->value.external.proto = ext_proto; + prop->value.external.index = object->external.index; + + if ((ext_proto->type & NJS_EXTERN_OBJECT) != 0) { + goto done; + } + + data = ext_proto->data; + + } else { + data = (uintptr_t) &pq->lhq.key; + } + + switch (pq->query) { + + case NJS_PROPERTY_QUERY_GET: + if (ext_proto->get != NULL) { + obj = njs_extern_object(vm, object); + ret = ext_proto->get(vm, &prop->value, obj, data); + if (nxt_slow_path(ret != NXT_OK)) { + return ret; + } + } + + break; + + case NJS_PROPERTY_QUERY_SET: + case NJS_PROPERTY_QUERY_DELETE: + + prop->type = NJS_PROPERTY_HANDLER; + prop->name = *object; + + if (pq->query == NJS_PROPERTY_QUERY_SET) { + prop->writable = (ext_proto->set != NULL); + prop->value.data.u.prop_handler = njs_external_property_set; + + } else { + prop->configurable = (ext_proto->find != NULL); + prop->value.data.u.prop_handler = njs_external_property_delete; + } + + pq->ext_data = data; + pq->ext_proto = ext_proto; + pq->ext_index = object->external.index; + + pq->lhq.value = prop; + + vm->stash = (uintptr_t) pq; + + return NXT_OK; + } + +done: + + if (ext_proto->type == NJS_EXTERN_METHOD) { + prop->value.type = NJS_FUNCTION; + prop->value.data.u.function = ext_proto->function; + prop->value.data.truth = 1; + } + + pq->lhq.value = prop; + + return ret; +} + + +static njs_ret_t +njs_external_property_set(njs_vm_t *vm, njs_value_t *value, njs_value_t *setval, + njs_value_t *retval) +{ + void *obj; + njs_ret_t ret; + nxt_str_t s; + njs_property_query_t *pq; + + pq = (njs_property_query_t *) vm->stash; + + ret = njs_vm_value_to_ext_string(vm, &s, setval, 0); + if (nxt_slow_path(ret != NXT_OK)) { + return ret; + } + + *retval = *setval; + + obj = njs_extern_index(vm, pq->ext_index); + + return pq->ext_proto->set(vm, obj, pq->ext_data, &s); +} + + +static njs_ret_t +njs_external_property_delete(njs_vm_t *vm, njs_value_t *value, + njs_value_t *unused, njs_value_t *unused2) +{ + void *obj; + njs_property_query_t *pq; + + pq = (njs_property_query_t *) vm->stash; + + obj = njs_extern_index(vm, pq->ext_index); + + return pq->ext_proto->find(vm, obj, pq->ext_data, 1); } @@ -527,6 +738,34 @@ njs_object_query_prop_handler(njs_proper njs_ret_t +njs_method_private_copy(njs_vm_t *vm, njs_property_query_t *pq) +{ + njs_function_t *function; + njs_object_prop_t *prop, *shared; + + prop = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_object_prop_t)); + if (nxt_slow_path(prop == NULL)) { + njs_memory_error(vm); + return NXT_ERROR; + } + + shared = pq->lhq.value; + *prop = *shared; + + function = njs_function_value_copy(vm, &prop->value); + if (nxt_slow_path(function == NULL)) { + return NXT_ERROR; + } + + pq->lhq.replace = 0; + pq->lhq.value = prop; + pq->lhq.pool = vm->mem_cache_pool; + + return nxt_lvlhsh_insert(&pq->prototype->hash, &pq->lhq); +} + + +njs_ret_t njs_object_constructor(njs_vm_t *vm, njs_value_t *args, nxt_uint_t nargs, njs_index_t unused) { @@ -988,68 +1227,75 @@ static njs_ret_t njs_object_get_own_property_descriptor(njs_vm_t *vm, njs_value_t *args, nxt_uint_t nargs, njs_index_t unused) { - double num; - uint32_t index; nxt_int_t ret; - njs_array_t *array; njs_object_t *descriptor; - njs_object_prop_t *pr, *prop, array_prop; + njs_object_prop_t *pr, *prop; const njs_value_t *value, *property, *setval; nxt_lvlhsh_query_t lhq; njs_property_query_t pq; value = njs_arg(args, nargs, 1); - if (!njs_is_object(value)) { - if (njs_is_null_or_void(value)) { - njs_type_error(vm, "cannot convert %s argument to object", - njs_type_string(value->type)); - return NXT_ERROR; - } - + if (njs_is_null_or_void(value)) { + njs_type_error(vm, "cannot convert %s argument to object", + njs_type_string(value->type)); + return NXT_ERROR; + } + + property = njs_arg(args, nargs, 2); + + njs_property_query_init(&pq, NJS_PROPERTY_QUERY_GET, 1); + + ret = njs_property_query(vm, &pq, (njs_value_t *) value, property); + + switch (ret) { + case NXT_OK: + break; + + case NXT_DECLINED: vm->retval = njs_value_void; return NXT_OK; + + case NJS_TRAP: + case NXT_ERROR: + default: + return ret; } - prop = NULL; - property = njs_arg(args, nargs, 2); - - if (njs_is_array(value)) { - array = value->data.u.array; - num = njs_string_to_index(property); - index = num; - - if ((double) index == num - && index < array->length - && njs_is_valid(&array->start[index])) - { - prop = &array_prop; - - array_prop.name = *property; - array_prop.value = array->start[index]; - - array_prop.configurable = 1; - array_prop.enumerable = 1; - array_prop.writable = 1; + prop = pq.lhq.value; + + switch (prop->type) { + case NJS_PROPERTY: + break; + + case NJS_PROPERTY_HANDLER: + pq.scratch = *prop; + prop = &pq.scratch; + ret = prop->value.data.u.prop_handler(vm, (njs_value_t *) value, + NULL, &prop->value); + if (nxt_slow_path(ret != NXT_OK)) { + return ret; } - } - - lhq.proto = &njs_object_hash_proto; - - if (prop == NULL) { - pq.query = NJS_PROPERTY_QUERY_GET; - pq.lhq.key.length = 0; - pq.lhq.key.start = NULL; - - ret = njs_property_query(vm, &pq, (njs_value_t *) value, - (njs_value_t *) property); - - if (ret != NXT_OK) { - vm->retval = njs_value_void; - return NXT_OK; + + break; + + case NJS_METHOD: + if (pq.shared) { + ret = njs_method_private_copy(vm, &pq); + + if (nxt_slow_path(ret != NXT_OK)) { + return ret; + } + + prop = pq.lhq.value; } - prop = pq.lhq.value; + break; + + default: + njs_type_error(vm, "unexpected property type: %s", + njs_prop_type_string(prop->type)); + return NXT_ERROR; } descriptor = njs_object_alloc(vm); @@ -1057,6 +1303,7 @@ njs_object_get_own_property_descriptor(n return NXT_ERROR; } + lhq.proto = &njs_object_hash_proto; lhq.replace = 0; lhq.pool = vm->mem_cache_pool; lhq.proto = &njs_object_hash_proto; @@ -1953,3 +2200,28 @@ const njs_object_init_t njs_object_prot njs_object_prototype_properties, nxt_nitems(njs_object_prototype_properties), }; + + +const char * +njs_prop_type_string(njs_object_property_type_t type) +{ + switch (type) { + case NJS_PROPERTY_REF: + return "property_ref"; + + case NJS_METHOD: + return "method"; + + case NJS_PROPERTY_HANDLER: + return "property handler"; + + case NJS_WHITEOUT: + return "whiteout"; + + case NJS_PROPERTY: + return "property"; + + default: + return "unknown"; + } +} diff -r 6b06b00139a1 -r 9d7b9cc03569 njs/njs_object.h --- a/njs/njs_object.h Sat Oct 06 17:52:40 2018 +0300 +++ b/njs/njs_object.h Fri Oct 19 20:52:57 2018 +0300 @@ -10,8 +10,7 @@ typedef enum { NJS_PROPERTY = 0, - NJS_GETTER, - NJS_SETTER, + NJS_PROPERTY_REF, NJS_METHOD, NJS_PROPERTY_HANDLER, NJS_WHITEOUT, @@ -47,13 +46,27 @@ typedef struct { /* scratch is used to get the value of an NJS_PROPERTY_HANDLER property. */ njs_object_prop_t scratch; + /* These three fields are used for NJS_EXTERNAL setters. */ + uintptr_t ext_data; + const njs_extern_t *ext_proto; + uint32_t ext_index; + njs_value_t value; njs_object_t *prototype; uint8_t query; uint8_t shared; + uint8_t own; } njs_property_query_t; +#define njs_property_query_init(pq, _query, _own) \ + do { \ + (pq)->lhq.key.length = 0; \ + (pq)->lhq.value = NULL; \ + (pq)->query = _query; \ + (pq)->own = _own; \ + } while (0) + struct njs_object_init_s { nxt_str_t name; @@ -67,10 +80,12 @@ njs_object_t *njs_object_value_copy(njs_ njs_object_t *njs_object_value_alloc(njs_vm_t *vm, const njs_value_t *value, nxt_uint_t type); njs_array_t *njs_object_keys_array(njs_vm_t *vm, const njs_value_t *object); +njs_ret_t njs_value_property(njs_vm_t *vm, njs_value_t *value, + const njs_value_t *property, njs_value_t *retval); njs_object_prop_t *njs_object_property(njs_vm_t *vm, const njs_object_t *obj, nxt_lvlhsh_query_t *lhq); njs_ret_t njs_property_query(njs_vm_t *vm, njs_property_query_t *pq, - njs_value_t *object, njs_value_t *property); + njs_value_t *object, const njs_value_t *property); nxt_int_t njs_object_hash_create(njs_vm_t *vm, nxt_lvlhsh_t *hash, const njs_object_prop_t *prop, nxt_uint_t n); njs_ret_t njs_object_constructor(njs_vm_t *vm, njs_value_t *args, @@ -90,6 +105,9 @@ njs_value_t *njs_property_constructor_cr njs_ret_t njs_object_prototype_to_string(njs_vm_t *vm, njs_value_t *args, nxt_uint_t nargs, njs_index_t unused); +njs_ret_t njs_method_private_copy(njs_vm_t *vm, njs_property_query_t *pq); +const char * njs_prop_type_string(njs_object_property_type_t type); + extern const njs_object_init_t njs_object_constructor_init; extern const njs_object_init_t njs_object_prototype_init; diff -r 6b06b00139a1 -r 9d7b9cc03569 njs/njs_vm.c --- a/njs/njs_vm.c Sat Oct 06 17:52:40 2018 +0300 +++ b/njs/njs_vm.c Fri Oct 19 20:52:57 2018 +0300 @@ -25,8 +25,6 @@ struct njs_property_next_s { static nxt_noinline njs_ret_t njs_string_concat(njs_vm_t *vm, njs_value_t *val1, njs_value_t *val2); -static njs_ret_t njs_method_private_copy(njs_vm_t *vm, - njs_property_query_t *pq); static nxt_noinline njs_ret_t njs_values_equal(njs_vm_t *vm, const njs_value_t *val1, const njs_value_t *val2); static nxt_noinline njs_ret_t njs_values_compare(njs_vm_t *vm, @@ -476,150 +474,14 @@ njs_ret_t njs_vmcode_property_get(njs_vm_t *vm, njs_value_t *object, njs_value_t *property) { - void *obj; - int32_t index; - uintptr_t data; - njs_ret_t ret; - njs_value_t *val, ext_val; - njs_slice_prop_t slice; - njs_string_prop_t string; - njs_object_prop_t *prop; - const njs_value_t *retval; - const njs_extern_t *ext_proto; - njs_property_query_t pq; - - pq.query = NJS_PROPERTY_QUERY_GET; - - ret = njs_property_query(vm, &pq, object, property); - - retval = &njs_value_void; - - switch (ret) { - - case NXT_OK: - prop = pq.lhq.value; - - switch (prop->type) { - - case NJS_METHOD: - if (pq.shared) { - ret = njs_method_private_copy(vm, &pq); - - if (nxt_slow_path(ret != NXT_OK)) { - return ret; - } - - prop = pq.lhq.value; - } - - /* Fall through. */ - - case NJS_PROPERTY: - retval = &prop->value; - break; - - default: - nxt_thread_log_alert("invalid property get type:%d", prop->type); - - return NXT_ERROR; - } - - break; - - case NXT_DECLINED: - case NJS_PRIMITIVE_VALUE: - break; - - case NJS_STRING_VALUE: - - /* string[n]. */ - - index = (int32_t) njs_value_to_index(property); - - if (nxt_fast_path(index >= 0)) { - slice.start = index; - slice.length = 1; - slice.string_length = njs_string_prop(&string, object); - - if (slice.start < slice.string_length) { - /* - * A single codepoint string fits in vm->retval - * so the function cannot fail. - */ - (void) njs_string_slice(vm, &vm->retval, &string, &slice); - - return sizeof(njs_vmcode_prop_get_t); - } - } - - break; - - case NJS_ARRAY_VALUE: - val = pq.lhq.value; - - if (njs_is_valid(val)) { - retval = val; - } - - break; - - case NJS_EXTERNAL_VALUE: - ext_proto = object->external.proto; - - ret = nxt_lvlhsh_find(&ext_proto->hash, &pq.lhq); - - if (ret == NXT_OK) { - ext_proto = pq.lhq.value; - - ext_val.type = NJS_EXTERNAL; - ext_val.data.truth = 1; - ext_val.external.proto = ext_proto; - ext_val.external.index = object->external.index; - - if ((ext_proto->type & NJS_EXTERN_OBJECT) != 0) { - retval = &ext_val; - break; - } - - data = ext_proto->data; - - } else { - data = (uintptr_t) &pq.lhq.key; - } - - vm->retval = njs_value_void; - - if (ext_proto->get != NULL) { - obj = njs_extern_object(vm, object); - - ret = ext_proto->get(vm, &vm->retval, obj, data); - if (nxt_slow_path(ret != NXT_OK)) { - return ret; - } - - /* The vm->retval is already retained by ext_proto->get(). */ - } - - if (ext_proto->type == NJS_EXTERN_METHOD) { - vm->retval.data.u.function = ext_proto->function; - vm->retval.type = NJS_FUNCTION; From xeioex at nginx.com Fri Oct 19 18:30:54 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 19 Oct 2018 18:30:54 +0000 Subject: [njs] Added njs_string_slice_string_prop(). Message-ID: details: http://hg.nginx.org/njs/rev/084484ab232b branches: changeset: 627:084484ab232b user: Dmitry Volyntsev date: Fri Oct 19 20:55:33 2018 +0300 description: Added njs_string_slice_string_prop(). diffstat: njs/njs_string.c | 22 ++++++++++++++++++---- njs/njs_string.h | 2 ++ 2 files changed, 20 insertions(+), 4 deletions(-) diffs (51 lines): diff -r bd9f53d9de27 -r 084484ab232b njs/njs_string.c --- a/njs/njs_string.c Fri Oct 19 20:55:30 2018 +0300 +++ b/njs/njs_string.c Fri Oct 19 20:55:33 2018 +0300 @@ -1287,8 +1287,8 @@ njs_string_slice_args(njs_slice_prop_t * } -nxt_noinline njs_ret_t -njs_string_slice(njs_vm_t *vm, njs_value_t *dst, +nxt_noinline void +njs_string_slice_string_prop(njs_string_prop_t *dst, const njs_string_prop_t *string, const njs_slice_prop_t *slice) { size_t size, n, length; @@ -1325,8 +1325,22 @@ njs_string_slice(njs_vm_t *vm, njs_value length -= n; } - if (nxt_fast_path(size != 0)) { - return njs_string_new(vm, dst, start, size, length); + dst->start = (u_char *) start; + dst->length = length; + dst->size = size; +} + + +nxt_noinline njs_ret_t +njs_string_slice(njs_vm_t *vm, njs_value_t *dst, + const njs_string_prop_t *string, const njs_slice_prop_t *slice) +{ + njs_string_prop_t prop; + + njs_string_slice_string_prop(&prop, string, slice); + + if (nxt_fast_path(prop.size != 0)) { + return njs_string_new(vm, dst, prop.start, prop.size, prop.length); } *dst = njs_string_empty; diff -r bd9f53d9de27 -r 084484ab232b njs/njs_string.h --- a/njs/njs_string.h Fri Oct 19 20:55:30 2018 +0300 +++ b/njs/njs_string.h Fri Oct 19 20:55:33 2018 +0300 @@ -145,6 +145,8 @@ njs_ret_t njs_string_constructor(njs_vm_ nxt_uint_t nargs, njs_index_t unused); nxt_bool_t njs_string_eq(const njs_value_t *val1, const njs_value_t *val2); nxt_int_t njs_string_cmp(const njs_value_t *val1, const njs_value_t *val2); +nxt_noinline void njs_string_slice_string_prop(njs_string_prop_t *dst, + const njs_string_prop_t *string, const njs_slice_prop_t *slice); njs_ret_t njs_string_slice(njs_vm_t *vm, njs_value_t *dst, const njs_string_prop_t *string, const njs_slice_prop_t *slice); const u_char *njs_string_offset(const u_char *start, const u_char *end, From xeioex at nginx.com Fri Oct 19 18:30:54 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 19 Oct 2018 18:30:54 +0000 Subject: [njs] Handling int overflow in njs_array_alloc() on 32bit archs. Message-ID: details: http://hg.nginx.org/njs/rev/8ab908b0f226 branches: changeset: 628:8ab908b0f226 user: Dmitry Volyntsev date: Fri Oct 19 20:55:38 2018 +0300 description: Handling int overflow in njs_array_alloc() on 32bit archs. diffstat: njs/njs_array.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diffs (25 lines): diff -r 084484ab232b -r 8ab908b0f226 njs/njs_array.c --- a/njs/njs_array.c Fri Oct 19 20:55:33 2018 +0300 +++ b/njs/njs_array.c Fri Oct 19 20:55:38 2018 +0300 @@ -109,7 +109,7 @@ static njs_ret_t njs_array_prototype_sor nxt_noinline njs_array_t * njs_array_alloc(njs_vm_t *vm, uint32_t length, uint32_t spare) { - size_t size; + uint64_t size; njs_array_t *array; array = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_array_t)); @@ -117,9 +117,9 @@ njs_array_alloc(njs_vm_t *vm, uint32_t l goto memory_error; } - size = (size_t) length + spare; - - if (nxt_slow_path(size * sizeof(njs_value_t) < size)) { + size = (uint64_t) length + spare; + + if (nxt_slow_path((size * sizeof(njs_value_t)) >= 0xffffffff)) { goto memory_error; } From postmaster at hubject.net Mon Oct 22 11:57:42 2018 From: postmaster at hubject.net (Postmaster) Date: Mon, 22 Oct 2018 13:57:42 +0200 Subject: NGINX with CGI on Alpine Linux In-Reply-To: <53EAC8D3-5365-4D3C-BC13-5A9C5D4AAB10@hubject.net> References: <43EB1F9C-98EE-4924-A597-70792283F47A@hubject.net> <53EAC8D3-5365-4D3C-BC13-5A9C5D4AAB10@hubject.net> Message-ID: <7786AB78-F0B9-465D-89A9-3B59249582B1@hubject.net> Hi Team, I?m looking for the easiest way to allow CGI Script in my web site ? > Any strait forward procedure available ? Best Regards Bernard From al-nginx at none.at Mon Oct 22 12:08:56 2018 From: al-nginx at none.at (Aleksandar Lazic) Date: Mon, 22 Oct 2018 14:08:56 +0200 Subject: NGINX with CGI on Alpine Linux In-Reply-To: <7786AB78-F0B9-465D-89A9-3B59249582B1@hubject.net> References: <43EB1F9C-98EE-4924-A597-70792283F47A@hubject.net> <53EAC8D3-5365-4D3C-BC13-5A9C5D4AAB10@hubject.net> <7786AB78-F0B9-465D-89A9-3B59249582B1@hubject.net> Message-ID: <72806a94-abd7-67ca-2c14-26262db5b905@none.at> Hi Postmaster (a.k.a. Bernard) I think this is a normal user user question not a development question so let's switch to nginx at nginx.org instead of nginx-devel at nginx.org. Am 22.10.2018 um 13:57 schrieb Postmaster: > Hi Team, > > I?m looking for the easiest way to allow CGI Script in my web site ? >> Any strait forward procedure available ? A short search shows at least two good options. https://stackoverflow.com/questions/11667489/how-to-run-cgi-scripts-on-nginx https://stackoverflow.com/questions/10252306/nginx-uwsgi-and-cgi-python-script Both recommend uwsgi, It's a robust solution. > Best Regards > > Bernard Best regards. Aleks > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > From xeioex at nginx.com Tue Oct 23 17:40:18 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 23 Oct 2018 17:40:18 +0000 Subject: [njs] Handling non-array values in Array.prototype.slice. Message-ID: details: http://hg.nginx.org/njs/rev/de0974c3e90f branches: changeset: 629:de0974c3e90f user: Dmitry Volyntsev date: Tue Oct 23 20:40:02 2018 +0300 description: Handling non-array values in Array.prototype.slice. diffstat: njs/njs_array.c | 213 +++++++++++++++++++++++++++++++++++----------- njs/njs_object_hash.h | 10 ++ njs/test/njs_unit_test.c | 84 ++++++++++++++++++ 3 files changed, 254 insertions(+), 53 deletions(-) diffs (386 lines): diff -r 8ab908b0f226 -r de0974c3e90f njs/njs_array.c --- a/njs/njs_array.c Fri Oct 19 20:55:38 2018 +0300 +++ b/njs/njs_array.c Tue Oct 23 20:40:02 2018 +0300 @@ -9,6 +9,19 @@ typedef struct { + union { + njs_continuation_t cont; + u_char padding[NJS_CONTINUATION_SIZE]; + } u; + /* + * This retval value must be aligned so the continuation is padded + * to aligned size. + */ + njs_value_t length; +} njs_array_slice_t; + + +typedef struct { njs_continuation_t cont; njs_value_t *values; uint32_t max; @@ -67,6 +80,10 @@ typedef struct { } njs_array_sort_t; +static njs_ret_t njs_array_prototype_slice_continuation(njs_vm_t *vm, + njs_value_t *args, nxt_uint_t nargs, njs_index_t unused); +static njs_ret_t njs_array_prototype_slice_copy(njs_vm_t *vm, + njs_value_t *this, int64_t start, int64_t length); static njs_ret_t njs_array_prototype_to_string_continuation(njs_vm_t *vm, njs_value_t *args, nxt_uint_t nargs, njs_index_t retval); static njs_ret_t njs_array_prototype_join_continuation(njs_vm_t *vm, @@ -434,58 +451,95 @@ static njs_ret_t njs_array_prototype_slice(njs_vm_t *vm, njs_value_t *args, nxt_uint_t nargs, njs_index_t unused) { - int32_t start, end, length; - uint32_t n; - njs_array_t *array; - njs_value_t *value; - - start = 0; - length = 0; - - if (njs_is_array(&args[0])) { - length = args[0].data.u.array->length; - - if (nargs > 1) { - start = args[1].data.u.number; - - if (start < 0) { - start += length; - - if (start < 0) { - start = 0; - } - } - - if (start >= length) { + njs_ret_t ret; + njs_array_slice_t *slice; + + static const njs_value_t njs_string_length = njs_string("length"); + + slice = njs_vm_continuation(vm); + slice->u.cont.function = njs_array_prototype_slice_continuation; + + ret = njs_value_property(vm, &args[0], &njs_string_length, &slice->length); + if (nxt_slow_path(ret == NXT_ERROR || ret == NJS_TRAP)) { + return ret; + } + + return njs_array_prototype_slice_continuation(vm, args, nargs, unused); +} + + +static njs_ret_t +njs_array_prototype_slice_continuation(njs_vm_t *vm, njs_value_t *args, + nxt_uint_t nargs, njs_index_t unused) +{ + int64_t start, end, length; + njs_array_slice_t *slice; + + slice = njs_vm_continuation(vm); + + if (nxt_slow_path(!njs_is_primitive(&slice->length))) { + njs_vm_trap_value(vm, &slice->length); + return njs_trap(vm, NJS_TRAP_NUMBER_ARG); + } + + start = (int32_t) njs_primitive_value_to_integer(njs_arg(args, nargs, 1)); + length = njs_primitive_value_to_integer(&slice->length); + + if (start < 0) { + start += length; + + if (start < 0) { + start = 0; + } + } + + if (start >= length) { + start = 0; + length = 0; + + } else { + if (!njs_is_void(njs_arg(args, nargs, 2))) { + end = (int32_t) njs_primitive_value_to_integer(&args[2]); + + } else { + end = length; + } + + if (end < 0) { + end += length; + } + + if (length >= end) { + length = end - start; + + if (length < 0) { start = 0; length = 0; - - } else { - end = length; - - if (nargs > 2) { - end = args[2].data.u.number; - - if (end < 0) { - end += length; - } - } - - if (length >= end) { - length = end - start; - - if (length < 0) { - start = 0; - length = 0; - } - - } else { - length -= start; - } } + + } else { + length -= start; } } + return njs_array_prototype_slice_copy(vm, &args[0], start, length); +} + + +static njs_ret_t +njs_array_prototype_slice_copy(njs_vm_t *vm, njs_value_t *this, + int64_t start, int64_t length) +{ + size_t size; + u_char *dst; + uint32_t n, len; + njs_ret_t ret; + njs_array_t *array; + njs_value_t *value, name; + const u_char *src, *end; + njs_slice_prop_t string_slice; + njs_string_prop_t string; + array = njs_array_alloc(vm, length, NJS_ARRAY_SPARE); if (nxt_slow_path(array == NULL)) { return NXT_ERROR; @@ -496,14 +550,66 @@ njs_array_prototype_slice(njs_vm_t *vm, vm->retval.data.truth = 1; if (length != 0) { - value = args[0].data.u.array->start; n = 0; - do { - /* GC: retain long string and object in values[start]. */ - array->start[n++] = value[start++]; - length--; - } while (length != 0); + if (nxt_fast_path(njs_is_array(this))) { + value = this->data.u.array->start; + + do { + /* GC: retain long string and object in values[start]. */ + array->start[n++] = value[start++]; + length--; + } while (length != 0); + + } else if (njs_is_string(this) || this->type == NJS_OBJECT_STRING) { + + if (this->type == NJS_OBJECT_STRING) { + this = &this->data.u.object_value->value; + } + + string_slice.start = start; + string_slice.length = length; + string_slice.string_length = njs_string_prop(&string, this); + + njs_string_slice_string_prop(&string, &string, &string_slice); + + src = string.start; + end = src + string.size; + + if (string.length == 0) { + /* Byte string. */ + len = 0; + + } else { + /* UTF-8 or ASCII string. */ + len = 1; + } + + do { + value = &array->start[n++]; + dst = njs_string_short_start(value); + dst = nxt_utf8_copy(dst, &src, end); + size = dst - njs_string_short_start(value); + njs_string_short_set(value, size, len); + + length--; + } while (length != 0); + + } else if (njs_is_object(this)) { + + do { + njs_uint32_to_string(&name, start++); + + value = &array->start[n++]; + ret = njs_value_property(vm, this, &name, value); + + if (ret != NXT_OK) { + *value = njs_value_invalid; + } + + length--; + } while (length != 0); + } } return NXT_OK; @@ -2101,7 +2207,8 @@ static const njs_object_prop_t njs_arra { .type = NJS_METHOD, .name = njs_string("slice"), - .value = njs_native_function(njs_array_prototype_slice, 0, + .value = njs_native_function(njs_array_prototype_slice, + njs_continuation_size(njs_array_slice_t), NJS_OBJECT_ARG, NJS_INTEGER_ARG, NJS_INTEGER_ARG), }, diff -r 8ab908b0f226 -r de0974c3e90f njs/njs_object_hash.h --- a/njs/njs_object_hash.h Fri Oct 19 20:55:38 2018 +0300 +++ b/njs/njs_object_hash.h Tue Oct 23 20:40:02 2018 +0300 @@ -108,6 +108,16 @@ 'j'), 'o'), 'i'), 'n') +#define NJS_LENGTH_HASH \ + nxt_djb_hash_add( \ + nxt_djb_hash_add( \ + nxt_djb_hash_add( \ + nxt_djb_hash_add( \ + nxt_djb_hash_add( \ + nxt_djb_hash_add(NXT_DJB_HASH_INIT, \ + 'l'), 'e'), 'n'), 'g'), 't'), 'h') + + #define NJS_NAME_HASH \ nxt_djb_hash_add( \ nxt_djb_hash_add( \ diff -r 8ab908b0f226 -r de0974c3e90f njs/test/njs_unit_test.c --- a/njs/test/njs_unit_test.c Fri Oct 19 20:55:38 2018 +0300 +++ b/njs/test/njs_unit_test.c Tue Oct 23 20:40:02 2018 +0300 @@ -3052,12 +3052,93 @@ static njs_unit_test_t njs_test[] = { nxt_string("Array.prototype.slice(1,2)"), nxt_string("") }, + { nxt_string("Array.prototype.slice.call(undefined)"), + nxt_string("TypeError: cannot convert void to object") }, + + { nxt_string("Array.prototype.slice.call(1)"), + nxt_string("") }, + + { nxt_string("Array.prototype.slice.call(false)"), + nxt_string("") }, + + { nxt_string("Array.prototype.slice.call({'0':'a', '1':'b', length:1})"), + nxt_string("a") }, + + { nxt_string("Array.prototype.slice.call({'0':'a', '1':'b', length:2})"), + nxt_string("a,b") }, + + { nxt_string("Array.prototype.slice.call({'0':'a', '1':'b', length:4})"), + nxt_string("a,b,,") }, + + { nxt_string("Array.prototype.slice.call({'0':'a', '1':'b', length:2}, 1)"), + nxt_string("b") }, + + { nxt_string("Array.prototype.slice.call({'0':'a', '1':'b', length:2}, 1, 2)"), + nxt_string("b") }, + + { nxt_string("Array.prototype.slice.call({length:'2'})"), + nxt_string(",") }, + + { nxt_string("njs.dump(Array.prototype.slice.call({length: 3, 1: undefined }))"), + nxt_string("[,undefined,]") }, + + { nxt_string("Array.prototype.slice.call({length:new Number(3)})"), + nxt_string(",,") }, + + { nxt_string("Array.prototype.slice.call({length: { valueOf: function() { return 2; } }})"), + nxt_string(",") }, + + { nxt_string("Array.prototype.slice.call({ length: Object.create(null) })"), + nxt_string("TypeError: Cannot convert object to primitive value") }, + + { nxt_string("Array.prototype.slice.call({length:-1})"), + nxt_string("MemoryError") }, + + { nxt_string("Array.prototype.slice.call('??Z?')"), + nxt_string("?,?,Z,?") }, + + { nxt_string("Array.prototype.slice.call('??Z?', 1)"), + nxt_string("?,Z,?") }, + + { nxt_string("Array.prototype.slice.call('??Z?', 2)"), + nxt_string("Z,?") }, + + { nxt_string("Array.prototype.slice.call('??Z?', 3)"), + nxt_string("?") }, + + { nxt_string("Array.prototype.slice.call('??Z?', 4)"), + nxt_string("") }, + + { nxt_string("Array.prototype.slice.call('??Z?', 0, 1)"), + nxt_string("?") }, + + { nxt_string("Array.prototype.slice.call('??Z?', 1, 2)"), + nxt_string("?") }, + + { nxt_string("Array.prototype.slice.call('??Z?').length"), + nxt_string("4") }, + + { nxt_string("Array.prototype.slice.call('??Z?')[1].length"), + nxt_string("1") }, + + { nxt_string("Array.prototype.slice.call(new String('??Z?'))"), + nxt_string("?,?,Z,?") }, + { nxt_string("Array.prototype.pop()"), nxt_string("undefined") }, { nxt_string("Array.prototype.shift()"), nxt_string("undefined") }, + { nxt_string("[0,1].slice()"), + nxt_string("0,1") }, + + { nxt_string("[0,1].slice(undefined)"), + nxt_string("0,1") }, + + { nxt_string("[0,1].slice(undefined, undefined)"), + nxt_string("0,1") }, + { nxt_string("[0,1,2,3,4].slice(1,4)"), nxt_string("1,2,3") }, @@ -9667,6 +9748,9 @@ static njs_unit_test_t njs_test[] = { nxt_string("njs.dump({a:1, b:[1,,2,{c:new Boolean(1)}]})"), nxt_string("{a:1,b:[1,,2,{c:[Boolean: true]}]}") }, + { nxt_string("njs.dump(Array.prototype.slice.call({'1':'b', length:2}))"), + nxt_string("[,'b']") }, + { nxt_string("njs.dump($r.props)"), nxt_string("{a:{type:\"property\",props:[\"getter\"]},b:{type:\"property\",props:[\"getter\"]}}") }, From xeioex at nginx.com Tue Oct 23 17:40:18 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 23 Oct 2018 17:40:18 +0000 Subject: [njs] Added arguments object. Message-ID: details: http://hg.nginx.org/njs/rev/72539416c466 branches: changeset: 630:72539416c466 user: Dmitry Volyntsev date: Tue Oct 23 20:40:08 2018 +0300 description: Added arguments object. This closes #19 issue on Github. diffstat: njs/njs_builtin.c | 23 +++++++++++ njs/njs_disassembler.c | 2 + njs/njs_function.c | 89 +++++++++++++++++++++++++++++++++++++++++++++ njs/njs_function.h | 12 +++++- njs/njs_generator.c | 30 +++++++++++++++ njs/njs_lexer_keyword.c | 1 + njs/njs_parser.c | 39 +++++++++++++++++++ njs/njs_parser.h | 3 + njs/njs_parser_expression.c | 15 ++++++- njs/njs_vm.c | 23 +++++++++++ njs/njs_vm.h | 9 ++++ njs/test/njs_unit_test.c | 68 ++++++++++++++++++++++++++++++++++ 12 files changed, 310 insertions(+), 4 deletions(-) diffs (542 lines): diff -r de0974c3e90f -r 72539416c466 njs/njs_builtin.c --- a/njs/njs_builtin.c Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_builtin.c Tue Oct 23 20:40:08 2018 +0300 @@ -127,6 +127,22 @@ const njs_function_init_t njs_native_fu }; +const njs_object_prop_t njs_arguments_object_properties[] = +{ + { + .type = NJS_PROPERTY_HANDLER, + .name = njs_string("caller"), + .value = njs_prop_handler(njs_function_arguments_thrower), + }, + + { + .type = NJS_PROPERTY_HANDLER, + .name = njs_string("callee"), + .value = njs_prop_handler(njs_function_arguments_thrower), + }, +}; + + static njs_ret_t njs_prototype_function(njs_vm_t *vm, njs_value_t *args, nxt_uint_t nargs, njs_index_t unused) @@ -236,6 +252,13 @@ njs_builtin_objects_create(njs_vm_t *vm) return NXT_ERROR; } + ret = njs_object_hash_create(vm, &vm->shared->arguments_object_hash, + njs_arguments_object_properties, + nxt_nitems(njs_arguments_object_properties)); + if (nxt_slow_path(ret != NXT_OK)) { + return NXT_ERROR; + } + objects = vm->shared->objects; for (i = NJS_OBJECT_THIS; i < NJS_OBJECT_MAX; i++) { diff -r de0974c3e90f -r 72539416c466 njs/njs_disassembler.c --- a/njs/njs_disassembler.c Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_disassembler.c Tue Oct 23 20:40:08 2018 +0300 @@ -24,6 +24,8 @@ static njs_code_name_t code_names[] = { nxt_string("OBJECT ") }, { njs_vmcode_function, sizeof(njs_vmcode_function_t), nxt_string("FUNCTION ") }, + { njs_vmcode_arguments, sizeof(njs_vmcode_arguments_t), + nxt_string("ARGUMENTS ") }, { njs_vmcode_regexp, sizeof(njs_vmcode_regexp_t), nxt_string("REGEXP ") }, { njs_vmcode_object_copy, sizeof(njs_vmcode_object_copy_t), diff -r de0974c3e90f -r 72539416c466 njs/njs_function.c --- a/njs/njs_function.c Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_function.c Tue Oct 23 20:40:08 2018 +0300 @@ -95,6 +95,87 @@ njs_function_value_copy(njs_vm_t *vm, nj } +/* + * ES5.1, 10.6: CreateArgumentsObject. + */ +njs_ret_t +njs_function_arguments_object_init(njs_vm_t *vm, njs_native_frame_t *frame) +{ + nxt_int_t ret; + nxt_uint_t nargs, n; + njs_value_t value; + njs_object_t *arguments; + njs_object_prop_t *prop; + nxt_lvlhsh_query_t lhq; + + static const njs_value_t njs_string_length = njs_string("length"); + + arguments = njs_object_alloc(vm); + if (nxt_slow_path(arguments == NULL)) { + return NXT_ERROR; + } + + arguments->shared_hash = vm->shared->arguments_object_hash; + + nargs = frame->nargs; + + njs_value_number_set(&value, nargs); + + prop = njs_object_prop_alloc(vm, &njs_string_length, &value, 1); + if (nxt_slow_path(prop == NULL)) { + return NXT_ERROR; + } + + prop->enumerable = 0; + + lhq.value = prop; + lhq.key_hash = NJS_LENGTH_HASH; + njs_string_get(&prop->name, &lhq.key); + + lhq.replace = 0; + lhq.pool = vm->mem_cache_pool; + lhq.proto = &njs_object_hash_proto; + + ret = nxt_lvlhsh_insert(&arguments->hash, &lhq); + if (nxt_slow_path(ret != NXT_OK)) { + njs_internal_error(vm, "lvlhsh insert failed"); + return NXT_ERROR; + } + + for (n = 0; n < nargs; n++) { + njs_uint32_to_string(&value, n); + + prop = njs_object_prop_alloc(vm, &value, &frame->arguments[n + 1], 1); + if (nxt_slow_path(prop == NULL)) { + return NXT_ERROR; + } + + lhq.value = prop; + njs_string_get(&prop->name, &lhq.key); + lhq.key_hash = nxt_djb_hash(lhq.key.start, lhq.key.length); + + ret = nxt_lvlhsh_insert(&arguments->hash, &lhq); + if (nxt_slow_path(ret != NXT_OK)) { + njs_internal_error(vm, "lvlhsh insert failed"); + return NXT_ERROR; + } + } + + frame->arguments_object = arguments; + + return NXT_OK; +} + + +njs_ret_t +njs_function_arguments_thrower(njs_vm_t *vm, njs_value_t *value, + njs_value_t *setval, njs_value_t *retval) +{ + njs_type_error(vm, "'caller', 'callee' properties may not be accessed"); + return NXT_ERROR; +} + + njs_ret_t njs_function_native_frame(njs_vm_t *vm, njs_function_t *function, const njs_value_t *this, njs_value_t *args, nxt_uint_t nargs, @@ -315,6 +396,7 @@ nxt_noinline njs_ret_t njs_function_call(njs_vm_t *vm, njs_index_t retval, size_t advance) { size_t size; + njs_ret_t ret; nxt_uint_t n, nesting; njs_frame_t *frame; njs_value_t *value; @@ -393,6 +475,13 @@ njs_function_call(njs_vm_t *vm, njs_inde vm->scopes[NJS_SCOPE_CLOSURE + n] = &closure->u.values; } + if (lambda->arguments_object) { + ret = njs_function_arguments_object_init(vm, &frame->native); + if (nxt_slow_path(ret != NXT_OK)) { + return NXT_ERROR; + } + } + vm->active_frame = frame; return NJS_APPLIED; diff -r de0974c3e90f -r 72539416c466 njs/njs_function.h --- a/njs/njs_function.h Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_function.h Tue Oct 23 20:40:08 2018 +0300 @@ -25,10 +25,12 @@ struct njs_function_lambda_s { uint32_t closure_size; /* Function nesting level. */ - uint8_t nesting; /* 4 bits */ + uint8_t nesting; /* 4 bits */ /* Function internal block closures levels. */ - uint8_t block_closures; /* 4 bits */ + uint8_t block_closures; /* 4 bits */ + + uint8_t arguments_object; /* 1 bit */ /* Initial values of local scope. */ njs_value_t *local_scope; @@ -102,7 +104,9 @@ struct njs_native_frame_s { njs_function_t *function; njs_native_frame_t *previous; + njs_value_t *arguments; + njs_object_t *arguments_object; njs_exception_t exception; @@ -147,6 +151,10 @@ struct njs_frame_s { njs_function_t *njs_function_alloc(njs_vm_t *vm); njs_function_t *njs_function_value_copy(njs_vm_t *vm, njs_value_t *value); njs_native_frame_t *njs_function_frame_alloc(njs_vm_t *vm, size_t size); +njs_ret_t njs_function_arguments_object_init(njs_vm_t *vm, + njs_native_frame_t *frame); +njs_ret_t njs_function_arguments_thrower(njs_vm_t *vm, njs_value_t *value, + njs_value_t *setval, njs_value_t *retval); njs_ret_t njs_function_prototype_create(njs_vm_t *vm, njs_value_t *value, njs_value_t *setval, njs_value_t *retval); njs_value_t *njs_function_property_prototype_create(njs_vm_t *vm, diff -r de0974c3e90f -r 72539416c466 njs/njs_generator.c --- a/njs/njs_generator.c Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_generator.c Tue Oct 23 20:40:08 2018 +0300 @@ -14,6 +14,8 @@ static nxt_int_t njs_generate_name(njs_v njs_parser_node_t *node); static nxt_int_t njs_generate_builtin_object(njs_vm_t *vm, njs_parser_t *parser, njs_parser_node_t *node); +static nxt_int_t njs_generate_arguments_object(njs_vm_t *vm, + njs_parser_t *parser, njs_parser_node_t *node); static nxt_int_t njs_generate_variable(njs_vm_t *vm, njs_parser_t *parser, njs_parser_node_t *node); static nxt_int_t njs_generate_var_statement(njs_vm_t *vm, njs_parser_t *parser, @@ -308,6 +310,9 @@ njs_generator(njs_vm_t *vm, njs_parser_t case NJS_TOKEN_CLEAR_TIMEOUT: return njs_generate_builtin_object(vm, parser, node); + case NJS_TOKEN_ARGUMENTS: + return njs_generate_arguments_object(vm, parser, node); + case NJS_TOKEN_FUNCTION: return njs_generate_function_declaration(vm, parser, node); @@ -396,6 +401,29 @@ njs_generate_builtin_object(njs_vm_t *vm static nxt_int_t +njs_generate_arguments_object(njs_vm_t *vm, njs_parser_t *parser, + njs_parser_node_t *node) +{ + njs_vmcode_arguments_t *gen; + + parser->arguments_object = 1; + + node->index = njs_generator_object_dest_index(vm, parser, node); + if (nxt_slow_path(node->index == NJS_INDEX_ERROR)) { + return NXT_ERROR; + } + + njs_generate_code(parser, njs_vmcode_arguments_t, gen); + gen->code.operation = njs_vmcode_arguments; + gen->code.operands = NJS_VMCODE_1OPERAND; + gen->code.retval = NJS_VMCODE_RETVAL; + gen->retval = node->index; + + return NXT_OK; +} + + +static nxt_int_t njs_generate_variable(njs_vm_t *vm, njs_parser_t *parser, njs_parser_node_t *node) { @@ -2010,6 +2038,7 @@ njs_generate_function_scope(njs_vm_t *vm parser->code_size += node->scope->argument_closures * sizeof(njs_vmcode_move_t); + parser->arguments_object = 0; ret = njs_generate_scope(vm, parser, node); if (nxt_fast_path(ret == NXT_OK)) { @@ -2023,6 +2052,7 @@ njs_generate_function_scope(njs_vm_t *vm lambda->nesting = node->scope->nesting; lambda->closure_size = size; + lambda->arguments_object = parser->arguments_object; lambda->local_size = parser->scope_size; lambda->local_scope = parser->local_scope; diff -r de0974c3e90f -r 72539416c466 njs/njs_lexer_keyword.c --- a/njs/njs_lexer_keyword.c Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_lexer_keyword.c Tue Oct 23 20:40:08 2018 +0300 @@ -53,6 +53,7 @@ static const njs_keyword_t njs_keywords /* Builtin objects. */ { nxt_string("this"), NJS_TOKEN_THIS, 0 }, + { nxt_string("arguments"), NJS_TOKEN_ARGUMENTS, 0 }, { nxt_string("njs"), NJS_TOKEN_NJS, 0 }, { nxt_string("Math"), NJS_TOKEN_MATH, 0 }, { nxt_string("JSON"), NJS_TOKEN_JSON, 0 }, diff -r de0974c3e90f -r 72539416c466 njs/njs_parser.c --- a/njs/njs_parser.c Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_parser.c Tue Oct 23 20:40:08 2018 +0300 @@ -455,6 +455,13 @@ njs_parser_function_declaration(njs_vm_t } if (token != NJS_TOKEN_NAME) { + if (token == NJS_TOKEN_ARGUMENTS || token == NJS_TOKEN_EVAL) { + njs_parser_syntax_error(vm, parser, "Identifier \"%.*s\" " + "is forbidden in function declaration", + (int) parser->lexer->text.length, + parser->lexer->text.start); + } + return NJS_TOKEN_ILLEGAL; } @@ -821,6 +828,13 @@ njs_parser_var_statement(njs_vm_t *vm, n } if (token != NJS_TOKEN_NAME) { + if (token == NJS_TOKEN_ARGUMENTS || token == NJS_TOKEN_EVAL) { + njs_parser_syntax_error(vm, parser, "Identifier \"%.*s\" " + "is forbidden in var declaration", + (int) parser->lexer->text.length, + parser->lexer->text.start); + } + return NJS_TOKEN_ILLEGAL; } @@ -1306,6 +1320,13 @@ njs_parser_for_var_statement(njs_vm_t *v } if (token != NJS_TOKEN_NAME) { + if (token == NJS_TOKEN_ARGUMENTS || token == NJS_TOKEN_EVAL) { + njs_parser_syntax_error(vm, parser, "Identifier \"%.*s\" " + "is forbidden in for-in var declaration", + (int) parser->lexer->text.length, + parser->lexer->text.start); + } + return NJS_TOKEN_ILLEGAL; } @@ -1973,6 +1994,24 @@ njs_parser_terminal(njs_vm_t *vm, njs_pa case NJS_TOKEN_JSON: return njs_parser_builtin_object(vm, parser, node); + case NJS_TOKEN_ARGUMENTS: + nxt_thread_log_debug("JS: arguments"); + + if (parser->scope->type <= NJS_SCOPE_GLOBAL) { + njs_parser_syntax_error(vm, parser, "\"%.*s\" object " + "in global scope", + (int) parser->lexer->text.length, + parser->lexer->text.start); + + return NJS_TOKEN_ILLEGAL; + } + + node->token = NJS_TOKEN_ARGUMENTS; + + parser->code_size += sizeof(njs_vmcode_arguments_t); + + break; + case NJS_TOKEN_OBJECT_CONSTRUCTOR: node->index = NJS_INDEX_OBJECT; break; diff -r de0974c3e90f -r 72539416c466 njs/njs_parser.h --- a/njs/njs_parser.h Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_parser.h Tue Oct 23 20:40:08 2018 +0300 @@ -161,6 +161,7 @@ typedef enum { NJS_TOKEN_THROW, NJS_TOKEN_THIS, + NJS_TOKEN_ARGUMENTS, #define NJS_TOKEN_FIRST_OBJECT NJS_TOKEN_GLOBAL_THIS @@ -346,6 +347,8 @@ struct njs_parser_s { u_char *code_end; njs_parser_t *parent; + + nxt_uint_t arguments_object; }; diff -r de0974c3e90f -r 72539416c466 njs/njs_parser_expression.c --- a/njs/njs_parser_expression.c Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_parser_expression.c Tue Oct 23 20:40:08 2018 +0300 @@ -414,8 +414,19 @@ njs_parser_assignment_expression(njs_vm_ } if (!njs_parser_is_lvalue(parser->node)) { - njs_parser_ref_error(vm, parser, - "Invalid left-hand side in assignment"); + token = parser->node->token; + + if (token == NJS_TOKEN_ARGUMENTS || token == NJS_TOKEN_EVAL) { + njs_parser_syntax_error(vm, parser, "Identifier \"%s\" " + "is forbidden as left-hand in assignment", + (token == NJS_TOKEN_EVAL) ? "eval" + : "arguments"); + + } else { + njs_parser_ref_error(vm, parser, + "Invalid left-hand side in assignment"); + } + return NJS_TOKEN_ILLEGAL; } diff -r de0974c3e90f -r 72539416c466 njs/njs_vm.c --- a/njs/njs_vm.c Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_vm.c Tue Oct 23 20:40:08 2018 +0300 @@ -413,6 +413,29 @@ njs_vmcode_function(njs_vm_t *vm, njs_va njs_ret_t +njs_vmcode_arguments(njs_vm_t *vm, njs_value_t *invld1, njs_value_t *invld2) +{ + njs_ret_t ret; + njs_frame_t *frame; + + frame = (njs_frame_t *) vm->active_frame; + + if (frame->native.arguments_object == NULL) { + ret = njs_function_arguments_object_init(vm, &frame->native); + if (nxt_slow_path(ret != NXT_OK)) { + return NXT_ERROR; + } + } + + vm->retval.data.u.object = frame->native.arguments_object; + vm->retval.type = NJS_OBJECT; + vm->retval.data.truth = 1; + + return sizeof(njs_vmcode_arguments_t); +} + + +njs_ret_t njs_vmcode_regexp(njs_vm_t *vm, njs_value_t *invld1, njs_value_t *invld2) { njs_regexp_t *regexp; diff -r de0974c3e90f -r 72539416c466 njs/njs_vm.h --- a/njs/njs_vm.h Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/njs_vm.h Tue Oct 23 20:40:08 2018 +0300 @@ -635,6 +635,12 @@ typedef struct { typedef struct { njs_vmcode_t code; njs_index_t retval; +} njs_vmcode_arguments_t; + + +typedef struct { + njs_vmcode_t code; + njs_index_t retval; uintptr_t length; } njs_vmcode_array_t; @@ -1084,6 +1090,7 @@ struct njs_vm_shared_s { nxt_lvlhsh_t keywords_hash; nxt_lvlhsh_t values_hash; nxt_lvlhsh_t function_prototype_hash; + nxt_lvlhsh_t arguments_object_hash; njs_object_t objects[NJS_OBJECT_MAX]; njs_function_t functions[NJS_FUNCTION_MAX]; @@ -1110,6 +1117,8 @@ njs_ret_t njs_vmcode_array(njs_vm_t *vm, njs_value_t *inlvd2); njs_ret_t njs_vmcode_function(njs_vm_t *vm, njs_value_t *inlvd1, njs_value_t *invld2); +njs_ret_t njs_vmcode_arguments(njs_vm_t *vm, njs_value_t *inlvd1, + njs_value_t *invld2); njs_ret_t njs_vmcode_regexp(njs_vm_t *vm, njs_value_t *inlvd1, njs_value_t *invld2); njs_ret_t njs_vmcode_object_copy(njs_vm_t *vm, njs_value_t *value, diff -r de0974c3e90f -r 72539416c466 njs/test/njs_unit_test.c --- a/njs/test/njs_unit_test.c Tue Oct 23 20:40:02 2018 +0300 +++ b/njs/test/njs_unit_test.c Tue Oct 23 20:40:08 2018 +0300 @@ -5713,6 +5713,74 @@ static njs_unit_test_t njs_test[] = "var b = a(); b(2)"), nxt_string("3") }, + /* arguments object. */ + + { nxt_string("var arguments"), + nxt_string("SyntaxError: Identifier \"arguments\" is forbidden in var declaration in 1") }, + + { nxt_string("for (var arguments in []) {}"), + nxt_string("SyntaxError: Identifier \"arguments\" is forbidden in for-in var declaration in 1") }, + + { nxt_string("function arguments(){}"), + nxt_string("SyntaxError: Identifier \"arguments\" is forbidden in function declaration in 1") }, + + { nxt_string("(function () {arguments = [];})"), + nxt_string("SyntaxError: Identifier \"arguments\" is forbidden as left-hand in assignment in 1") }, + + { nxt_string("(function(){return arguments[0];})(1,2,3)"), + nxt_string("1") }, + + { nxt_string("(function(){return arguments[2];})(1,2,3)"), + nxt_string("3") }, + + { nxt_string("(function(){return arguments[3];})(1,2,3)"), + nxt_string("undefined") }, + + { nxt_string("(function(a,b,c){return a;})(1,2,3)"), + nxt_string("1") }, + + { nxt_string("(function(a,b,c){arguments[0] = 4; return a;})(1,2,3)"), + nxt_string("1") }, + + { nxt_string("(function(a,b,c){a = 4; return arguments[0];})(1,2,3)"), + nxt_string("1") }, + + { nxt_string("function check(v) {if (v == false) {throw TypeError('Too few arguments')}}; " + "function f() {check(arguments.length > 1); return 1}; f()"), + nxt_string("TypeError: Too few arguments") }, + + { nxt_string("function check(v) {if (v == false) {throw TypeError('Too few arguments')}}; " + "function f() {check(arguments.length > 1); return 1}; f(1,2)"), + nxt_string("1") }, + + { nxt_string("(function(a,b){delete arguments[0]; return arguments[0]})(1,1)"), + nxt_string("undefined") }, + + { nxt_string("(function(){return arguments.length;})()"), + nxt_string("0") }, + + { nxt_string("(function(){return arguments.length;})(1,2,3)"), + nxt_string("3") }, + + { nxt_string("(function(){arguments.length = 1; return arguments.length;})(1,2,3)"), + nxt_string("1") }, + + { nxt_string("(function(){return arguments.callee;})()"), + nxt_string("TypeError: 'caller', 'callee' properties may not be accessed") }, + + { nxt_string("(function(){return arguments.caller;})()"), + nxt_string("TypeError: 'caller', 'callee' properties may not be accessed") }, + + { nxt_string("function sum() { var args = Array.prototype.slice.call(arguments); " + "return args.reduce(function(prev, curr) {return prev + curr})};" + "[sum(1), sum(1,2), sum(1,2,3), sum(1,2,3,4)]"), + nxt_string("1,3,6,10") }, + + { nxt_string("function concat(sep) { var args = Array.prototype.slice.call(arguments, 1); " + "return args.join(sep)};" + "[concat('.',1,2,3), concat('+',1,2,3,4)]"), + nxt_string("1.2.3,1+2+3+4") }, + /* Scopes. */ { nxt_string("function f(x) { a = x } var a; f(5); a"), From mdounin at mdounin.ru Tue Oct 23 19:19:27 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 23 Oct 2018 19:19:27 +0000 Subject: [nginx] SSL: explicitly set maximum version (ticket #1654). Message-ID: details: http://hg.nginx.org/nginx/rev/ed8738b1c7c4 branches: changeset: 7372:ed8738b1c7c4 user: Maxim Dounin date: Tue Oct 23 22:11:48 2018 +0300 description: SSL: explicitly set maximum version (ticket #1654). With maximum version explicitly set, TLSv1.3 will not be unexpectedly enabled if nginx compiled with OpenSSL 1.1.0 (without TLSv1.3 support) will be run with OpenSSL 1.1.1 (with TLSv1.3 support). diffstat: src/event/ngx_event_openssl.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diffs (15 lines): diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -345,6 +345,11 @@ ngx_ssl_create(ngx_ssl_t *ssl, ngx_uint_ } #endif +#ifdef SSL_CTX_set_min_proto_version + SSL_CTX_set_min_proto_version(ssl->ctx, 0); + SSL_CTX_set_max_proto_version(ssl->ctx, TLS1_2_VERSION); +#endif + #ifdef TLS1_3_VERSION SSL_CTX_set_min_proto_version(ssl->ctx, 0); SSL_CTX_set_max_proto_version(ssl->ctx, TLS1_3_VERSION); From mdounin at mdounin.ru Tue Oct 23 19:44:38 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 23 Oct 2018 19:44:38 +0000 Subject: [nginx] A minor code clean for macro ngx_event_get_conf in ngx_event.h. Message-ID: details: http://hg.nginx.org/nginx/rev/874d47ac871a branches: changeset: 7373:874d47ac871a user: chronolaw date: Fri Oct 19 13:50:36 2018 +0800 description: A minor code clean for macro ngx_event_get_conf in ngx_event.h. diffstat: src/event/ngx_event.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/src/event/ngx_event.h b/src/event/ngx_event.h --- a/src/event/ngx_event.h +++ b/src/event/ngx_event.h @@ -499,7 +499,7 @@ extern ngx_module_t ngx_event_ #define ngx_event_get_conf(conf_ctx, module) \ - (*(ngx_get_conf(conf_ctx, ngx_events_module))) [module.ctx_index]; + (*(ngx_get_conf(conf_ctx, ngx_events_module))) [module.ctx_index] From peters at yandex.ru Sun Oct 28 14:58:45 2018 From: peters at yandex.ru (Peter Shchuchkin) Date: Sun, 28 Oct 2018 17:58:45 +0300 Subject: [PATCH] Allow using nodelay=N semantics in limit_req configuration Message-ID: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> # HG changeset patch # User Peter Shchuchkin # Date 1540737213 -10800 # Sun Oct 28 17:33:33 2018 +0300 # Node ID 70c0d476999d9b893c644606624134248ac7abad # Parent 874d47ac871a4b62fbe0ff5d469a8ad7bc5a4160 Allow using nodelay=N semantics in limit_req configuration This allows to use reasonably low limits while not forcing delay on normal users. In addition to standard "burst=A nodelay" form now the following form of limit_req may be used: burst=A nodelay=B, where B must be 0 <= B <= burst burst=A nodelay=0 means the same as just "burst=A" burst=A nodelay=A means the same as "burst=A nodelay" burst=A nodelay=B means the first B requests matching limit_zone variable will not be delayed and next requests will be delayed. The delay is calculated against excess over B thus B+1 request will have effective excess=1. When using limit_req with nodelay the responsibility of limiting requests speed is on the client. If client don't want or can't correctly limit its speed it will get 503 errors and you will get numerous messages in error and access logs. When using limit_req without nodelay, then every request that comes faster then expected speed will be delayed. This is not always convenient. Sometimes you want to allow normal client to make a bunch of requests as fast as possible while still having configured limit on request speed. Using this new semantics you can get the best from two worlds. Specifying burst=A nodelay=B you allow clients to make B requests without any delay (and without warnings in error log). If B requests are exceeded by client then further requests are delayed, effectively limiting client rps to desired limit without returning 503 errors. Thus one can ensure maximum speed for clients with expected usage profile and limit all other clients to certain speed without errors. diff -r 874d47ac871a -r 70c0d476999d src/http/modules/ngx_http_limit_req_module.c --- a/src/http/modules/ngx_http_limit_req_module.c Fri Oct 19 13:50:36 2018 +0800 +++ b/src/http/modules/ngx_http_limit_req_module.c Sun Oct 28 17:33:33 2018 +0300 @@ -499,12 +499,11 @@ excess = *ep; - if (excess == 0 || (*limit)->nodelay) { + if (excess == 0 || ((ngx_uint_t) excess < (*limit)->nodelay)) { max_delay = 0; - } else { ctx = (*limit)->shm_zone->data; - max_delay = excess * 1000 / ctx->rate; + max_delay = (excess - (*limit)->nodelay) * 1000 / ctx->rate; } while (n--) { @@ -544,11 +543,16 @@ ctx->node = NULL; - if (limits[n].nodelay) { + /* + * Delay when: + * excess > 0, nodelay = 0 + * excess > 0, nodelay > 0, excess >= nodelay + */ + if ((ngx_uint_t) excess < limits[n].nodelay) { continue; } - delay = excess * 1000 / ctx->rate; + delay = (excess - limits[n].nodelay) * 1000 / ctx->rate; if (delay > max_delay) { max_delay = delay; @@ -875,7 +879,7 @@ { ngx_http_limit_req_conf_t *lrcf = conf; - ngx_int_t burst; + ngx_int_t burst, nodelay_val; ngx_str_t *value, s; ngx_uint_t i, nodelay; ngx_shm_zone_t *shm_zone; @@ -885,6 +889,7 @@ shm_zone = NULL; burst = 0; + nodelay_val = -1; nodelay = 0; for (i = 1; i < cf->args->nelts; i++) { @@ -915,7 +920,20 @@ continue; } - if (ngx_strcmp(value[i].data, "nodelay") == 0) { + if (ngx_strncmp(value[i].data, "nodelay=", 8) == 0) { + + nodelay_val = ngx_atoi(value[i].data + 8, value[i].len - 8); + nodelay = 1; + if (nodelay_val < 0) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid nodelay \"%V\" - " + "must not be less then 0", + &value[i]); + return NGX_CONF_ERROR; + } + + continue; + } else if (ngx_strcmp(value[i].data, "nodelay") == 0) { nodelay = 1; continue; } @@ -925,6 +943,23 @@ return NGX_CONF_ERROR; } + if (nodelay) { + /* nodelay without explicit value */ + if (nodelay_val < 0) { + nodelay_val = burst; + } + + if (nodelay_val > burst) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid nodelay value: %i - " + "must not be greater then burst: %i", + nodelay_val, burst); + return NGX_CONF_ERROR; + } + } else { + nodelay_val = 0; + } + if (shm_zone == NULL) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "\"%V\" must have \"zone\" parameter", @@ -956,7 +991,7 @@ limit->shm_zone = shm_zone; limit->burst = burst * 1000; - limit->nodelay = nodelay; + limit->nodelay = nodelay_val * 1000; return NGX_CONF_OK; } From peters at yandex.ru Sun Oct 28 15:01:39 2018 From: peters at yandex.ru (Peter Shchuchkin) Date: Sun, 28 Oct 2018 18:01:39 +0300 Subject: [PATCH] Allow using nodelay=N semantics in limit_req configuration In-Reply-To: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> References: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> Message-ID: <70c0d476999d9b893c64.1540738899@petr-desktop.superjob.local> # HG changeset patch # User Peter Shchuchkin # Date 1540737213 -10800 # Sun Oct 28 17:33:33 2018 +0300 # Node ID 70c0d476999d9b893c644606624134248ac7abad # Parent 874d47ac871a4b62fbe0ff5d469a8ad7bc5a4160 Allow using nodelay=N semantics in limit_req configuration This allows to use reasonably low limits while not forcing delay on normal users. In addition to standard "burst=A nodelay" form now the following form of limit_req may be used: burst=A nodelay=B, where B must be 0 <= B <= burst burst=A nodelay=0 means the same as just "burst=A" burst=A nodelay=A means the same as "burst=A nodelay" burst=A nodelay=B means the first B requests matching limit_zone variable will not be delayed and next requests will be delayed. The delay is calculated against excess over B thus B+1 request will have effective excess=1. When using limit_req with nodelay the responsibility of limiting requests speed is on the client. If client don't want or can't correctly limit its speed it will get 503 errors and you will get numerous messages in error and access logs. When using limit_req without nodelay, then every request that comes faster then expected speed will be delayed. This is not always convenient. Sometimes you want to allow normal client to make a bunch of requests as fast as possible while still having configured limit on request speed. Using this new semantics you can get the best from two worlds. Specifying burst=A nodelay=B you allow clients to make B requests without any delay (and without warnings in error log). If B requests are exceeded by client then further requests are delayed, effectively limiting client rps to desired limit without returning 503 errors. Thus one can ensure maximum speed for clients with expected usage profile and limit all other clients to certain speed without errors. diff -r 874d47ac871a -r 70c0d476999d src/http/modules/ngx_http_limit_req_module.c --- a/src/http/modules/ngx_http_limit_req_module.c Fri Oct 19 13:50:36 2018 +0800 +++ b/src/http/modules/ngx_http_limit_req_module.c Sun Oct 28 17:33:33 2018 +0300 @@ -499,12 +499,11 @@ excess = *ep; - if (excess == 0 || (*limit)->nodelay) { + if (excess == 0 || ((ngx_uint_t) excess < (*limit)->nodelay)) { max_delay = 0; - } else { ctx = (*limit)->shm_zone->data; - max_delay = excess * 1000 / ctx->rate; + max_delay = (excess - (*limit)->nodelay) * 1000 / ctx->rate; } while (n--) { @@ -544,11 +543,16 @@ ctx->node = NULL; - if (limits[n].nodelay) { + /* + * Delay when: + * excess > 0, nodelay = 0 + * excess > 0, nodelay > 0, excess >= nodelay + */ + if ((ngx_uint_t) excess < limits[n].nodelay) { continue; } - delay = excess * 1000 / ctx->rate; + delay = (excess - limits[n].nodelay) * 1000 / ctx->rate; if (delay > max_delay) { max_delay = delay; @@ -875,7 +879,7 @@ { ngx_http_limit_req_conf_t *lrcf = conf; - ngx_int_t burst; + ngx_int_t burst, nodelay_val; ngx_str_t *value, s; ngx_uint_t i, nodelay; ngx_shm_zone_t *shm_zone; @@ -885,6 +889,7 @@ shm_zone = NULL; burst = 0; + nodelay_val = -1; nodelay = 0; for (i = 1; i < cf->args->nelts; i++) { @@ -915,7 +920,20 @@ continue; } - if (ngx_strcmp(value[i].data, "nodelay") == 0) { + if (ngx_strncmp(value[i].data, "nodelay=", 8) == 0) { + + nodelay_val = ngx_atoi(value[i].data + 8, value[i].len - 8); + nodelay = 1; + if (nodelay_val < 0) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid nodelay \"%V\" - " + "must not be less then 0", + &value[i]); + return NGX_CONF_ERROR; + } + + continue; + } else if (ngx_strcmp(value[i].data, "nodelay") == 0) { nodelay = 1; continue; } @@ -925,6 +943,23 @@ return NGX_CONF_ERROR; } + if (nodelay) { + /* nodelay without explicit value */ + if (nodelay_val < 0) { + nodelay_val = burst; + } + + if (nodelay_val > burst) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid nodelay value: %i - " + "must not be greater then burst: %i", + nodelay_val, burst); + return NGX_CONF_ERROR; + } + } else { + nodelay_val = 0; + } + if (shm_zone == NULL) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "\"%V\" must have \"zone\" parameter", @@ -956,7 +991,7 @@ limit->shm_zone = shm_zone; limit->burst = burst * 1000; - limit->nodelay = nodelay; + limit->nodelay = nodelay_val * 1000; return NGX_CONF_OK; } From peters at yandex.ru Sun Oct 28 15:04:24 2018 From: peters at yandex.ru (Peter Shchuchkin) Date: Sun, 28 Oct 2018 18:04:24 +0300 Subject: [PATCH] Tests: testing limit_req with "burst=A nodelay=B" configuration In-Reply-To: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> References: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> Message-ID: <9c629d98af7d8191f8ec.1540739064@petr-desktop.superjob.local> # HG changeset patch # User Peter Shchuchkin # Date 1540715681 -10800 # Sun Oct 28 11:34:41 2018 +0300 # Node ID 9c629d98af7d8191f8ec5664b3d6ad33cb870d5c # Parent 73a9504ae6fd61f75a3983784248b9cc0bd6e6c7 Tests: testing limit_req with "burst=A nodelay=B" configuration diff -r 73a9504ae6fd -r 9c629d98af7d limit_req.t --- a/limit_req.t Fri Oct 19 18:49:45 2018 +0300 +++ b/limit_req.t Sun Oct 28 11:34:41 2018 +0300 @@ -21,7 +21,7 @@ select STDERR; $| = 1; select STDOUT; $| = 1; -my $t = Test::Nginx->new()->has(qw/http limit_req/)->plan(6); +my $t = Test::Nginx->new()->has(qw/http limit_req/)->plan(8); $t->write_file_expand('nginx.conf', <<'EOF'); @@ -38,6 +38,7 @@ limit_req_zone $binary_remote_addr zone=one:1m rate=2r/s; limit_req_zone $binary_remote_addr zone=long:1m rate=2r/s; limit_req_zone $binary_remote_addr zone=fast:1m rate=1000r/s; + limit_req_zone $binary_remote_addr zone=mixed:1m rate=10r/s; server { listen 127.0.0.1:8080; @@ -56,6 +57,12 @@ location /fast { limit_req zone=fast burst=1; } + location /mixed { + limit_req zone=mixed burst=2 nodelay=1; + } + location /mixed-pass { + limit_req zone=mixed burst=2 nodelay; + } } } @@ -64,6 +71,8 @@ $t->write_file('test1.html', 'XtestX'); $t->write_file('long.html', "1234567890\n" x (1 << 16)); $t->write_file('fast.html', 'XtestX'); +$t->write_file('mixed.html', 'XtestX'); +$t->write_file('mixed-pass.html', 'XtestX'); $t->run(); ############################################################################### @@ -94,4 +103,20 @@ select undef, undef, undef, 0.1; like(http_get('/fast.html'), qr/^HTTP\/1.. 200 /m, 'negative excess'); +# make sure requests are delayed when 0 < nodelay < burst +# this test should fail when using nodelay without =number + +http_get('/mixed.html'); +http_get('/mixed.html'); +http_get('/mixed.html'); +like(http_get('/mixed.html'), qr/^HTTP\/1.. 200 /m, 'mixed request is not rejected'); + +# make sure it is possible to partially fill up excess through /mixed.html and get last request rejected +# this test should fail when using "burst=N" without nodelay + +http_get('/mixed.html'); +http_get('/mixed.html'); +http_get('/mixed-pass.html'); +like(http_get('/mixed-pass.html'), qr/^HTTP\/1.. 503 /m, 'mixed request is rejected'); + ############################################################################### From peters at yandex.ru Sun Oct 28 15:13:18 2018 From: peters at yandex.ru (=?utf-8?B?0KnRg9GH0LrQuNC9INCf0LXRgtGA?=) Date: Sun, 28 Oct 2018 18:13:18 +0300 Subject: [PATCH] Allow using nodelay=N semantics in limit_req configuration References: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> Message-ID: <17950261540739598@myt6-67cd1de25d8a.qloud-c.yandex.net> Hello! Sorry, I've duplicated this message. Additionally I've tried to send with hg email changes to nginx-test in the same thread, but I'm not sure I've succeed and instead new thread was created with changes in tests project. Please review this changes, I hope this may be really useful for some real world use cases. I'm using nodelay_val = -1 when parsing config to allow nodelay=0 and thus to be able to distinguish between value not set and value set to 0. Also there is casting of excess variable to (ngx_uint_t) in several because I've seen such example in the same file when comparing burst to excess. Thank you! From mdounin at mdounin.ru Mon Oct 29 12:58:43 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 29 Oct 2018 15:58:43 +0300 Subject: [PATCH] Allow using nodelay=N semantics in limit_req configuration In-Reply-To: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> References: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> Message-ID: <20181029125842.GN56558@mdounin.ru> Hello! On Sun, Oct 28, 2018 at 05:58:45PM +0300, Peter Shchuchkin wrote: > # HG changeset patch > # User Peter Shchuchkin > # Date 1540737213 -10800 > # Sun Oct 28 17:33:33 2018 +0300 > # Node ID 70c0d476999d9b893c644606624134248ac7abad > # Parent 874d47ac871a4b62fbe0ff5d469a8ad7bc5a4160 > Allow using nodelay=N semantics in limit_req configuration > > This allows to use reasonably low limits while not forcing delay on normal > users. > > In addition to standard "burst=A nodelay" form now the following form of > limit_req may be used: > burst=A nodelay=B, where B must be 0 <= B <= burst > > burst=A nodelay=0 means the same as just "burst=A" > burst=A nodelay=A means the same as "burst=A nodelay" > burst=A nodelay=B means the first B requests matching limit_zone variable will > not be delayed and next requests will be delayed. The delay is calculated > against excess over B thus B+1 request will have effective excess=1. > > When using limit_req with nodelay the responsibility of limiting requests speed > is on the client. > If client don't want or can't correctly limit its speed it will get 503 errors > and you will get numerous messages in error and access logs. > When using limit_req without nodelay, then every request that comes faster then > expected speed will be delayed. This is not always convenient. Sometimes you > want to allow normal client to make a bunch of requests as fast as possible > while still having configured limit on request speed. > > Using this new semantics you can get the best from two worlds. Specifying > burst=A nodelay=B you allow clients to make B requests without any delay (and > without warnings in error log). If B requests are exceeded by client then > further requests are delayed, effectively limiting client rps to desired limit > without returning 503 errors. Thus one can ensure maximum speed for clients > with expected usage profile and limit all other clients to certain speed > without errors. [...] I've posted a patch for this a while ago, see here: http://mailman.nginx.org/pipermail/nginx-devel/2016-April/008136.html Please take a look if it works for you, and/or you have any comments. The most noticeable difference I see is how delay is calculated, and probably your variant with only counting excess above the nodelay level is more logical. -- Maxim Dounin http://mdounin.ru/ From peters at yandex.ru Mon Oct 29 14:14:13 2018 From: peters at yandex.ru (=?utf-8?B?0KnRg9GH0LrQuNC9INCf0LXRgtGA?=) Date: Mon, 29 Oct 2018 17:14:13 +0300 Subject: [PATCH] Allow using nodelay=N semantics in limit_req configuration In-Reply-To: <20181029125842.GN56558@mdounin.ru> References: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> <20181029125842.GN56558@mdounin.ru> Message-ID: <1489321540822453@iva5-84bc572481fe.qloud-c.yandex.net> An HTML attachment was scrubbed... URL: From xeioex at nginx.com Mon Oct 29 16:49:05 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Mon, 29 Oct 2018 16:49:05 +0000 Subject: [njs] Stream: marking incoming chain as processed. Message-ID: details: http://hg.nginx.org/njs/rev/647de74fd2c7 branches: changeset: 631:647de74fd2c7 user: Dmitry Volyntsev date: Mon Oct 29 17:36:43 2018 +0300 description: Stream: marking incoming chain as processed. Previously, the buffers of proxy module were not reclaimed. This resulted in stopping the processing of new data chunks from a client after proxy_buffer_size bytes were received. diffstat: nginx/ngx_stream_js_module.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diffs (12 lines): diff -r 72539416c466 -r 647de74fd2c7 nginx/ngx_stream_js_module.c --- a/nginx/ngx_stream_js_module.c Tue Oct 23 20:40:08 2018 +0300 +++ b/nginx/ngx_stream_js_module.c Mon Oct 29 17:36:43 2018 +0300 @@ -570,6 +570,8 @@ ngx_stream_js_body_filter(ngx_stream_ses goto exception; } + ctx->buf->pos = ctx->buf->last; + } else { cl = ngx_alloc_chain_link(c->pool); if (cl == NULL) { From vbart at nginx.com Mon Oct 29 18:27:39 2018 From: vbart at nginx.com (Valentin Bartenev) Date: Mon, 29 Oct 2018 18:27:39 +0000 Subject: [njs] Non-integer fractions. Message-ID: details: http://hg.nginx.org/njs/rev/0677cbeee41a branches: changeset: 632:0677cbeee41a user: Valentin Bartenev date: Mon Oct 29 21:06:24 2018 +0300 description: Non-integer fractions. diffstat: njs/njs_lexer.c | 17 ++++++++++++++--- njs/test/njs_unit_test.c | 20 +++++++++++++++++++- 2 files changed, 33 insertions(+), 4 deletions(-) diffs (123 lines): diff -r 647de74fd2c7 -r 0677cbeee41a njs/njs_lexer.c --- a/njs/njs_lexer.c Mon Oct 29 17:36:43 2018 +0300 +++ b/njs/njs_lexer.c Mon Oct 29 21:06:24 2018 +0300 @@ -290,7 +290,7 @@ njs_lexer_token(njs_lexer_t *lexer) static njs_token_t njs_lexer_next_token(njs_lexer_t *lexer) { - u_char c; + u_char c, *p; nxt_uint_t n; njs_token_t token; const njs_lexer_multi_t *multi; @@ -315,6 +315,16 @@ njs_lexer_next_token(njs_lexer_t *lexer) case NJS_TOKEN_SINGLE_QUOTE: return njs_lexer_string(lexer, c); + case NJS_TOKEN_DOT: + p = lexer->start; + + if (p == lexer->end || njs_tokens[*p] != NJS_TOKEN_DIGIT) { + lexer->text.length = p - lexer->text.start; + return NJS_TOKEN_DOT; + } + + /* Fall through. */ + case NJS_TOKEN_DIGIT: return njs_lexer_number(lexer, c); @@ -405,7 +415,6 @@ njs_lexer_next_token(njs_lexer_t *lexer) case NJS_TOKEN_CLOSE_BRACKET: case NJS_TOKEN_OPEN_BRACE: case NJS_TOKEN_CLOSE_BRACE: - case NJS_TOKEN_DOT: case NJS_TOKEN_COMMA: case NJS_TOKEN_COLON: case NJS_TOKEN_SEMICOLON: @@ -527,6 +536,8 @@ njs_lexer_number(njs_lexer_t *lexer, u_c { const u_char *p; + lexer->text.start = lexer->start - 1; + p = lexer->start; if (c == '0' && p != lexer->end) { @@ -594,6 +605,7 @@ njs_lexer_number(njs_lexer_t *lexer, u_c done: lexer->start = (u_char *) p; + lexer->text.length = p - lexer->text.start; return NJS_TOKEN_NUMBER; @@ -603,7 +615,6 @@ illegal_trailer: illegal_token: - lexer->text.start = lexer->start - 1; lexer->text.length = p - lexer->text.start; return NJS_TOKEN_ILLEGAL; diff -r 647de74fd2c7 -r 0677cbeee41a njs/test/njs_unit_test.c --- a/njs/test/njs_unit_test.c Mon Oct 29 17:36:43 2018 +0300 +++ b/njs/test/njs_unit_test.c Mon Oct 29 21:06:24 2018 +0300 @@ -101,9 +101,18 @@ static njs_unit_test_t njs_test[] = { nxt_string("-0"), nxt_string("-0") }, + { nxt_string(".0"), + nxt_string("0") }, + { nxt_string("0.1"), nxt_string("0.1") }, + { nxt_string(".9"), + nxt_string("0.9") }, + + { nxt_string("-.01"), + nxt_string("-0.01") }, + { nxt_string("0.000001"), nxt_string("0.000001") }, @@ -140,6 +149,9 @@ static njs_unit_test_t njs_test[] = { nxt_string("+1\n"), nxt_string("1") }, + { nxt_string("."), + nxt_string("SyntaxError: Unexpected token \".\" in 1") }, + /* Octal Numbers. */ { nxt_string("0o0"), @@ -264,6 +276,9 @@ static njs_unit_test_t njs_test[] = { nxt_string("1.1e+01"), nxt_string("11") }, + { nxt_string("-.01e-01"), + nxt_string("-0.001") }, + { nxt_string("1e9"), nxt_string("1000000000") }, @@ -297,6 +312,9 @@ static njs_unit_test_t njs_test[] = { nxt_string("1eZ"), nxt_string("SyntaxError: Unexpected token \"eZ\" in 1") }, + { nxt_string(".e1"), + nxt_string("SyntaxError: Unexpected token \".\" in 1") }, + /* Indexes. */ { nxt_string("var a = []; a[-1] = 2; a[-1] == a['-1']"), @@ -4039,7 +4057,7 @@ static njs_unit_test_t njs_test[] = nxt_string("NaN") }, { nxt_string("var a = 'abcdef'; a.3"), - nxt_string("SyntaxError: Unexpected token \"3\" in 1") }, + nxt_string("SyntaxError: Unexpected token \".3\" in 1") }, { nxt_string("'abcdef'[3]"), nxt_string("d") }, From xeioex at nginx.com Tue Oct 30 11:20:53 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 30 Oct 2018 11:20:53 +0000 Subject: [njs] Added tag 0.2.5 for changeset 3315f6aa6000 Message-ID: details: http://hg.nginx.org/njs/rev/0095a845ad02 branches: changeset: 634:0095a845ad02 user: Dmitry Volyntsev date: Tue Oct 30 14:20:31 2018 +0300 description: Added tag 0.2.5 for changeset 3315f6aa6000 diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff -r 3315f6aa6000 -r 0095a845ad02 .hgtags --- a/.hgtags Tue Oct 30 14:16:39 2018 +0300 +++ b/.hgtags Tue Oct 30 14:20:31 2018 +0300 @@ -19,3 +19,4 @@ 2a0a59728b5f197379ca62a334a516fabd4ea392 4adbb035caa39dae58611a061d78bc974652231e 0.2.2 e83f41520613987542472275d49633a9aa5955d1 0.2.3 3e6c38f64bdbc53e783813541559034ed6890aee 0.2.4 +3315f6aa6000ce6d2dbc74c73660becf4178a549 0.2.5 From xeioex at nginx.com Tue Oct 30 11:20:53 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 30 Oct 2018 11:20:53 +0000 Subject: [njs] Version 0.2.5. Message-ID: details: http://hg.nginx.org/njs/rev/3315f6aa6000 branches: changeset: 633:3315f6aa6000 user: Dmitry Volyntsev date: Tue Oct 30 14:16:39 2018 +0300 description: Version 0.2.5. diffstat: CHANGES | 41 +++++++++++++++++++++++++++++++++++++++++ 1 files changed, 41 insertions(+), 0 deletions(-) diffs (48 lines): diff -r 0677cbeee41a -r 3315f6aa6000 CHANGES --- a/CHANGES Mon Oct 29 21:06:24 2018 +0300 +++ b/CHANGES Tue Oct 30 14:16:39 2018 +0300 @@ -1,3 +1,44 @@ + +Changes with njs 0.2.5 30 Oct 2018 + + nginx modules: + + *) Bugfix: fixed counting pending events in stream module. + + *) Bugfix: fixed s.off() in stream module. + + *) Bugfix: fixed processing of data chunks in js_filter in stream module. + + *) Bugfix: fixed http status and contentType getter in http module. + + *) Bugfix: fixed http response and parent getters in http module. + + Core: + + *) Feature: arguments object support. + + *) Feature: non-integer fractions support. + + *) Improvement: handling non-array values in Array.prototype.slice(). + + *) Bugfix: fixed Array.prototype.length setter. + + *) Bugfix: fixed njs_array_alloc() for length > 2**31. + + *) Bugfix: handling int overflow in njs_array_alloc() on 32bit + archs. + + *) Bugfix: fixed code size mismatch error message. + + *) Bugfix: fixed delete operator in a loop. + + *) Bugfix: fixed Object.getOwnPropertyDescriptor() for complex + object (inherited from Array and string values). + + *) Bugfix: fixed Object.prototype.hasOwnProperty() for non-object + properties. + + *) Bugfix: miscellaneous additional bugs have been fixed. Changes with njs 0.2.4 18 Aug 2018 From xeioex at nginx.com Tue Oct 30 11:23:15 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 30 Oct 2018 11:23:15 +0000 Subject: [njs] Version bump. Message-ID: details: http://hg.nginx.org/njs/rev/f0e55f6cb91d branches: changeset: 635:f0e55f6cb91d user: Dmitry Volyntsev date: Tue Oct 30 14:22:20 2018 +0300 description: Version bump. diffstat: njs/njs.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 0095a845ad02 -r f0e55f6cb91d njs/njs.h --- a/njs/njs.h Tue Oct 30 14:20:31 2018 +0300 +++ b/njs/njs.h Tue Oct 30 14:22:20 2018 +0300 @@ -11,7 +11,7 @@ #include -#define NJS_VERSION "0.2.5" +#define NJS_VERSION "0.2.6" #include From mdounin at mdounin.ru Tue Oct 30 15:45:39 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 30 Oct 2018 18:45:39 +0300 Subject: [PATCH] Allow using nodelay=N semantics in limit_req configuration In-Reply-To: <1489321540822453@iva5-84bc572481fe.qloud-c.yandex.net> References: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> <20181029125842.GN56558@mdounin.ru> <1489321540822453@iva5-84bc572481fe.qloud-c.yandex.net> Message-ID: <20181030154539.GT56558@mdounin.ru> Hello! On Mon, Oct 29, 2018 at 05:14:13PM +0300, ?????? ???? wrote: > 2016 year patch seems pretty same to me :) Interesting that I also > made my patch at 2016, but we have started to use it in production a > while ago (when upgraded to nginx 1.15.4 where you've fixed limit_req > issue) > > I think calculating excess above nodelay should be more expected > behavior. Think about setting large burst and large nodelay, for ex. > burst=10000 nodelay=9999. Even at relatively fast limit rate (10r/s) > this will therefore lead to very large delay which I think is not > expected nor by user neither by server administrator. I think it is > more intuitive to think of it as you give nodelay requests for for free > and additional requests with limited speed, instead of you give nodelay > requests as a credit and then force client to wait all that "credited" > time when he wants to make additional requests. Yes, I agree. As previously said, only taking into account the excess above the nodelay level looks more logical. > Also I think it doesn't have sense to allow nodelay to be larger than > burst because in any case requests that are more then burst will be > rejected. May be it's better to just warn user and implicitly reduce > nodelay value to burst value in that case instead of error. There is no difference between using the nodelay level as is or reducing it to burst level. Nodelay level above the burst limit means that all requests will not be delayed - they are either allowed, or rejected due to burst limit. Moreover, the default value as in my patch actualy uses very large nodelay value to implement "nodelay" without an explicitly specified value. While limiting users from inadvertently using nodelay= larger than burst= may help to diagnose some configuration issues, I don't think it worth the effort. > So I would suggest to use behavior that calculates delay based on > excess above nodelay. Here is an updated patch, which calculates delay based on the specified level. Also, the parameter was renamed to "delay=", as this name looks more logical. # HG changeset patch # User Maxim Dounin # Date 1540913077 -10800 # Tue Oct 30 18:24:37 2018 +0300 # Node ID 9697dadddc63e793501152b61c08519edab72f0f # Parent 874d47ac871a4b62fbe0ff5d469a8ad7bc5a4160 Limit req: "delay=" parameter. This parameter specifies an additional "soft" burst limit at which requests become delayed (but not yet rejected as it happens if "burst=" limit is exceeded). Defaults to 0, i.e., all excess requests are delayed. Originally inspired by Vladislav Shabanov (http://mailman.nginx.org/pipermail/nginx-devel/2016-April/008126.html). Further improved based on a patch by Peter Shchuchkin (http://mailman.nginx.org/pipermail/nginx-devel/2018-October/011522.html). diff --git a/src/http/modules/ngx_http_limit_req_module.c b/src/http/modules/ngx_http_limit_req_module.c --- a/src/http/modules/ngx_http_limit_req_module.c +++ b/src/http/modules/ngx_http_limit_req_module.c @@ -44,7 +44,7 @@ typedef struct { ngx_shm_zone_t *shm_zone; /* integer value, 1 corresponds to 0.001 r/s */ ngx_uint_t burst; - ngx_uint_t nodelay; /* unsigned nodelay:1 */ + ngx_uint_t delay; } ngx_http_limit_req_limit_t; @@ -499,12 +499,12 @@ ngx_http_limit_req_account(ngx_http_limi excess = *ep; - if (excess == 0 || (*limit)->nodelay) { + if ((ngx_uint_t) excess <= (*limit)->delay) { max_delay = 0; } else { ctx = (*limit)->shm_zone->data; - max_delay = excess * 1000 / ctx->rate; + max_delay = (excess - (*limit)->delay) * 1000 / ctx->rate; } while (n--) { @@ -544,11 +544,11 @@ ngx_http_limit_req_account(ngx_http_limi ctx->node = NULL; - if (limits[n].nodelay) { + if ((ngx_uint_t) excess <= limits[n].delay) { continue; } - delay = excess * 1000 / ctx->rate; + delay = (excess - limits[n].delay) * 1000 / ctx->rate; if (delay > max_delay) { max_delay = delay; @@ -875,9 +875,9 @@ ngx_http_limit_req(ngx_conf_t *cf, ngx_c { ngx_http_limit_req_conf_t *lrcf = conf; - ngx_int_t burst; + ngx_int_t burst, delay; ngx_str_t *value, s; - ngx_uint_t i, nodelay; + ngx_uint_t i; ngx_shm_zone_t *shm_zone; ngx_http_limit_req_limit_t *limit, *limits; @@ -885,7 +885,7 @@ ngx_http_limit_req(ngx_conf_t *cf, ngx_c shm_zone = NULL; burst = 0; - nodelay = 0; + delay = 0; for (i = 1; i < cf->args->nelts; i++) { @@ -915,8 +915,20 @@ ngx_http_limit_req(ngx_conf_t *cf, ngx_c continue; } + if (ngx_strncmp(value[i].data, "delay=", 6) == 0) { + + delay = ngx_atoi(value[i].data + 6, value[i].len - 6); + if (delay <= 0) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid delay value \"%V\"", &value[i]); + return NGX_CONF_ERROR; + } + + continue; + } + if (ngx_strcmp(value[i].data, "nodelay") == 0) { - nodelay = 1; + delay = NGX_MAX_INT_T_VALUE / 1000; continue; } @@ -956,7 +968,7 @@ ngx_http_limit_req(ngx_conf_t *cf, ngx_c limit->shm_zone = shm_zone; limit->burst = burst * 1000; - limit->nodelay = nodelay; + limit->delay = delay * 1000; return NGX_CONF_OK; } -- Maxim Dounin http://mdounin.ru/ From peters at yandex.ru Wed Oct 31 07:45:29 2018 From: peters at yandex.ru (=?utf-8?B?0KnRg9GH0LrQuNC9INCf0LXRgtGA?=) Date: Wed, 31 Oct 2018 10:45:29 +0300 Subject: [PATCH] Allow using nodelay=N semantics in limit_req configuration In-Reply-To: <20181030154539.GT56558@mdounin.ru> References: <70c0d476999d9b893c64.1540738725@petr-desktop.superjob.local> <20181029125842.GN56558@mdounin.ru> <1489321540822453@iva5-84bc572481fe.qloud-c.yandex.net> <20181030154539.GT56558@mdounin.ru> Message-ID: <5852531540971929@myt3-964eae3a5b05.qloud-c.yandex.net> An HTML attachment was scrubbed... URL: From xeioex at nginx.com Wed Oct 31 13:06:16 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 31 Oct 2018 13:06:16 +0000 Subject: [njs] Fixed description for explicit_bzero script. Message-ID: details: http://hg.nginx.org/njs/rev/a3a0d5850b1a branches: changeset: 636:a3a0d5850b1a user: Dmitry Volyntsev date: Wed Oct 31 16:06:06 2018 +0300 description: Fixed description for explicit_bzero script. diffstat: nxt/auto/explicit_bzero | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r f0e55f6cb91d -r a3a0d5850b1a nxt/auto/explicit_bzero --- a/nxt/auto/explicit_bzero Tue Oct 30 14:22:20 2018 +0300 +++ b/nxt/auto/explicit_bzero Wed Oct 31 16:06:06 2018 +0300 @@ -3,7 +3,7 @@ # Copyright (C) NGINX, Inc. -# Linux (glibc and musl from 1.1.20), OpenBSD, FreeBSD and NetBSD. +# Linux (glibc and musl from 1.1.20), OpenBSD and FreeBSD. nxt_feature="explicit_bzero()" nxt_feature_name=NXT_HAVE_EXPLICIT_BZERO From mdounin at mdounin.ru Wed Oct 31 14:42:18 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 31 Oct 2018 14:42:18 +0000 Subject: [nginx] Cache: fixed minimum cache keys zone size limit. Message-ID: details: http://hg.nginx.org/nginx/rev/de50fa05fbeb branches: changeset: 7374:de50fa05fbeb user: Maxim Dounin date: Wed Oct 31 16:49:39 2018 +0300 description: Cache: fixed minimum cache keys zone size limit. Size of a shared memory zones must be at least two pages - one page for slab allocator internal data, and another page for actual allocations. Using 8192 instead is wrong, as there are systems with page sizes other than 4096. Note well that two pages is usually too low as well. In particular, cache is likely to use two allocations of different sizes for global structures, and at least four pages will be needed to properly allocate cache nodes. Except in a few very special cases, with keys zone of just two pages nginx won't be able to start. Other uses of shared memory impose a limit of 8 pages, which provides some room for global allocations. This patch doesn't try to address this though. Inspired by ticket #1665. diffstat: src/http/ngx_http_file_cache.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c --- a/src/http/ngx_http_file_cache.c +++ b/src/http/ngx_http_file_cache.c @@ -2427,7 +2427,7 @@ ngx_http_file_cache_set_slot(ngx_conf_t s.data = p; size = ngx_parse_size(&s); - if (size > 8191) { + if (size >= (ssize_t) (2 * ngx_pagesize)) { continue; } } From mdounin at mdounin.ru Wed Oct 31 14:42:19 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 31 Oct 2018 14:42:19 +0000 Subject: [nginx] Cache: improved keys zone size error reporting. Message-ID: details: http://hg.nginx.org/nginx/rev/bddacdaaec9e branches: changeset: 7375:bddacdaaec9e user: Maxim Dounin date: Wed Oct 31 16:49:40 2018 +0300 description: Cache: improved keys zone size error reporting. After this change, too small keys zones are explicitly reported as such, much like in the other modules which use shared memory. diffstat: src/http/ngx_http_file_cache.c | 39 ++++++++++++++++++++++++--------------- 1 files changed, 24 insertions(+), 15 deletions(-) diffs (51 lines): diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c --- a/src/http/ngx_http_file_cache.c +++ b/src/http/ngx_http_file_cache.c @@ -2418,23 +2418,32 @@ ngx_http_file_cache_set_slot(ngx_conf_t p = (u_char *) ngx_strchr(name.data, ':'); - if (p) { - name.len = p - name.data; - - p++; - - s.len = value[i].data + value[i].len - p; - s.data = p; - - size = ngx_parse_size(&s); - if (size >= (ssize_t) (2 * ngx_pagesize)) { - continue; - } + if (p == NULL) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid keys zone size \"%V\"", &value[i]); + return NGX_CONF_ERROR; } - ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, - "invalid keys zone size \"%V\"", &value[i]); - return NGX_CONF_ERROR; + name.len = p - name.data; + + s.data = p + 1; + s.len = value[i].data + value[i].len - s.data; + + size = ngx_parse_size(&s); + + if (size == NGX_ERROR) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid keys zone size \"%V\"", &value[i]); + return NGX_CONF_ERROR; + } + + if (size < (ssize_t) (2 * ngx_pagesize)) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "keys zone \"%V\" is too small", &value[i]); + return NGX_CONF_ERROR; + } + + continue; } if (ngx_strncmp(value[i].data, "inactive=", 9) == 0) {