From ryo.hirafuji at link-u.co.jp Mon Feb 3 07:55:06 2020 From: ryo.hirafuji at link-u.co.jp (=?UTF-8?B?5bmz6Jek54eO?=) Date: Mon, 3 Feb 2020 16:55:06 +0900 Subject: [PATCH] MIME: Added avif/avifs types. Message-ID: Hello, I added new mime-types, "image/avif" and "image/avif-sequence". Please take a look, # HG changeset patch # User Ryo Hirafuji # Date 1580715197 -32400 # Mon Feb 03 16:33:17 2020 +0900 # Branch mime-avif-avifs # Node ID 28c8371bb45be38623f8ac0e958d14b55600198a # Parent b8a512c6466c3b2f77876edf14061c5d97e6159f MIME: Added avif/avifs types. "avif" is an image format based on AV1, royalty-free video coding format. "avifs" is an image sequence format, that contains multiple avif images like animated gifs. [Reference] avif/avifs: https://aomediacodec.github.io/av1-avif/ AV1: https://aomediacodec.github.io/av1-spec/av1-spec.pdf diff -r b8a512c6466c -r 28c8371bb45b conf/mime.types --- a/conf/mime.types Tue Jan 21 16:39:42 2020 +0300 +++ b/conf/mime.types Mon Feb 03 16:33:17 2020 +0900 @@ -20,6 +20,8 @@ image/tiff tif tiff; image/vnd.wap.wbmp wbmp; image/webp webp; + image/avif avif; + image/avif-sequence avifs; image/x-icon ico; image/x-jng jng; image/x-ms-bmp bmp; -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Feb 3 17:20:41 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 3 Feb 2020 20:20:41 +0300 Subject: [PATCH] MIME: Added avif/avifs types. In-Reply-To: References: Message-ID: <20200203172041.GY12894@mdounin.ru> Hello! On Mon, Feb 03, 2020 at 04:55:06PM +0900, ??? wrote: > Hello, I added new mime-types, "image/avif" and "image/avif-sequence". > Please take a look, > > # HG changeset patch > # User Ryo Hirafuji > # Date 1580715197 -32400 > # Mon Feb 03 16:33:17 2020 +0900 > # Branch mime-avif-avifs > # Node ID 28c8371bb45be38623f8ac0e958d14b55600198a > # Parent b8a512c6466c3b2f77876edf14061c5d97e6159f > MIME: Added avif/avifs types. > > "avif" is an image format based on AV1, royalty-free video coding format. > "avifs" is an image sequence format, that contains multiple avif images > like animated gifs. > > [Reference] > avif/avifs: https://aomediacodec.github.io/av1-avif/ > AV1: https://aomediacodec.github.io/av1-spec/av1-spec.pdf > > diff -r b8a512c6466c -r 28c8371bb45b conf/mime.types > --- a/conf/mime.types Tue Jan 21 16:39:42 2020 +0300 > +++ b/conf/mime.types Mon Feb 03 16:33:17 2020 +0900 > @@ -20,6 +20,8 @@ > image/tiff tif tiff; > image/vnd.wap.wbmp wbmp; > image/webp webp; > + image/avif avif; > + image/avif-sequence avifs; > image/x-icon ico; > image/x-jng jng; > image/x-ms-bmp bmp; Thank you for the patch. Note that nginx doesn't try to maintain all the mime types known, but rather tries to supply types generally needed for a web server. Particular MIME types don't seem to be widely used, and not even registered by IANA[1]. If you want to use these types in your configuration, consider modifying mime.types locally or extending the existing MIME types list using the "types" directive (http://nginx.org/r/types). [1] https://www.iana.org/assignments/media-types -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Mon Feb 3 17:49:56 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 3 Feb 2020 20:49:56 +0300 Subject: submitting a bug report In-Reply-To: References: Message-ID: <20200203174956.GZ12894@mdounin.ru> Hello! On Fri, Jan 31, 2020 at 10:38:57AM -0500, Val Kulkov wrote: > Some futher information on the problem reported earlier. On attempting > to login to Trac, I am getting the following today: > > Oops? > Trac detected an internal error: > InvalidClientSecretsError: ('Error opening file', '', 'No such file or > directory', 2) > There was an internal error in Trac. It is recommended that you notify > your local Trac administrator with the information needed to reproduce > the issue. > > To that end, you could a ticket. > > The action that triggered the error was: > > GET: /login > TracGuide ? The Trac User and Administration Guide Trac was being moved to another server last week, and there were some issues due to this. Now it should be fine, please report if you'll see any further issues. > > On Wed, 29 Jan 2020 at 22:26, Val Kulkov wrote: > > > > The "How to report bug" page at trac.nginx.org says: > > > > "You need to authorize with Google, GitHub, StackExchange or Yandex to > > submit bug report." > > > > I have logged in to Trac with my GitHub account. I am seeing "logged > > in as Val Kulkov" in the top line of the page. But there is no way to > > submit a bug report! There is no button, no link, nothing. I can only > > see reports, but I cannot open a new ticket. > > > > Is there some problem with the Trac server [again]? Can someone check > > what is going on please? Thanks. -- Maxim Dounin http://mdounin.ru/ From val.kulkov at gmail.com Mon Feb 3 17:52:29 2020 From: val.kulkov at gmail.com (Val Kulkov) Date: Mon, 3 Feb 2020 12:52:29 -0500 Subject: submitting a bug report In-Reply-To: <20200203174956.GZ12894@mdounin.ru> References: <20200203174956.GZ12894@mdounin.ru> Message-ID: Hi Maxim, Thanks for responding. It is not presently possible to attach a file to a ticket in Trac. Please see ticket #1928[1] in Trac. [1] https://trac.nginx.org/nginx/ticket/1928 On Mon, 3 Feb 2020 at 12:50, Maxim Dounin wrote: > > Hello! > > On Fri, Jan 31, 2020 at 10:38:57AM -0500, Val Kulkov wrote: > > > Some futher information on the problem reported earlier. On attempting > > to login to Trac, I am getting the following today: > > > > Oops? > > Trac detected an internal error: > > InvalidClientSecretsError: ('Error opening file', '', 'No such file or > > directory', 2) > > There was an internal error in Trac. It is recommended that you notify > > your local Trac administrator with the information needed to reproduce > > the issue. > > > > To that end, you could a ticket. > > > > The action that triggered the error was: > > > > GET: /login > > TracGuide ? The Trac User and Administration Guide > > Trac was being moved to another server last week, and there were > some issues due to this. Now it should be fine, please report > if you'll see any further issues. > > > > > On Wed, 29 Jan 2020 at 22:26, Val Kulkov wrote: > > > > > > The "How to report bug" page at trac.nginx.org says: > > > > > > "You need to authorize with Google, GitHub, StackExchange or Yandex to > > > submit bug report." > > > > > > I have logged in to Trac with my GitHub account. I am seeing "logged > > > in as Val Kulkov" in the top line of the page. But there is no way to > > > submit a bug report! There is no button, no link, nothing. I can only > > > see reports, but I cannot open a new ticket. > > > > > > Is there some problem with the Trac server [again]? Can someone check > > > what is going on please? Thanks. > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From mdounin at mdounin.ru Mon Feb 3 18:14:12 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 3 Feb 2020 21:14:12 +0300 Subject: submitting a bug report In-Reply-To: References: <20200203174956.GZ12894@mdounin.ru> Message-ID: <20200203181412.GA12894@mdounin.ru> Hello! On Mon, Feb 03, 2020 at 12:52:29PM -0500, Val Kulkov wrote: > Thanks for responding. It is not presently possible to attach a file > to a ticket in Trac. Please see ticket #1928[1] in Trac. Yes, thanks, I've reported it, should be fixed soon. The problem itself as described in the ticket seems to be a result of incorrect crossbuild patches, I've added a comment there. -- Maxim Dounin http://mdounin.ru/ From ryo.hirafuji at link-u.co.jp Tue Feb 4 09:19:18 2020 From: ryo.hirafuji at link-u.co.jp (Ryo Hirafuji) Date: Tue, 4 Feb 2020 18:19:18 +0900 Subject: [PATCH] MIME: Added avif/avifs types. In-Reply-To: <20200203172041.GY12894@mdounin.ru> References: <20200203172041.GY12894@mdounin.ru> Message-ID: Thank you for reviewing, Dounin! Yeah, I know that image/avif or image/avif-sequence are not registered in IANA list yet, but image/webp is also not registered in IANA list, isn't it? WebP and AVIF are similar in some points. They are both based on Royalty-free Video format, VP8 for WebP[1] and AV1 (based upon VP9) for AVIF[2]. AVIF is already supported in Windows[3], and it is planned to support in Chrome[4] and Firefox[5] in the near future. So I think it is worth to add "image/avif" to mime.types, if "image/webp" is worth to be listed in mime.types. Please tell your opinion! [1] A new image format for the Web | WebP | Google Developers https://developers.google.com/speed/webp/ [2] AV1 Image File Format (AVIF) https://aomediacodec.github.io/av1-avif/ [3] AOMediaCodec/av1-avif Wiki https://github.com/AOMediaCodec/av1-avif/wiki#demuxers--players [4] 960620 - Support AVIF - An open-source project to help move the web forward. - Monorail https://bugs.chromium.org/p/chromium/issues/detail?id=960620 [5] 1443863 - (AVIF) Implement support for AV1 Still Image File Format (AVIF) https://bugzilla.mozilla.org/show_bug.cgi?id=1443863 2020?2?4?(?) 2:20 Maxim Dounin : > Hello! > > On Mon, Feb 03, 2020 at 04:55:06PM +0900, ??? wrote: > > > Hello, I added new mime-types, "image/avif" and "image/avif-sequence". > > Please take a look, > > > > # HG changeset patch > > # User Ryo Hirafuji > > # Date 1580715197 -32400 > > # Mon Feb 03 16:33:17 2020 +0900 > > # Branch mime-avif-avifs > > # Node ID 28c8371bb45be38623f8ac0e958d14b55600198a > > # Parent b8a512c6466c3b2f77876edf14061c5d97e6159f > > MIME: Added avif/avifs types. > > > > "avif" is an image format based on AV1, royalty-free video coding format. > > "avifs" is an image sequence format, that contains multiple avif images > > like animated gifs. > > > > [Reference] > > avif/avifs: https://aomediacodec.github.io/av1-avif/ > > AV1: https://aomediacodec.github.io/av1-spec/av1-spec.pdf > > > > diff -r b8a512c6466c -r 28c8371bb45b conf/mime.types > > --- a/conf/mime.types Tue Jan 21 16:39:42 2020 +0300 > > +++ b/conf/mime.types Mon Feb 03 16:33:17 2020 +0900 > > @@ -20,6 +20,8 @@ > > image/tiff tif tiff; > > image/vnd.wap.wbmp wbmp; > > image/webp webp; > > + image/avif avif; > > + image/avif-sequence avifs; > > image/x-icon ico; > > image/x-jng jng; > > image/x-ms-bmp bmp; > > Thank you for the patch. > > Note that nginx doesn't try to maintain all the mime types known, > but rather tries to supply types generally needed for a web > server. Particular MIME types don't seem to be widely used, > and not even registered by IANA[1]. > > If you want to use these types in your configuration, consider > modifying mime.types locally or extending the existing MIME types > list using the "types" directive (http://nginx.org/r/types). > > [1] https://www.iana.org/assignments/media-types > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Feb 4 12:32:35 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 4 Feb 2020 15:32:35 +0300 Subject: [PATCH] MIME: Added avif/avifs types. In-Reply-To: References: <20200203172041.GY12894@mdounin.ru> Message-ID: <20200204123235.GB12894@mdounin.ru> Hello! On Tue, Feb 04, 2020 at 06:19:18PM +0900, Ryo Hirafuji wrote: > Thank you for reviewing, Dounin! > > Yeah, I know that image/avif or image/avif-sequence are not registered in > IANA list yet, > but image/webp is also not registered in IANA list, isn't it? > > WebP and AVIF are similar in some points. > They are both based on Royalty-free Video format, VP8 for WebP[1] and AV1 > (based upon VP9) for AVIF[2]. > > AVIF is already supported in Windows[3], and it is planned to support in > Chrome[4] and Firefox[5] in the near future. > > So I think it is worth to add "image/avif" to mime.types, if "image/webp" > is worth to be listed in mime.types. > > Please tell your opinion! The major difference is that image/webp and video/webm are supported by browsers[1][2] and used in the wild, while image/avif and image/avif-sequence are not. As previously suggested, if you want to use these types in your configuration, consider modifying mime.types locally or extending the existing MIME types list using the "types" directive (http://nginx.org/r/types). [1] https://caniuse.com/#feat=webp [2] https://caniuse.com/#feat=webm -- Maxim Dounin http://mdounin.ru/ From ryo.hirafuji at link-u.co.jp Tue Feb 4 12:41:48 2020 From: ryo.hirafuji at link-u.co.jp (Ryo Hirafuji) Date: Tue, 4 Feb 2020 21:41:48 +0900 Subject: [PATCH] MIME: Added avif/avifs types. In-Reply-To: <20200204123235.GB12894@mdounin.ru> References: <20200203172041.GY12894@mdounin.ru> <20200204123235.GB12894@mdounin.ru> Message-ID: Thanks for quick response! OK, I understand. I will submit this patch again when major browsers support AVIF. 2020?2?4?(?) 21:32 Maxim Dounin : > Hello! > > On Tue, Feb 04, 2020 at 06:19:18PM +0900, Ryo Hirafuji wrote: > > > Thank you for reviewing, Dounin! > > > > Yeah, I know that image/avif or image/avif-sequence are not registered in > > IANA list yet, > > but image/webp is also not registered in IANA list, isn't it? > > > > WebP and AVIF are similar in some points. > > They are both based on Royalty-free Video format, VP8 for WebP[1] and AV1 > > (based upon VP9) for AVIF[2]. > > > > AVIF is already supported in Windows[3], and it is planned to support in > > Chrome[4] and Firefox[5] in the near future. > > > > So I think it is worth to add "image/avif" to mime.types, if "image/webp" > > is worth to be listed in mime.types. > > > > Please tell your opinion! > > The major difference is that image/webp and video/webm are > supported by browsers[1][2] and used in the wild, while image/avif > and image/avif-sequence are not. > > As previously suggested, if you want to use these types in your > configuration, consider modifying mime.types locally or extending > the existing MIME types list using the "types" directive > (http://nginx.org/r/types). > > [1] https://caniuse.com/#feat=webp > [2] https://caniuse.com/#feat=webm > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Tue Feb 4 17:35:26 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 04 Feb 2020 17:35:26 +0000 Subject: [njs] Added fast path in [[Get]] and [[Set]] for arrays and typed-arrays. Message-ID: details: https://hg.nginx.org/njs/rev/f5afb325896f branches: changeset: 1315:f5afb325896f user: Dmitry Volyntsev date: Tue Feb 04 20:35:00 2020 +0300 description: Added fast path in [[Get]] and [[Set]] for arrays and typed-arrays. diffstat: src/njs_array.c | 1 + src/njs_number.h | 12 ++++- src/njs_object.c | 3 + src/njs_typed_array.c | 1 + src/njs_value.c | 108 +++++++++++++++++++++++++++++++++++++++++++++- src/njs_value.h | 1 + src/njs_vmcode.c | 2 +- src/test/njs_unit_test.c | 4 +- 8 files changed, 123 insertions(+), 9 deletions(-) diffs (276 lines): diff -r 2b75ee955589 -r f5afb325896f src/njs_array.c --- a/src/njs_array.c Mon Jan 20 13:24:09 2020 +0300 +++ b/src/njs_array.c Tue Feb 04 20:35:00 2020 +0300 @@ -67,6 +67,7 @@ njs_array_alloc(njs_vm_t *vm, uint64_t l array->object.type = NJS_ARRAY; array->object.shared = 0; array->object.extensible = 1; + array->object.fast_array = 1; array->size = size; array->length = length; diff -r 2b75ee955589 -r f5afb325896f src/njs_number.h --- a/src/njs_number.h Mon Jan 20 13:24:09 2020 +0300 +++ b/src/njs_number.h Tue Feb 04 20:35:00 2020 +0300 @@ -30,16 +30,24 @@ njs_int_t njs_number_parse_float(njs_vm_ njs_inline njs_bool_t -njs_number_is_integer_index(double num, const njs_value_t *value) +njs_number_is_integer_index(double num) { uint32_t u32; u32 = num; - return (u32 == num && u32 != 0xffffffff) + return (u32 == num && u32 != 0xffffffff); +} + + +njs_inline njs_bool_t +njs_key_is_integer_index(double num, const njs_value_t *value) +{ + return (njs_number_is_integer_index(num)) && !(njs_is_string(value) && num == 0 && signbit(num)); } + njs_inline int64_t njs_number_to_int64(double num) { diff -r 2b75ee955589 -r f5afb325896f src/njs_object.c --- a/src/njs_object.c Mon Jan 20 13:24:09 2020 +0300 +++ b/src/njs_object.c Tue Feb 04 20:35:00 2020 +0300 @@ -51,6 +51,8 @@ njs_object_alloc(njs_vm_t *vm) object->shared = 0; object->extensible = 1; object->error_data = 0; + object->fast_array = 0; + return object; } @@ -108,6 +110,7 @@ njs_object_value_alloc(njs_vm_t *vm, con ov->object.type = njs_object_value_type(type); ov->object.shared = 0; ov->object.extensible = 1; + ov->object.fast_array = 0; index = njs_primitive_prototype_index(type); ov->object.__proto__ = &vm->prototypes[index].object; diff -r 2b75ee955589 -r f5afb325896f src/njs_typed_array.c --- a/src/njs_typed_array.c Mon Jan 20 13:24:09 2020 +0300 +++ b/src/njs_typed_array.c Tue Feb 04 20:35:00 2020 +0300 @@ -179,6 +179,7 @@ njs_typed_array_constructor(njs_vm_t *vm array->object.type = NJS_TYPED_ARRAY; array->object.shared = 0; array->object.extensible = 1; + array->object.fast_array = 1; njs_set_typed_array(&vm->retval, array); diff -r 2b75ee955589 -r f5afb325896f src/njs_value.c --- a/src/njs_value.c Mon Jan 20 13:24:09 2020 +0300 +++ b/src/njs_value.c Tue Feb 04 20:35:00 2020 +0300 @@ -549,7 +549,7 @@ njs_property_query(njs_vm_t *vm, njs_pro case NJS_STRING: if (njs_fast_path(!njs_is_null_or_undefined_or_boolean(key))) { num = njs_key_to_index(key); - if (njs_fast_path(njs_number_is_integer_index(num, key))) { + if (njs_fast_path(njs_key_is_integer_index(num, key))) { return njs_string_property_query(vm, pq, value, num); } } @@ -654,7 +654,7 @@ njs_object_property_query(njs_vm_t *vm, switch (proto->type) { case NJS_ARRAY: num = njs_key_to_index(key); - if (njs_fast_path(njs_number_is_integer_index(num, key))) { + if (njs_fast_path(njs_key_is_integer_index(num, key))) { array = (njs_array_t *) proto; return njs_array_property_query(vm, pq, array, num); } @@ -663,7 +663,7 @@ njs_object_property_query(njs_vm_t *vm, case NJS_TYPED_ARRAY: num = njs_key_to_index(key); - if (njs_fast_path(njs_number_is_integer_index(num, key))) { + if (njs_fast_path(njs_key_is_integer_index(num, key))) { tarray = (njs_typed_array_t *) proto; return njs_typed_array_property_query(vm, pq, tarray, num); } @@ -676,7 +676,7 @@ njs_object_property_query(njs_vm_t *vm, case NJS_OBJECT_STRING: num = njs_key_to_index(key); - if (njs_fast_path(njs_number_is_integer_index(num, key))) { + if (njs_fast_path(njs_key_is_integer_index(num, key))) { ov = (njs_object_value_t *) proto; ret = njs_string_property_query(vm, pq, &ov->value, num); @@ -1021,10 +1021,60 @@ njs_int_t njs_value_property(njs_vm_t *vm, njs_value_t *value, njs_value_t *key, njs_value_t *retval) { + double num; + uint32_t index; njs_int_t ret; + njs_array_t *array; njs_object_prop_t *prop; + njs_typed_array_t *tarray; njs_property_query_t pq; + if (njs_fast_path(njs_is_number(key))) { + num = njs_number(key); + + if (njs_slow_path(!njs_number_is_integer_index(num))) { + goto slow_path; + } + + index = (uint32_t) num; + + if (njs_is_typed_array(value)) { + tarray = njs_typed_array(value); + + if (njs_slow_path(index >= njs_typed_array_length(tarray))) { + goto slow_path; + } + + njs_set_number(retval, njs_typed_array_get(tarray, index)); + + return NJS_OK; + } + + if (njs_slow_path(!(njs_is_object(value) + && njs_object(value)->fast_array))) + { + goto slow_path; + } + + /* NJS_ARRAY */ + + array = njs_array(value); + + if (njs_slow_path(index >= array->length)) { + goto slow_path; + } + + *retval = array->start[index]; + + if (njs_slow_path(!njs_is_valid(retval))) { + njs_set_undefined(retval); + } + + return NJS_OK; + } + +slow_path: + njs_property_query_init(&pq, NJS_PROPERTY_QUERY_GET, 0); ret = njs_property_query(vm, &pq, value, key); @@ -1093,16 +1143,66 @@ njs_int_t njs_value_property_set(njs_vm_t *vm, njs_value_t *value, njs_value_t *key, njs_value_t *setval) { + double num; + uint32_t index; njs_int_t ret; + njs_array_t *array; njs_object_prop_t *prop; + njs_typed_array_t *tarray; njs_property_query_t pq; + if (njs_fast_path(njs_is_number(key))) { + num = njs_number(key); + + if (njs_slow_path(!njs_number_is_integer_index(num))) { + goto slow_path; + } + + index = (uint32_t) num; + + if (njs_is_typed_array(value)) { + tarray = njs_typed_array(value); + + if (njs_fast_path(index < njs_typed_array_length(tarray))) { + ret = njs_value_to_number(vm, setval, &num); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + njs_typed_array_set(tarray, index, num); + } + + return NJS_OK; + } + + if (njs_slow_path(!(njs_is_object(value) + && njs_object(value)->fast_array))) + { + goto slow_path; + } + + /* NJS_ARRAY */ + + array = njs_array(value); + + if (njs_slow_path(index >= array->length)) { + goto slow_path; + } + + array->start[index] = *setval; + + return NJS_OK; + } + +slow_path: + if (njs_is_primitive(value)) { njs_type_error(vm, "property set on primitive %s type", njs_type_string(value->type)); return NJS_ERROR; } + njs_property_query_init(&pq, NJS_PROPERTY_QUERY_SET, 0); ret = njs_property_query(vm, &pq, value, key); diff -r 2b75ee955589 -r f5afb325896f src/njs_value.h --- a/src/njs_value.h Mon Jan 20 13:24:09 2020 +0300 +++ b/src/njs_value.h Tue Feb 04 20:35:00 2020 +0300 @@ -215,6 +215,7 @@ struct njs_object_s { uint8_t extensible:1; uint8_t error_data:1; + uint8_t fast_array:1; }; diff -r 2b75ee955589 -r f5afb325896f src/njs_vmcode.c --- a/src/njs_vmcode.c Mon Jan 20 13:24:09 2020 +0300 +++ b/src/njs_vmcode.c Tue Feb 04 20:35:00 2020 +0300 @@ -1152,7 +1152,7 @@ njs_vmcode_property_init(njs_vm_t *vm, n switch (value->type) { case NJS_ARRAY: num = njs_key_to_index(key); - if (njs_slow_path(!njs_number_is_integer_index(num, key))) { + if (njs_slow_path(!njs_key_is_integer_index(num, key))) { njs_internal_error(vm, "invalid index while property initialization"); return NJS_ERROR; diff -r 2b75ee955589 -r f5afb325896f src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Mon Jan 20 13:24:09 2020 +0300 +++ b/src/test/njs_unit_test.c Tue Feb 04 20:35:00 2020 +0300 @@ -17772,8 +17772,8 @@ njs_string_to_index_test(njs_vm_t *vm, n } } - is_integer_index = njs_number_is_integer_index(njs_number(&vm->retval), - &tests[i].value); + is_integer_index = njs_key_is_integer_index(njs_number(&vm->retval), + &tests[i].value); if (tests[i].is_integer_index != is_integer_index) { njs_string_get(&tests[i].value, &string); From xeioex at nginx.com Tue Feb 4 17:35:28 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 04 Feb 2020 17:35:28 +0000 Subject: [njs] Introduced memory-efficient arrays. Message-ID: details: https://hg.nginx.org/njs/rev/ccfa84cea2b3 branches: changeset: 1316:ccfa84cea2b3 user: Dmitry Volyntsev date: Tue Feb 04 20:35:02 2020 +0300 description: Introduced memory-efficient arrays. 1) If "length" of ordinary array exceeds 32768 it is converted to to array-like object. 2) Array.prototype.concat() is rewritten according to the spec. 3) Array.prototype.slice() is rewritten according to the spec. diffstat: src/njs_array.c | 997 +++++++++++++++++++++++++++++++++------------- src/njs_array.h | 20 +- src/njs_array_buffer.c | 2 + src/njs_chb.h | 2 +- src/njs_clang.h | 3 + src/njs_crypto.c | 2 + src/njs_date.c | 2 + src/njs_error.c | 2 + src/njs_json.c | 362 ++++++++++------ src/njs_module.c | 2 + src/njs_object.c | 23 +- src/njs_object.h | 2 + src/njs_object_prop.c | 87 +++- src/njs_promise.c | 3 +- src/njs_regexp.c | 2 + src/njs_typed_array.c | 1 - src/njs_value.c | 125 ++++- src/njs_value.h | 8 + src/test/njs_benchmark.c | 6 +- src/test/njs_unit_test.c | 280 ++++++++++++- 20 files changed, 1415 insertions(+), 516 deletions(-) diffs (truncated from 3354 to 1000 lines): diff -r f5afb325896f -r ccfa84cea2b3 src/njs_array.c --- a/src/njs_array.c Tue Feb 04 20:35:00 2020 +0300 +++ b/src/njs_array.c Tue Feb 04 20:35:02 2020 +0300 @@ -8,7 +8,7 @@ #include -#define NJS_ARRAY_LARGE_OBJECT_LENGTH 4096 +#define njs_fast_object(_sz) ((_sz) <= NJS_ARRAY_FAST_OBJECT_LENGTH) typedef struct { @@ -29,35 +29,36 @@ typedef njs_int_t (*njs_array_iterator_h static njs_int_t njs_array_prototype_slice_copy(njs_vm_t *vm, njs_value_t *this, int64_t start, int64_t length); -static njs_value_t *njs_array_copy(njs_value_t *dst, njs_value_t *src); -static njs_array_t *njs_object_indexes(njs_vm_t *vm, njs_value_t *object); njs_array_t * njs_array_alloc(njs_vm_t *vm, uint64_t length, uint32_t spare) { uint64_t size; + njs_int_t ret; njs_array_t *array; + njs_value_t value; if (njs_slow_path(length > UINT32_MAX)) { goto overflow; } - size = length + spare; - - if (njs_slow_path(size > NJS_ARRAY_MAX_LENGTH)) { - goto memory_error; - } - array = njs_mp_alloc(vm->mem_pool, sizeof(njs_array_t)); if (njs_slow_path(array == NULL)) { goto memory_error; } - array->data = njs_mp_align(vm->mem_pool, sizeof(njs_value_t), - size * sizeof(njs_value_t)); - if (njs_slow_path(array->data == NULL)) { - goto memory_error; + size = length + spare; + + if (size <= NJS_ARRAY_LARGE_OBJECT_LENGTH) { + array->data = njs_mp_align(vm->mem_pool, sizeof(njs_value_t), + size * sizeof(njs_value_t)); + if (njs_slow_path(array->data == NULL)) { + goto memory_error; + } + + } else { + array->data = NULL; } array->start = array->data; @@ -67,9 +68,23 @@ njs_array_alloc(njs_vm_t *vm, uint64_t l array->object.type = NJS_ARRAY; array->object.shared = 0; array->object.extensible = 1; - array->object.fast_array = 1; - array->size = size; - array->length = length; + array->object.error_data = 0; + array->object.fast_array = (array->data != NULL); + + if (njs_fast_path(array->object.fast_array)) { + array->size = size; + array->length = length; + + } else { + array->size = 0; + array->length = 0; + + njs_set_array(&value, array); + ret = njs_array_length_redefine(vm, &value, length); + if (njs_slow_path(ret != NJS_OK)) { + return NULL; + } + } return array; @@ -88,6 +103,130 @@ overflow: njs_int_t +njs_array_convert_to_slow_array(njs_vm_t *vm, njs_array_t *array) +{ + uint32_t i, length; + njs_value_t index, value; + njs_object_prop_t *prop; + + njs_set_array(&value, array); + array->object.fast_array = 0; + + length = array->length; + + for (i = 0; i < length; i++) { + njs_uint32_to_string(&index, i); + + if (njs_is_valid(&array->start[i])) { + prop = njs_object_property_add(vm, &value, &index, 0); + if (njs_slow_path(prop == NULL)) { + return NJS_ERROR; + } + + prop->value = array->start[i]; + } + } + + /* GC: release value. */ + + njs_mp_free(vm->mem_pool, array->start); + array->start = NULL; + + return NJS_OK; +} + + +njs_int_t +njs_array_length_redefine(njs_vm_t *vm, njs_value_t *value, uint32_t length) +{ + njs_object_prop_t *prop; + + static const njs_value_t string_length = njs_string("length"); + + if (njs_slow_path(!njs_is_array(value))) { + njs_internal_error(vm, "njs_array_length_redefine() " + "applied to non-array"); + return NJS_ERROR; + } + + prop = njs_object_property_add(vm, value, njs_value_arg(&string_length), 1); + if (njs_slow_path(prop == NULL)) { + njs_internal_error(vm, "njs_array_length_redefine() " + "cannot redefine \"length\""); + return NJS_ERROR; + } + + prop->enumerable = 0; + prop->configurable = 0; + + njs_value_number_set(&prop->value, length); + + return NJS_OK; +} + + +njs_int_t +njs_array_length_set(njs_vm_t *vm, njs_value_t *value, + njs_object_prop_t *prev, njs_value_t *setval) +{ + double num, idx; + uint32_t i, length, prev_length; + njs_int_t ret; + njs_array_t *array, *keys; + + array = njs_object_proto_lookup(njs_object(value), NJS_ARRAY, njs_array_t); + if (njs_slow_path(array == NULL)) { + return NJS_DECLINED; + } + + ret = njs_value_to_number(vm, setval, &num); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + length = njs_number_to_length(num); + if ((double) length != num) { + njs_range_error(vm, "Invalid array length"); + return NJS_ERROR; + } + + ret = njs_value_to_length(vm, &prev->value, &prev_length); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + if (length < prev_length) { + keys = njs_array_indices(vm, value); + if (njs_slow_path(keys == NULL)) { + return NJS_ERROR; + } + + if (keys->length != 0) { + i = keys->length - 1; + + do { + idx = njs_string_to_index(&keys->start[i]); + if (idx >= length) { + ret = njs_value_property_delete(vm, value, &keys->start[i], + NULL); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + } + } while (i-- != 0); + } + } + + ret = njs_array_length_redefine(vm, value, length); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + return NJS_OK; +} + + +njs_int_t njs_array_add(njs_vm_t *vm, njs_array_t *array, njs_value_t *value) { njs_int_t ret; @@ -206,24 +345,25 @@ njs_array_constructor(njs_vm_t *vm, njs_ if (njs_fast_path(array != NULL)) { - value = array->start; - - if (args == NULL) { - while (size != 0) { - njs_set_invalid(value); - value++; - size--; - } - - } else { - while (size != 0) { - njs_retain(args); - *value++ = *args++; - size--; + if (array->object.fast_array) { + value = array->start; + + if (args == NULL) { + while (size != 0) { + njs_set_invalid(value); + value++; + size--; + } + + } else { + while (size != 0) { + njs_retain(args); + *value++ = *args++; + size--; + } } } - njs_set_array(&vm->retval, array); return NJS_OK; @@ -253,26 +393,28 @@ njs_array_is_array(njs_vm_t *vm, njs_val static njs_int_t -njs_array_of(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) +njs_array_of(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t unused) { - uint32_t length, i; - njs_array_t *array; - - length = nargs > 1 ? nargs - 1 : 0; - - array = njs_array_alloc(vm, length, NJS_ARRAY_SPARE); - if (njs_slow_path(array == NULL)) { - return NJS_ERROR; - } - - njs_set_array(&vm->retval, array); - + uint32_t length, i; + njs_array_t *array; + + length = nargs > 1 ? nargs - 1 : 0; + + array = njs_array_alloc(vm, length, NJS_ARRAY_SPARE); + if (njs_slow_path(array == NULL)) { + return NJS_ERROR; + } + + njs_set_array(&vm->retval, array); + + if (array->object.fast_array) { for (i = 0; i < length; i++) { array->start[i] = args[i + 1]; } - - return NJS_OK; + } + + return NJS_OK; } @@ -358,18 +500,12 @@ njs_array_length(njs_vm_t *vm,njs_object return NJS_DECLINED; } - if (njs_slow_path(!njs_is_number(setval))) { - ret = njs_value_to_number(vm, setval, &num); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - } else { - num = njs_number(setval); + ret = njs_value_to_number(vm, setval, &num); + if (njs_slow_path(ret != NJS_OK)) { + return ret; } length = njs_number_to_length(num); - if ((double) length != num) { njs_range_error(vm, "Invalid array length"); return NJS_ERROR; @@ -377,35 +513,46 @@ njs_array_length(njs_vm_t *vm,njs_object array = (njs_array_t *) proto; - size = (int64_t) length - array->length; - - if (size > 0) { - ret = njs_array_expand(vm, array, 0, size); - if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; + if (njs_fast_path(array->object.fast_array)) { + if (njs_fast_path(length <= NJS_ARRAY_LARGE_OBJECT_LENGTH)) { + size = (int64_t) length - array->length; + + if (size > 0) { + ret = njs_array_expand(vm, array, 0, size); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + + val = &array->start[array->length]; + + do { + njs_set_invalid(val); + val++; + size--; + } while (size != 0); + } + + array->length = length; + + *retval = *setval; + return NJS_OK; } - val = &array->start[array->length]; - - do { - njs_set_invalid(val); - val++; - size--; - } while (size != 0); + ret = njs_array_convert_to_slow_array(vm, array); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } } - array->length = length; + prop->type = NJS_PROPERTY; + njs_set_number(&prop->value, length); *retval = *setval; + return NJS_OK; } -/* - * Array.slice(start[, end]). - * JavaScript 1.2, ECMAScript 3. - */ - static njs_int_t njs_array_prototype_slice(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) @@ -413,16 +560,16 @@ njs_array_prototype_slice(njs_vm_t *vm, int64_t start, end, length; uint32_t object_length; njs_int_t ret; - njs_value_t *value; - - value = njs_arg(args, nargs, 0); - - ret = njs_value_to_object(vm, value); + njs_value_t *this; + + this = njs_argument(args, 0); + + ret = njs_value_to_object(vm, this); if (njs_slow_path(ret != NJS_OK)) { return ret; } - ret = njs_object_length(vm, value, &object_length); + ret = njs_object_length(vm, this, &object_length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -474,7 +621,7 @@ njs_array_prototype_slice(njs_vm_t *vm, } } - return njs_array_prototype_slice_copy(vm, value, start, length); + return njs_array_prototype_slice_copy(vm, this, start, length); } @@ -486,8 +633,8 @@ njs_array_prototype_slice_copy(njs_vm_t u_char *dst; uint32_t n; njs_int_t ret; - njs_array_t *array; - njs_value_t *value, name; + njs_array_t *array, *keys; + njs_value_t *value, index, retval, array_value; const u_char *src, *end; njs_slice_prop_t string_slice; njs_string_prop_t string; @@ -497,17 +644,37 @@ njs_array_prototype_slice_copy(njs_vm_t return NJS_ERROR; } - njs_set_array(&vm->retval, array); - - if (length != 0) { - n = 0; - - if (njs_fast_path(njs_is_array(this))) { + if (njs_slow_path(length == 0)) { + goto done; + } + + n = 0; + + if (njs_fast_path(array->object.fast_array)) { + if (njs_fast_path(njs_is_fast_array(this))) { value = njs_array_start(this); do { - /* GC: retain long string and object in values[start]. */ - array->start[n++] = value[start++]; + if (njs_fast_path(njs_is_valid(&value[start]))) { + array->start[n++] = value[start++]; + + } else { + + /* src value may be in Array.prototype object. */ + + njs_uint32_to_string(&index, start++); + + value = &array->start[n++]; + ret = njs_value_property(vm, this, &index, value); + if (njs_slow_path(ret == NJS_ERROR)) { + return NJS_ERROR; + } + + if (ret != NJS_OK) { + njs_set_invalid(value); + } + } + length--; } while (length != 0); @@ -553,10 +720,10 @@ njs_array_prototype_slice_copy(njs_vm_t } else if (njs_is_object(this)) { do { - njs_uint32_to_string(&name, start++); + njs_uint32_to_string(&index, start++); value = &array->start[n++]; - ret = njs_value_property(vm, this, &name, value); + ret = njs_value_property(vm, this, &index, value); if (ret != NJS_OK) { njs_set_invalid(value); @@ -576,8 +743,56 @@ njs_array_prototype_slice_copy(njs_vm_t length--; } while (length != 0); } + + goto done; } + njs_set_array(&array_value, array); + + if (njs_fast_object(length)) { + do { + njs_uint32_to_string(&index, start++); + + ret = njs_value_property(vm, this, &index, &retval); + if (njs_slow_path(ret == NJS_ERROR)) { + return NJS_ERROR; + } + + if (ret == NJS_OK) { + ret = njs_value_property_set(vm, &array_value, &index, &retval); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + } + + length--; + } while (length != 0); + + goto done; + } + + keys = njs_array_indices(vm, this); + if (njs_slow_path(keys == NULL)) { + return NJS_ERROR; + } + + for (n = 0; n < keys->length; n++) { + ret = njs_value_property(vm, this, &keys->start[n], &retval); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + + ret = njs_value_property_set(vm, &array_value, &keys->start[n], + &retval); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + } + +done: + + njs_set_array(&vm->retval, array); + return NJS_OK; } @@ -590,18 +805,18 @@ njs_array_prototype_push(njs_vm_t *vm, n njs_int_t ret; njs_uint_t i; njs_array_t *array; - njs_value_t *value, index; - - value = njs_arg(args, nargs, 0); + njs_value_t *this, index; + length = 0; - - ret = njs_value_to_object(vm, value); + this = njs_argument(args, 0); + + ret = njs_value_to_object(vm, this); if (njs_slow_path(ret != NJS_OK)) { return ret; } - if (njs_is_array(&args[0])) { - array = njs_array(&args[0]); + if (njs_is_fast_array(this)) { + array = njs_array(this); if (nargs != 0) { ret = njs_array_expand(vm, array, 0, nargs); @@ -620,7 +835,7 @@ njs_array_prototype_push(njs_vm_t *vm, n return NJS_OK; } - ret = njs_object_length(vm, value, &length); + ret = njs_object_length(vm, this, &length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -628,13 +843,13 @@ njs_array_prototype_push(njs_vm_t *vm, n for (i = 1; i < nargs; i++) { njs_uint32_to_string(&index, length++); - ret = njs_value_property_set(vm, value, &index, &args[i]); + ret = njs_value_property_set(vm, this, &index, &args[i]); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } } - ret = njs_object_length_set(vm, value, length); + ret = njs_object_length_set(vm, this, length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -652,19 +867,19 @@ njs_array_prototype_pop(njs_vm_t *vm, nj uint32_t length; njs_int_t ret; njs_array_t *array; - njs_value_t *value, *entry, index; - - value = njs_arg(args, nargs, 0); - - ret = njs_value_to_object(vm, value); + njs_value_t *this, *entry, index; + + this = njs_argument(args, 0); + + ret = njs_value_to_object(vm, this); if (njs_slow_path(ret != NJS_OK)) { return ret; } njs_set_undefined(&vm->retval); - if (njs_is_array(&args[0])) { - array = njs_array(&args[0]); + if (njs_is_fast_array(this)) { + array = njs_array(this); if (array->length != 0) { array->length--; @@ -678,7 +893,7 @@ njs_array_prototype_pop(njs_vm_t *vm, nj return NJS_OK; } - ret = njs_object_length(vm, value, &length); + ret = njs_object_length(vm, this, &length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -686,13 +901,13 @@ njs_array_prototype_pop(njs_vm_t *vm, nj if (length != 0) { njs_uint32_to_string(&index, --length); - ret = njs_value_property_delete(vm, value, &index, &vm->retval); + ret = njs_value_property_delete(vm, this, &index, &vm->retval); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } } - ret = njs_object_length_set(vm, value, length); + ret = njs_object_length_set(vm, this, length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -710,19 +925,19 @@ njs_array_prototype_unshift(njs_vm_t *vm njs_int_t ret; njs_uint_t n; njs_array_t *array, *keys; - njs_value_t *value, entry, index; - - value = njs_arg(args, nargs, 0); + njs_value_t *this, entry, index; + + this = njs_argument(args, 0); length = 0; n = nargs - 1; - ret = njs_value_to_object(vm, value); + ret = njs_value_to_object(vm, this); if (njs_slow_path(ret != NJS_OK)) { return ret; } - if (njs_is_array(value)) { - array = njs_array(value); + if (njs_is_fast_array(this)) { + array = njs_array(this); if (array->length > (UINT32_MAX - n)) { njs_type_error(vm, "Invalid length"); @@ -751,7 +966,7 @@ njs_array_prototype_unshift(njs_vm_t *vm return NJS_OK; } - ret = njs_object_length(vm, value, &length); + ret = njs_object_length(vm, this, &length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -765,8 +980,8 @@ njs_array_prototype_unshift(njs_vm_t *vm return NJS_ERROR; } - if (length > NJS_ARRAY_LARGE_OBJECT_LENGTH) { - keys = njs_object_indexes(vm, value); + if (!njs_fast_object(length)) { + keys = njs_array_indices(vm, this); if (njs_slow_path(keys == NULL)) { return NJS_ERROR; } @@ -774,7 +989,7 @@ njs_array_prototype_unshift(njs_vm_t *vm from = keys->length; while (from > 0) { - ret = njs_value_property_delete(vm, value, &keys->start[--from], + ret = njs_value_property_delete(vm, this, &keys->start[--from], &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; @@ -785,7 +1000,7 @@ njs_array_prototype_unshift(njs_vm_t *vm njs_uint32_to_string(&index, (uint32_t) idx + nargs - 1); - ret = njs_value_property_set(vm, value, &index, &entry); + ret = njs_value_property_set(vm, this, &index, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -804,7 +1019,7 @@ njs_array_prototype_unshift(njs_vm_t *vm while (from > 0) { njs_uint32_to_string(&index, --from); - ret = njs_value_property_delete(vm, value, &index, &entry); + ret = njs_value_property_delete(vm, this, &index, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -814,7 +1029,7 @@ njs_array_prototype_unshift(njs_vm_t *vm if (ret == NJS_OK) { njs_uint32_to_string(&index, to); - ret = njs_value_property_set(vm, value, &index, &entry); + ret = njs_value_property_set(vm, this, &index, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -826,7 +1041,7 @@ copy: for (n = 1; n < nargs; n++) { njs_uint32_to_string(&index, n - 1); - ret = njs_value_property_set(vm, value, &index, &args[n]); + ret = njs_value_property_set(vm, this, &index, &args[n]); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -834,7 +1049,7 @@ copy: done: - ret = njs_object_length_set(vm, value, length); + ret = njs_object_length_set(vm, this, length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -852,20 +1067,20 @@ njs_array_prototype_shift(njs_vm_t *vm, uint32_t i, length; njs_int_t ret; njs_array_t *array; - njs_value_t *value, *item, entry, index; - - value = njs_arg(args, nargs, 0); + njs_value_t *this, *item, entry, index; + + this = njs_argument(args, 0); length = 0; - ret = njs_value_to_object(vm, value); + ret = njs_value_to_object(vm, this); if (njs_slow_path(ret != NJS_OK)) { return ret; } njs_set_undefined(&vm->retval); - if (njs_is_array(&args[0])) { - array = njs_array(&args[0]); + if (njs_is_fast_array(this)) { + array = njs_array(this); if (array->length != 0) { array->length--; @@ -881,7 +1096,7 @@ njs_array_prototype_shift(njs_vm_t *vm, return NJS_OK; } - ret = njs_object_length(vm, value, &length); + ret = njs_object_length(vm, this, &length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -892,7 +1107,7 @@ njs_array_prototype_shift(njs_vm_t *vm, njs_uint32_to_string(&index, 0); - ret = njs_value_property_delete(vm, value, &index, &vm->retval); + ret = njs_value_property_delete(vm, this, &index, &vm->retval); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -900,7 +1115,7 @@ njs_array_prototype_shift(njs_vm_t *vm, for (i = 1; i < length; i++) { njs_uint32_to_string(&index, i); - ret = njs_value_property_delete(vm, value, &index, &entry); + ret = njs_value_property_delete(vm, this, &index, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -908,7 +1123,7 @@ njs_array_prototype_shift(njs_vm_t *vm, if (ret == NJS_OK) { njs_uint32_to_string(&index, i - 1); - ret = njs_value_property_set(vm, value, &index, &entry); + ret = njs_value_property_set(vm, this, &index, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -919,7 +1134,7 @@ njs_array_prototype_shift(njs_vm_t *vm, done: - ret = njs_object_length_set(vm, value, length); + ret = njs_object_length_set(vm, this, length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -935,35 +1150,33 @@ njs_array_prototype_splice(njs_vm_t *vm, int64_t n, start, length, items, delta, delete; njs_int_t ret; njs_uint_t i; - njs_value_t *value; + njs_value_t *this; njs_array_t *array, *deleted; - value = njs_arg(args, nargs, 0); - - ret = njs_value_to_object(vm, value); + this = njs_argument(args, 0); + + ret = njs_value_to_object(vm, this); if (njs_slow_path(ret != NJS_OK)) { return ret; } + if (njs_slow_path(!njs_is_fast_array(this))) { + njs_internal_error(vm, "splice() is not implemented yet for objects"); + return NJS_ERROR; + } + array = NULL; start = 0; delete = 0; - if (njs_is_array(value)) { - array = njs_array(value); + if (njs_is_fast_array(this)) { + array = njs_array(this); length = array->length; if (nargs > 1) { - value = njs_argument(args, 1); - - if (njs_slow_path(!njs_is_number(value))) { - ret = njs_value_to_integer(vm, value, &start); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - } else { - start = njs_number_to_integer(njs_number(value)); + ret = njs_value_to_integer(vm, njs_arg(args, nargs, 1), &start); + if (njs_slow_path(ret != NJS_OK)) { + return ret; } if (start < 0) { @@ -980,16 +1193,9 @@ njs_array_prototype_splice(njs_vm_t *vm, delete = length - start; if (nargs > 2) { - value = njs_argument(args, 2); - - if (njs_slow_path(!njs_is_number(value))) { - ret = njs_value_to_integer(vm, value, &n); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - } else { - n = njs_number_to_integer(njs_number(value)); + ret = njs_value_to_integer(vm, njs_arg(args, nargs, 2), &n); + if (njs_slow_path(ret != NJS_OK)) { + return ret; } if (n < 0) { @@ -1007,6 +1213,11 @@ njs_array_prototype_splice(njs_vm_t *vm, return NJS_ERROR; } + if (njs_slow_path(!deleted->object.fast_array)) { + njs_internal_error(vm, "deleted is not a fast_array"); + return NJS_ERROR; + } + if (array != NULL && (delete >= 0 || nargs > 3)) { /* Move deleted items to a new array to return. */ @@ -1055,19 +1266,35 @@ static njs_int_t njs_array_prototype_reverse(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { + uint32_t length; njs_int_t ret; - njs_uint_t i, n, length; + njs_uint_t i, n; njs_value_t value, *this; njs_array_t *array; - this = njs_arg(args, nargs, 0); + this = njs_argument(args, 0); ret = njs_value_to_object(vm, this); if (njs_slow_path(ret != NJS_OK)) { return ret; } - if (njs_is_array(this)) { + ret = njs_object_length(vm, this, &length); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + + if (njs_slow_path(length == 0)) { + vm->retval = *this; + return NJS_OK; + } + + if (njs_slow_path(!njs_is_fast_array(this))) { + njs_internal_error(vm, "reverse() is not implemented yet for objects"); + return NJS_ERROR; + } + + if (njs_is_fast_array(this)) { array = njs_array(this); length = array->length; @@ -1080,10 +1307,6 @@ njs_array_prototype_reverse(njs_vm_t *vm } njs_set_array(&vm->retval, array); - - } else { - /* STUB */ - vm->retval = *this; } return NJS_OK; @@ -1126,15 +1349,18 @@ njs_array_prototype_join(njs_vm_t *vm, n u_char *p, *last; size_t size; ssize_t length; + uint32_t len; njs_int_t ret; njs_chb_t chain; njs_utf8_t utf8; njs_uint_t i; njs_array_t *array; - njs_value_t *value; + njs_value_t *value, *this, index, entry; njs_string_prop_t separator, string; - ret = njs_value_to_object(vm, &args[0]); + this = njs_argument(args, 0); + + ret = njs_value_to_object(vm, this); if (njs_slow_path(ret != NJS_OK)) { return ret; } @@ -1155,20 +1381,47 @@ njs_array_prototype_join(njs_vm_t *vm, n (void) njs_string_prop(&separator, value); - if (!njs_is_array(&args[0]) || njs_array_len(&args[0]) == 0) { + if (njs_slow_path(!njs_is_object(this))) { vm->retval = njs_string_empty; return NJS_OK; } - array = njs_array(&args[0]); + length = 0; + array = NULL; + utf8 = njs_is_byte_string(&separator) ? NJS_STRING_BYTE : NJS_STRING_UTF8; + + if (njs_is_fast_array(this)) { + array = njs_array(this); + len = array->length; + + } else { From xeioex at nginx.com Wed Feb 5 14:58:21 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 05 Feb 2020 14:58:21 +0000 Subject: [njs] Removed dead store assignment after ccfa84cea2b3. Message-ID: details: https://hg.nginx.org/njs/rev/53cad4cfc9cb branches: changeset: 1317:53cad4cfc9cb user: Dmitry Volyntsev date: Wed Feb 05 15:26:18 2020 +0300 description: Removed dead store assignment after ccfa84cea2b3. Found by Clang static analyzer. diffstat: src/njs_value.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diffs (12 lines): diff -r ccfa84cea2b3 -r 53cad4cfc9cb src/njs_value.c --- a/src/njs_value.c Tue Feb 04 20:35:02 2020 +0300 +++ b/src/njs_value.c Wed Feb 05 15:26:18 2020 +0300 @@ -710,8 +710,6 @@ njs_object_property_query(njs_vm_t *vm, ret = njs_lvlhsh_find(&proto->shared_hash, &pq->lhq); if (ret == NJS_OK) { - prop = pq->lhq.value; - return njs_prop_private_copy(vm, pq); } } From xeioex at nginx.com Wed Feb 5 14:58:23 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 05 Feb 2020 14:58:23 +0000 Subject: [njs] Fixed njs_key_string_get() usage. Message-ID: details: https://hg.nginx.org/njs/rev/d228145f0178 branches: changeset: 1318:d228145f0178 user: Dmitry Volyntsev date: Wed Feb 05 15:54:56 2020 +0300 description: Fixed njs_key_string_get() usage. njs_key_string_get() converts Symbol instances to strings, while Symbols should never be converted to strings implicitly. The function is expected to be used only while generating Error messages. Found by Clang static analyzer. diffstat: src/njs_object_prop.c | 10 +++++++--- src/njs_value.c | 2 -- 2 files changed, 7 insertions(+), 5 deletions(-) diffs (47 lines): diff -r 53cad4cfc9cb -r d228145f0178 src/njs_object_prop.c --- a/src/njs_object_prop.c Wed Feb 05 15:26:18 2020 +0300 +++ b/src/njs_object_prop.c Wed Feb 05 15:54:56 2020 +0300 @@ -100,6 +100,7 @@ njs_object_property_add(njs_vm_t *vm, nj njs_bool_t replace) { njs_int_t ret; + njs_value_t key_value; njs_object_prop_t *prop; njs_lvlhsh_query_t lhq; @@ -108,8 +109,13 @@ njs_object_property_add(njs_vm_t *vm, nj return NULL; } + ret = njs_value_to_key(vm, &key_value, key); + if (njs_slow_path(ret != NJS_OK)) { + return NULL; + } + lhq.proto = &njs_object_hash_proto; - njs_key_string_get(vm, key, &lhq.key); + njs_string_get(&key_value, &lhq.key); lhq.key_hash = njs_djb_hash(lhq.key.start, lhq.key.length); lhq.value = prop; lhq.replace = replace; @@ -432,8 +438,6 @@ done: } else { if (njs_slow_path(pq.lhq.key_hash == NJS_LENGTH_HASH)) { - njs_key_string_get(vm, &pq.key, &pq.lhq.key); - if (njs_strstr_eq(&pq.lhq.key, &length_key)) { ret = njs_array_length_set(vm, object, prev, &prop->value); if (ret != NJS_DECLINED) { diff -r 53cad4cfc9cb -r d228145f0178 src/njs_value.c --- a/src/njs_value.c Wed Feb 05 15:26:18 2020 +0300 +++ b/src/njs_value.c Wed Feb 05 15:54:56 2020 +0300 @@ -1288,8 +1288,6 @@ slow_path: switch (prop->type) { case NJS_PROPERTY: if (njs_slow_path(pq.lhq.key_hash == NJS_LENGTH_HASH)) { - njs_key_string_get(vm, &pq.key, &pq.lhq.key); - if (njs_strstr_eq(&pq.lhq.key, &length_key)) { ret = njs_array_length_set(vm, value, prop, setval); if (ret != NJS_DECLINED) { From xeioex at nginx.com Wed Feb 5 14:58:25 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 05 Feb 2020 14:58:25 +0000 Subject: [njs] Removed dead code in njs_json_parse_iterator() after ccfa84cea2b3. Message-ID: details: https://hg.nginx.org/njs/rev/5cec0811cafb branches: changeset: 1319:5cec0811cafb user: Dmitry Volyntsev date: Wed Feb 05 17:30:20 2020 +0300 description: Removed dead code in njs_json_parse_iterator() after ccfa84cea2b3. diffstat: src/njs_json.c | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diffs (14 lines): diff -r d228145f0178 -r 5cec0811cafb src/njs_json.c --- a/src/njs_json.c Wed Feb 05 15:54:56 2020 +0300 +++ b/src/njs_json.c Wed Feb 05 17:30:20 2020 +0300 @@ -933,10 +933,6 @@ njs_json_parse_iterator(njs_vm_t *vm, nj return ret; } } - - njs_internal_error(vm, "njs_json_parse_iterator() unexpected exit"); - - return NJS_ERROR; } From pluknet at nginx.com Wed Feb 5 15:10:05 2020 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 05 Feb 2020 15:10:05 +0000 Subject: [nginx] Version bump. Message-ID: details: https://hg.nginx.org/nginx/rev/2955192fb210 branches: changeset: 7622:2955192fb210 user: Sergey Kandaurov date: Wed Feb 05 16:29:14 2020 +0300 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r b8a512c6466c -r 2955192fb210 src/core/nginx.h --- a/src/core/nginx.h Tue Jan 21 16:39:42 2020 +0300 +++ b/src/core/nginx.h Wed Feb 05 16:29:14 2020 +0300 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1017008 -#define NGINX_VERSION "1.17.8" +#define nginx_version 1017009 +#define NGINX_VERSION "1.17.9" #define NGINX_VER "nginx/" NGINX_VERSION #ifdef NGX_BUILD From pluknet at nginx.com Wed Feb 5 15:10:08 2020 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 05 Feb 2020 15:10:08 +0000 Subject: [nginx] HTTP/2: fixed socket leak with an incomplete HEADERS frame. Message-ID: details: https://hg.nginx.org/nginx/rev/72b792bb3885 branches: changeset: 7623:72b792bb3885 user: Sergey Kandaurov date: Wed Feb 05 16:29:23 2020 +0300 description: HTTP/2: fixed socket leak with an incomplete HEADERS frame. A connection could get stuck without timers if a client has partially sent the HEADERS frame such that it was split on the individual header boundary. In this case, it cannot be processed without the rest of the HEADERS frame. The fix is to call ngx_http_v2_state_headers_save() in this case. Normally, it would be called from the ngx_http_v2_state_header_block() handler on the next iteration, when there is not enough data to continue processing. This isn't the case if recv_buffer became empty and there's no more data to read. diffstat: src/http/v2/ngx_http_v2.c | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-) diffs (19 lines): diff -r 2955192fb210 -r 72b792bb3885 src/http/v2/ngx_http_v2.c --- a/src/http/v2/ngx_http_v2.c Wed Feb 05 16:29:14 2020 +0300 +++ b/src/http/v2/ngx_http_v2.c Wed Feb 05 16:29:23 2020 +0300 @@ -1719,8 +1719,13 @@ ngx_http_v2_state_header_complete(ngx_ht ngx_http_v2_stream_t *stream; if (h2c->state.length) { - h2c->state.handler = ngx_http_v2_state_header_block; - return pos; + if (end - pos > 0) { + h2c->state.handler = ngx_http_v2_state_header_block; + return pos; + } + + return ngx_http_v2_state_headers_save(h2c, pos, end, + ngx_http_v2_state_header_block); } if (!(h2c->state.flags & NGX_HTTP_V2_END_HEADERS_FLAG)) { From xeioex at nginx.com Fri Feb 7 14:35:02 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 07 Feb 2020 14:35:02 +0000 Subject: [njs] Moving modules_hash to njs_vm_shared_t. Message-ID: details: https://hg.nginx.org/njs/rev/f15c819f79e9 branches: changeset: 1320:f15c819f79e9 user: Dmitry Volyntsev date: Thu Feb 06 20:23:12 2020 +0300 description: Moving modules_hash to njs_vm_shared_t. As it is shared between VM instances. diffstat: src/njs_builtin.c | 4 +- src/njs_module.c | 92 ++++++++++++++++++++++++++++------------------- src/njs_vm.c | 6 ++- src/njs_vm.h | 2 + src/test/njs_unit_test.c | 12 ++++++ 5 files changed, 74 insertions(+), 42 deletions(-) diffs (247 lines): diff -r 5cec0811cafb -r f15c819f79e9 src/njs_builtin.c --- a/src/njs_builtin.c Wed Feb 05 17:30:20 2020 +0300 +++ b/src/njs_builtin.c Thu Feb 06 20:23:12 2020 +0300 @@ -208,7 +208,7 @@ njs_builtin_objects_create(njs_vm_t *vm) return NJS_ERROR; } - njs_lvlhsh_init(&vm->modules_hash); + njs_lvlhsh_init(&shared->modules_hash); lhq.replace = 0; lhq.pool = vm->mem_pool; @@ -258,7 +258,7 @@ njs_builtin_objects_create(njs_vm_t *vm) lhq.proto = &njs_modules_hash_proto; lhq.value = module; - ret = njs_lvlhsh_insert(&vm->modules_hash, &lhq); + ret = njs_lvlhsh_insert(&shared->modules_hash, &lhq); if (njs_fast_path(ret != NJS_OK)) { return NJS_ERROR; } diff -r 5cec0811cafb -r f15c819f79e9 src/njs_module.c --- a/src/njs_module.c Wed Feb 05 17:30:20 2020 +0300 +++ b/src/njs_module.c Thu Feb 06 20:23:12 2020 +0300 @@ -24,7 +24,8 @@ static njs_int_t njs_module_absolute_pat static njs_bool_t njs_module_realpath_equal(const njs_str_t *path1, const njs_str_t *path2); static njs_int_t njs_module_read(njs_vm_t *vm, int fd, njs_str_t *body); -static njs_module_t *njs_module_find(njs_vm_t *vm, njs_str_t *name); +static njs_module_t *njs_module_find(njs_vm_t *vm, njs_str_t *name, + njs_bool_t local); static njs_module_t *njs_module_add(njs_vm_t *vm, njs_str_t *name); static njs_int_t njs_module_insert(njs_vm_t *vm, njs_module_t *module); @@ -111,7 +112,7 @@ njs_parser_module(njs_vm_t *vm, njs_pars parser->node = NULL; - module = njs_module_find(vm, &name); + module = njs_module_find(vm, &name, 0); if (module != NULL) { goto found; } @@ -413,8 +414,11 @@ const njs_lvlhsh_proto_t njs_modules_ha static njs_module_t * -njs_module_find(njs_vm_t *vm, njs_str_t *name) +njs_module_find(njs_vm_t *vm, njs_str_t *name, njs_bool_t local) { + njs_int_t ret; + njs_module_t *module, *shared; + njs_object_t *object; njs_lvlhsh_query_t lhq; lhq.key = *name; @@ -425,6 +429,38 @@ njs_module_find(njs_vm_t *vm, njs_str_t return lhq.value; } + if (njs_lvlhsh_find(&vm->shared->modules_hash, &lhq) == NJS_OK) { + shared = lhq.value; + + if (!local) { + return shared; + } + + module = njs_mp_alloc(vm->mem_pool, sizeof(njs_module_t)); + if (njs_slow_path(module == NULL)) { + njs_memory_error(vm); + return NULL; + } + + memcpy(module, shared, sizeof(njs_module_t)); + object = &module->object; + + object->__proto__ = &vm->prototypes[NJS_OBJ_TYPE_OBJECT].object; + object->shared = 0; + object->extensible = 1; + object->error_data = 0; + object->fast_array = 0; + + lhq.replace = 0; + lhq.value = module; + lhq.pool = vm->mem_pool; + + ret = njs_lvlhsh_insert(&vm->modules_hash, &lhq); + if (njs_fast_path(ret == NJS_OK)) { + return module; + } + } + return NULL; } @@ -444,7 +480,6 @@ njs_module_add(njs_vm_t *vm, njs_str_t * ret = njs_name_copy(vm, &module->name, name); if (njs_slow_path(ret != NJS_OK)) { - njs_mp_free(vm->mem_pool, module); njs_memory_error(vm); return NULL; } @@ -457,7 +492,6 @@ njs_module_add(njs_vm_t *vm, njs_str_t * lhq.proto = &njs_modules_hash_proto; ret = njs_lvlhsh_insert(&vm->modules_hash, &lhq); - if (njs_fast_path(ret == NJS_OK)) { return module; } @@ -507,49 +541,31 @@ njs_int_t njs_module_require(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { - njs_int_t ret; - njs_object_t *object; - njs_module_t *module; - njs_lvlhsh_query_t lhq; + njs_int_t ret; + njs_str_t name; + njs_value_t *path; + njs_module_t *module; if (nargs < 2) { njs_type_error(vm, "missing path"); return NJS_ERROR; } - if (njs_slow_path(!njs_is_string(&args[1]))) { - ret = njs_value_to_string(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } + path = njs_argument(args, 1); + ret = njs_value_to_string(vm, path, path); + if (njs_slow_path(ret != NJS_OK)) { + return ret; } - njs_string_get(&args[1], &lhq.key); - lhq.key_hash = njs_djb_hash(lhq.key.start, lhq.key.length); - lhq.proto = &njs_modules_hash_proto; - - if (njs_lvlhsh_find(&vm->modules_hash, &lhq) == NJS_OK) { - module = lhq.value; + njs_string_get(path, &name); - object = njs_mp_alloc(vm->mem_pool, sizeof(njs_object_t)); - if (njs_slow_path(object == NULL)) { - njs_memory_error(vm); - return NJS_ERROR; - } + module = njs_module_find(vm, &name, 1); + if (njs_slow_path(module == NULL)) { + njs_error(vm, "Cannot find module \"%V\"", &name); - *object = module->object; - object->__proto__ = &vm->prototypes[NJS_OBJ_TYPE_OBJECT].object; - object->shared = 0; - object->extensible = 0; - object->error_data = 0; - object->fast_array = 0; - - njs_set_object(&vm->retval, object); - - return NJS_OK; + return NJS_ERROR; } - njs_error(vm, "Cannot find module \"%V\"", &lhq.key); - - return NJS_ERROR; + njs_set_object(&vm->retval, &module->object); + return NJS_OK; } diff -r 5cec0811cafb -r f15c819f79e9 src/njs_vm.c --- a/src/njs_vm.c Wed Feb 05 17:30:20 2020 +0300 +++ b/src/njs_vm.c Thu Feb 06 20:23:12 2020 +0300 @@ -44,6 +44,8 @@ njs_vm_create(njs_vm_opt_t *options) return NULL; } + njs_lvlhsh_init(&vm->values_hash); + vm->options = *options; if (options->shared != NULL) { @@ -56,8 +58,6 @@ njs_vm_create(njs_vm_opt_t *options) } } - njs_lvlhsh_init(&vm->values_hash); - vm->external = options->external; vm->external_objects = njs_arr_create(vm->mem_pool, 4, sizeof(void *)); @@ -302,6 +302,8 @@ njs_vm_init(njs_vm_t *vm) return NJS_ERROR; } + njs_lvlhsh_init(&vm->modules_hash); + njs_lvlhsh_init(&vm->events_hash); njs_queue_init(&vm->posted_events); njs_queue_init(&vm->promise_events); diff -r 5cec0811cafb -r f15c819f79e9 src/njs_vm.h --- a/src/njs_vm.h Wed Feb 05 17:30:20 2020 +0300 +++ b/src/njs_vm.h Thu Feb 06 20:23:12 2020 +0300 @@ -276,6 +276,8 @@ struct njs_vm_shared_s { njs_lvlhsh_t arguments_object_instance_hash; njs_lvlhsh_t regexp_instance_hash; + njs_lvlhsh_t modules_hash; + njs_lvlhsh_t env_hash; njs_object_t string_object; diff -r 5cec0811cafb -r f15c819f79e9 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Wed Feb 05 17:30:20 2020 +0300 +++ b/src/test/njs_unit_test.c Thu Feb 06 20:23:12 2020 +0300 @@ -15825,6 +15825,15 @@ static njs_unit_test_t njs_test[] = { njs_str("var fs = require('fs'); typeof fs"), njs_str("object") }, + { njs_str("var fs = require('fs'); Object.isExtensible(fs)"), + njs_str("true") }, + + { njs_str("require('fs') === require('fs')"), + njs_str("true") }, + + { njs_str("require('fs').a = 1; require('fs').a"), + njs_str("1") }, + /* require('fs').readFile() */ { njs_str("var fs = require('fs');" @@ -16472,6 +16481,9 @@ static njs_unit_test_t njs_shared_test[ { njs_str("import cr from 'crypto'; cr.createHash('md5')"), njs_str("[object Hash]") }, + { njs_str("var fs = require('fs'); fs.a = 1; fs.a"), + njs_str("1") }, + { njs_str("isFinite()"), njs_str("false") }, From xeioex at nginx.com Fri Feb 7 14:35:04 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 07 Feb 2020 14:35:04 +0000 Subject: [njs] Introduced fs.promises API. Message-ID: details: https://hg.nginx.org/njs/rev/03ec7b441da8 branches: changeset: 1321:03ec7b441da8 user: Artem S. Povalyukhin date: Sat Jan 25 21:55:50 2020 +0300 description: Introduced fs.promises API. diffstat: src/njs_fs.c | 759 ++++++++++++++++++++++---------------------- src/njs_object.h | 2 + src/njs_object_prop.c | 51 +++ src/test/njs_unit_test.c | 25 +- test/js/fs_promises_001.js | 60 +++ test/njs_expect_test.exp | 9 + 6 files changed, 522 insertions(+), 384 deletions(-) diffs (truncated from 1262 to 1000 lines): diff -r f15c819f79e9 -r 03ec7b441da8 src/njs_fs.c --- a/src/njs_fs.c Thu Feb 06 20:23:12 2020 +0300 +++ b/src/njs_fs.c Sat Jan 25 21:55:50 2020 +0300 @@ -25,39 +25,33 @@ typedef enum { } njs_fs_writemode_t; +typedef enum { + NJS_FS_ENC_INVALID, + NJS_FS_ENC_NONE, + NJS_FS_ENC_UTF8, +} njs_fs_encoding_t; + + typedef struct { njs_str_t name; int value; } njs_fs_entry_t; -typedef struct { - int errn; - const char *desc; - const char *syscall; -} njs_fs_ioerror_t; - +static njs_int_t njs_fs_fd_read(njs_vm_t *vm, int fd, njs_str_t *data); -static njs_int_t njs_fs_read_file(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t calltype); -static njs_int_t njs_fs_write_file(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t magic); -static njs_int_t njs_fs_rename_sync(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused); +static njs_int_t njs_fs_error(njs_vm_t *vm, const char *syscall, + const char *desc, njs_value_t *path, int errn, njs_value_t *retval); +static njs_int_t njs_fs_result(njs_vm_t *vm, njs_value_t *result, + njs_index_t calltype, const njs_value_t* callback, njs_uint_t nargs); -static njs_int_t njs_fs_fd_read(njs_vm_t *vm, int fd, njs_str_t *data, - njs_fs_ioerror_t *ioerror); -static njs_int_t njs_fs_error(njs_vm_t *vm, const char *syscall, - const char *description, njs_value_t *path, int errn, njs_value_t *retval); -static int njs_fs_flags(njs_str_t *value); -static mode_t njs_fs_mode(njs_value_t *value); +static int njs_fs_flags(njs_vm_t *vm, njs_value_t *value, int default_flags); +static mode_t njs_fs_mode(njs_vm_t *vm, njs_value_t *value, + mode_t default_mode); +static njs_fs_encoding_t njs_fs_encoding(njs_vm_t *vm, njs_value_t *value); + static njs_int_t njs_fs_add_event(njs_vm_t *vm, const njs_value_t *callback, - const njs_value_t *err, const njs_value_t *result); - - -static const njs_value_t njs_fs_errno_string = njs_string("errno"); -static const njs_value_t njs_fs_path_string = njs_string("path"); -static const njs_value_t njs_fs_syscall_string = njs_string("syscall"); + const njs_value_t *args, njs_uint_t nargs); static njs_fs_entry_t njs_flags_table[] = { @@ -101,130 +95,106 @@ njs_fs_path_arg(njs_vm_t *vm, const char } -njs_inline void -njs_fs_set_ioerr(njs_fs_ioerror_t *ioerror, int errn, const char *desc, - const char *syscall) -{ - ioerror->errn = errn; - ioerror->desc = desc; - ioerror->syscall = syscall; -} - - static njs_int_t njs_fs_read_file(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t calltype) { - int fd, flags; - u_char *start; - size_t size; - ssize_t length; - njs_str_t flag, encoding, data; - njs_int_t ret; - const char *path; - struct stat sb; - const njs_value_t *callback, *options; - njs_value_t err; - njs_object_prop_t *prop; - njs_lvlhsh_query_t lhq; - njs_fs_ioerror_t ioerror; + int fd, flags; + u_char *start; + size_t size; + ssize_t length; + njs_str_t data; + njs_int_t ret; + const char *file_path; + njs_value_t flag, encoding, retval, *callback, *options, *path; + struct stat sb; + njs_fs_encoding_t enc; - ret = njs_fs_path_arg(vm, &path, njs_arg(args, nargs, 1), - &njs_str_value("path")); + static const njs_value_t string_flag = njs_string("flag"); + static const njs_value_t string_encoding = njs_string("encoding"); + + path = njs_arg(args, nargs, 1); + ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path")); if (njs_slow_path(ret != NJS_OK)) { return ret; } + callback = NULL; + options = njs_arg(args, nargs, 2); - if (njs_slow_path(calltype == NJS_FS_CALLBACK)) { + if (calltype == NJS_FS_CALLBACK) { callback = njs_arg(args, nargs, njs_min(nargs - 1, 3)); if (!njs_is_function(callback)) { njs_type_error(vm, "\"callback\" must be a function"); return NJS_ERROR; } + if (options == callback) { - options = &njs_value_undefined; - } - - } else { - /* GCC complains about uninitialized callback. */ - callback = NULL; - } - - flag.start = NULL; - encoding.length = 0; - encoding.start = NULL; - - if (njs_slow_path(!njs_is_undefined(options))) { - if (njs_is_string(&args[2])) { - njs_string_get(&args[2], &encoding); - - } else if (njs_is_object(&args[2])) { - lhq.key_hash = NJS_FLAG_HASH; - lhq.key = njs_str_value("flag"); - lhq.proto = &njs_object_hash_proto; - - ret = njs_lvlhsh_find(njs_object_hash(&args[2]), &lhq); - if (ret == NJS_OK) { - prop = lhq.value; - njs_string_get(&prop->value, &flag); - } - - lhq.key_hash = NJS_ENCODING_HASH; - lhq.key = njs_str_value("encoding"); - lhq.proto = &njs_object_hash_proto; - - ret = njs_lvlhsh_find(njs_object_hash(&args[2]), &lhq); - if (ret == NJS_OK) { - prop = lhq.value; - njs_string_get(&prop->value, &encoding); - } - - } else { - njs_type_error(vm, "Unknown options type " - "(a string or object required)"); - return NJS_ERROR; + options = njs_value_arg(&njs_value_undefined); } } - if (flag.start == NULL) { - flag = njs_str_value("r"); + njs_set_undefined(&flag); + njs_set_undefined(&encoding); + + switch (options->type) { + case NJS_STRING: + encoding = *options; + break; + + case NJS_UNDEFINED: + break; + + default: + if (!njs_is_object(options)) { + njs_type_error(vm, "Unknown options type: \"%s\" " + "(a string or object required)", + njs_type_string(options->type)); + return NJS_ERROR; + } + + ret = njs_value_property(vm, options, njs_value_arg(&string_flag), + &flag); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + + ret = njs_value_property(vm, options, njs_value_arg(&string_encoding), + &encoding); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } } - flags = njs_fs_flags(&flag); + flags = njs_fs_flags(vm, &flag, O_RDONLY); if (njs_slow_path(flags == -1)) { - njs_type_error(vm, "Unknown file open flags: \"%V\"", &flag); return NJS_ERROR; } - if (encoding.length != 0 - && (encoding.length != 4 || memcmp(encoding.start, "utf8", 4) != 0)) - { - njs_type_error(vm, "Unknown encoding: \"%V\"", &encoding); + enc = njs_fs_encoding(vm, &encoding); + if (njs_slow_path(enc == NJS_FS_ENC_INVALID)) { return NJS_ERROR; } - njs_fs_set_ioerr(&ioerror, 0, NULL, NULL); - - fd = open(path, flags); + fd = open(file_path, flags); if (njs_slow_path(fd < 0)) { - njs_fs_set_ioerr(&ioerror, errno, strerror(errno), "open"); + ret = njs_fs_error(vm, "open", strerror(errno), path, errno, &retval); goto done; } ret = fstat(fd, &sb); if (njs_slow_path(ret == -1)) { - njs_fs_set_ioerr(&ioerror, errno, strerror(errno), "fstat"); + ret = njs_fs_error(vm, "stat", strerror(errno), path, errno, &retval); goto done; } if (njs_slow_path(!S_ISREG(sb.st_mode))) { - njs_fs_set_ioerr(&ioerror, 0, "File is not regular", "stat"); + ret = njs_fs_error(vm, "stat", "File is not regular", path, 0, &retval); goto done; } - if (encoding.length != 0) { + if (enc == NJS_FS_ENC_UTF8) { length = sb.st_size; if (length > NJS_STRING_MAP_STRIDE) { @@ -244,21 +214,23 @@ njs_fs_read_file(njs_vm_t *vm, njs_value size = sb.st_size; if (njs_fast_path(size != 0)) { - start = njs_string_alloc(vm, &vm->retval, size, length); + start = njs_string_alloc(vm, &retval, size, length); if (njs_slow_path(start == NULL)) { - goto fail; + ret = NJS_ERROR; + goto done; } data.start = start; data.length = size; - ret = njs_fs_fd_read(vm, fd, &data, &ioerror); - if (ret != NJS_OK) { - if (ioerror.desc != NULL) { - goto done; + ret = njs_fs_fd_read(vm, fd, &data); + if (njs_slow_path(ret != NJS_OK)) { + if (ret == NJS_DECLINED) { + ret = njs_fs_error(vm, "read", strerror(errno), path, errno, + &retval); } - goto fail; + goto done; } start = data.start; @@ -268,34 +240,34 @@ njs_fs_read_file(njs_vm_t *vm, njs_value data.length = 0; - ret = njs_fs_fd_read(vm, fd, &data, &ioerror); - if (ret != NJS_OK) { - if (ioerror.desc != NULL) { - goto done; + ret = njs_fs_fd_read(vm, fd, &data); + if (njs_slow_path(ret != NJS_OK)) { + if (ret == NJS_DECLINED) { + ret = njs_fs_error(vm, "read", strerror(errno), path, errno, + &retval); } - goto fail; + goto done; } size = data.length; start = data.start; - ret = njs_string_new(vm, &vm->retval, start, size, length); + ret = njs_string_new(vm, &retval, start, size, length); if (njs_slow_path(ret != NJS_OK)) { - goto fail; + goto done; } } - if (encoding.length != 0) { + if (enc == NJS_FS_ENC_UTF8) { length = njs_utf8_length(start, size); if (length >= 0) { - njs_string_length_set(&vm->retval, length); + njs_string_length_set(&retval, length); } else { - njs_fs_set_ioerr(&ioerror, 0, - "Non-UTF8 file, convertion is not implemented", - NULL); + ret = njs_fs_error(vm, NULL, "Non-UTF8 file, convertion " + "is not implemented", path, 0, &retval); goto done; } } @@ -306,51 +278,8 @@ done: (void) close(fd); } - if (njs_fast_path(calltype == NJS_FS_DIRECT)) { - if (njs_slow_path(ioerror.desc != NULL)) { - (void) njs_fs_error(vm, ioerror.syscall, ioerror.desc, - &args[1], ioerror.errn, &vm->retval); - return NJS_ERROR; - } - - return NJS_OK; - } - - if (calltype == NJS_FS_PROMISE) { - njs_internal_error(vm, "promise callback is not implemented"); - return NJS_ERROR; - } - - if (calltype == NJS_FS_CALLBACK) { - if (njs_slow_path(ioerror.desc)) { - ret = njs_fs_error(vm, ioerror.syscall, ioerror.desc, - &args[1], ioerror.errn, &err); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - njs_set_undefined(&vm->retval); - - } else { - njs_set_undefined(&err); - } - - ret = njs_fs_add_event(vm, callback, &err, &vm->retval); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - njs_set_undefined(&vm->retval); - return NJS_OK; - } - - njs_internal_error(vm, "invalid calltype"); - return NJS_ERROR; - -fail: - - if (fd != -1) { - (void) close(fd); + if (ret == NJS_OK) { + return njs_fs_result(vm, &retval, calltype, callback, 2); } return NJS_ERROR; @@ -361,27 +290,30 @@ static njs_int_t njs_fs_write_file(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t magic) { - int fd, flags; - u_char *p, *end; - mode_t md; - ssize_t n; - njs_str_t data, flag, encoding; - njs_int_t ret; - const char *path; - njs_value_t *mode, err; - njs_fs_calltype_t calltype; - const njs_value_t *callback, *options; - njs_fs_ioerror_t ioerror; - njs_object_prop_t *prop; - njs_lvlhsh_query_t lhq; + int fd, flags; + u_char *p, *end; + mode_t md; + ssize_t n; + njs_str_t content; + njs_int_t ret; + const char *file_path; + njs_value_t flag, mode, encoding, retval, + *path, *data, *callback, *options; + njs_fs_encoding_t enc; + njs_fs_calltype_t calltype; - ret = njs_fs_path_arg(vm, &path, njs_arg(args, nargs, 1), - &njs_str_value("path")); + static const njs_value_t string_flag = njs_string("flag"); + static const njs_value_t string_mode = njs_string("mode"); + static const njs_value_t string_encoding = njs_string("encoding"); + + path = njs_arg(args, nargs, 1); + ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path")); if (njs_slow_path(ret != NJS_OK)) { return ret; } - if (njs_slow_path(!njs_is_string(njs_arg(args, nargs, 2)))) { + data = njs_arg(args, nargs, 2); + if (njs_slow_path(!njs_is_string(data))) { njs_type_error(vm, "\"data\" must be a string"); return NJS_ERROR; } @@ -390,104 +322,84 @@ njs_fs_write_file(njs_vm_t *vm, njs_valu calltype = magic & 3; options = njs_arg(args, nargs, 3); - if (njs_slow_path(calltype == NJS_FS_CALLBACK)) { + if (calltype == NJS_FS_CALLBACK) { callback = njs_arg(args, nargs, njs_min(nargs - 1, 4)); if (!njs_is_function(callback)) { njs_type_error(vm, "\"callback\" must be a function"); return NJS_ERROR; } + if (options == callback) { - options = &njs_value_undefined; + options = njs_value_arg(&njs_value_undefined); } } - mode = NULL; - /* GCC complains about uninitialized flag.length. */ - flag.length = 0; - flag.start = NULL; - encoding.length = 0; - encoding.start = NULL; - - if (njs_slow_path(!njs_is_undefined(options))) { - if (njs_is_string(&args[3])) { - njs_string_get(&args[3], &encoding); + njs_set_undefined(&flag); + njs_set_undefined(&mode); + njs_set_undefined(&encoding); - } else if (njs_is_object(&args[3])) { - lhq.key_hash = NJS_FLAG_HASH; - lhq.key = njs_str_value("flag"); - lhq.proto = &njs_object_hash_proto; + switch (options->type) { + case NJS_STRING: + encoding = *options; + break; - ret = njs_lvlhsh_find(njs_object_hash(&args[3]), &lhq); - if (ret == NJS_OK) { - prop = lhq.value; - njs_string_get(&prop->value, &flag); - } + case NJS_UNDEFINED: + break; - lhq.key_hash = NJS_ENCODING_HASH; - lhq.key = njs_str_value("encoding"); - lhq.proto = &njs_object_hash_proto; - - ret = njs_lvlhsh_find(njs_object_hash(&args[3]), &lhq); - if (ret == NJS_OK) { - prop = lhq.value; - njs_string_get(&prop->value, &encoding); - } + default: + if (!njs_is_object(options)) { + njs_type_error(vm, "Unknown options type: \"%s\" " + "(a string or object required)", + njs_type_string(options->type)); + return NJS_ERROR; + } - lhq.key_hash = NJS_MODE_HASH; - lhq.key = njs_str_value("mode"); - lhq.proto = &njs_object_hash_proto; + ret = njs_value_property(vm, options, njs_value_arg(&string_flag), + &flag); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } - ret = njs_lvlhsh_find(njs_object_hash(&args[3]), &lhq); - if (ret == NJS_OK) { - prop = lhq.value; - mode = &prop->value; - } + ret = njs_value_property(vm, options, njs_value_arg(&string_mode), + &mode); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } - } else { - njs_type_error(vm, "Unknown options type " - "(a string or object required)"); - return NJS_ERROR; + ret = njs_value_property(vm, options, njs_value_arg(&string_encoding), + &encoding); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; } } - if (flag.start != NULL) { - flags = njs_fs_flags(&flag); - if (njs_slow_path(flags == -1)) { - njs_type_error(vm, "Unknown file open flags: \"%V\"", &flag); - return NJS_ERROR; - } - - } else { - flags = O_CREAT | O_WRONLY; - flags |= ((magic >> 2) == NJS_FS_APPEND) ? O_APPEND : O_TRUNC; + flags = njs_fs_flags(vm, &flag, O_CREAT | O_WRONLY); + if (njs_slow_path(flags == -1)) { + return NJS_ERROR; } - if (mode != NULL) { - md = njs_fs_mode(mode); + flags |= ((magic >> 2) == NJS_FS_APPEND) ? O_APPEND : O_TRUNC; - } else { - md = 0666; - } - - if (encoding.length != 0 - && (encoding.length != 4 || memcmp(encoding.start, "utf8", 4) != 0)) - { - njs_type_error(vm, "Unknown encoding: \"%V\"", &encoding); + md = njs_fs_mode(vm, &mode, 0666); + if (njs_slow_path(md == (mode_t) -1)) { return NJS_ERROR; } - njs_fs_set_ioerr(&ioerror, 0, NULL, NULL); + enc = njs_fs_encoding(vm, &encoding); + if (njs_slow_path(enc == NJS_FS_ENC_INVALID)) { + return NJS_ERROR; + } - fd = open(path, flags, md); + fd = open(file_path, flags, md); if (njs_slow_path(fd < 0)) { - njs_fs_set_ioerr(&ioerror, errno, strerror(errno), "open"); + ret = njs_fs_error(vm, "open", strerror(errno), path, errno, &retval); goto done; } - njs_string_get(&args[2], &data); + njs_string_get(data, &content); - p = data.start; - end = p + data.length; + p = content.start; + end = p + content.length; while (p < end) { n = write(fd, p, end - p); @@ -496,57 +408,27 @@ njs_fs_write_file(njs_vm_t *vm, njs_valu continue; } - njs_fs_set_ioerr(&ioerror, errno, strerror(errno), "write"); + ret = njs_fs_error(vm, "write", strerror(errno), path, errno, + &retval); goto done; } p += n; } + ret = NJS_OK; + njs_set_undefined(&retval); + done: if (fd != -1) { (void) close(fd); } - if (njs_fast_path(calltype == NJS_FS_DIRECT)) { - if (njs_slow_path(ioerror.desc != NULL)) { - (void) njs_fs_error(vm, ioerror.syscall, ioerror.desc, &args[1], - ioerror.errn, &vm->retval); - return NJS_ERROR; - } - - njs_set_undefined(&vm->retval); - return NJS_OK; - } - - if (calltype == NJS_FS_PROMISE) { - njs_internal_error(vm, "not implemented"); - return NJS_ERROR; + if (ret == NJS_OK) { + return njs_fs_result(vm, &retval, calltype, callback, 1); } - if (calltype == NJS_FS_CALLBACK) { - if (ioerror.desc != NULL) { - ret = njs_fs_error(vm, ioerror.syscall, ioerror.desc, &args[1], - ioerror.errn, &err); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - } else { - njs_set_undefined(&err); - } - - ret = njs_fs_add_event(vm, callback, &err, NULL); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - njs_set_undefined(&vm->retval); - return NJS_OK; - } - - njs_internal_error(vm, "invalid calltype"); return NJS_ERROR; } @@ -584,8 +466,7 @@ njs_fs_rename_sync(njs_vm_t *vm, njs_val static njs_int_t -njs_fs_fd_read(njs_vm_t *vm, int fd, njs_str_t *data, - njs_fs_ioerror_t *ioerror) +njs_fs_fd_read(njs_vm_t *vm, int fd, njs_str_t *data) { u_char *p, *end, *start; size_t size; @@ -610,7 +491,6 @@ njs_fs_fd_read(njs_vm_t *vm, int fd, njs n = read(fd, p, end - p); if (njs_slow_path(n < 0)) { - njs_fs_set_ioerr(ioerror, errno, strerror(errno), "read"); return NJS_DECLINED; } @@ -645,152 +525,224 @@ njs_fs_fd_read(njs_vm_t *vm, int fd, njs } +static int +njs_fs_flags(njs_vm_t *vm, njs_value_t *value, int default_flags) +{ + njs_str_t flags; + njs_int_t ret; + njs_fs_entry_t *fl; + + if (njs_is_undefined(value)) { + return default_flags; + } + + ret = njs_value_to_string(vm, value, value); + if (njs_slow_path(ret != NJS_OK)) { + return -1; + } + + njs_string_get(value, &flags); + + for (fl = &njs_flags_table[0]; fl->name.length != 0; fl++) { + if (njs_strstr_eq(&flags, &fl->name)) { + return fl->value; + } + } + + njs_type_error(vm, "Unknown file open flags: \"%V\"", &flags); + + return -1; +} + + +static mode_t +njs_fs_mode(njs_vm_t *vm, njs_value_t *value, mode_t default_mode) +{ + uint32_t u32; + njs_int_t ret; + + if (njs_is_undefined(value)) { + return default_mode; + } + + ret = njs_value_to_uint32(vm, value, &u32); + if (njs_slow_path(ret != NJS_OK)) { + return (mode_t) -1; + } + + return (mode_t) u32; +} + + +static njs_fs_encoding_t +njs_fs_encoding(njs_vm_t *vm, njs_value_t *value) +{ + njs_str_t enc; + njs_int_t ret; + + if (njs_is_undefined(value)) { + return NJS_FS_ENC_NONE; + } + + ret = njs_value_to_string(vm, value, value); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_FS_ENC_INVALID; + } + + njs_string_get(value, &enc); + + if (enc.length != 4 || memcmp(enc.start, "utf8", 4) != 0) { + njs_type_error(vm, "Unknown encoding: \"%V\"", &enc); + return NJS_FS_ENC_INVALID; + } + + return NJS_FS_ENC_UTF8; +} + + static njs_int_t njs_fs_error(njs_vm_t *vm, const char *syscall, const char *description, njs_value_t *path, int errn, njs_value_t *retval) { - size_t size; - njs_int_t ret; - njs_value_t string, value; - njs_object_t *error; - njs_object_prop_t *prop; - njs_lvlhsh_query_t lhq; + size_t size; + njs_int_t ret; + njs_value_t value; + njs_object_t *error; + + static const njs_value_t string_errno = njs_string("errno"); + static const njs_value_t string_path = njs_string("path"); + static const njs_value_t string_syscall = njs_string("syscall"); size = description != NULL ? njs_strlen(description) : 0; - ret = njs_string_new(vm, &string, (u_char *) description, size, size); + ret = njs_string_new(vm, &value, (u_char *) description, size, size); if (njs_slow_path(ret != NJS_OK)) { return NJS_ERROR; } - error = njs_error_alloc(vm, NJS_OBJ_TYPE_ERROR, NULL, &string); + error = njs_error_alloc(vm, NJS_OBJ_TYPE_ERROR, NULL, &value); if (njs_slow_path(error == NULL)) { return NJS_ERROR; } - lhq.replace = 0; - lhq.pool = vm->mem_pool; + njs_set_object(retval, error); if (errn != 0) { - lhq.key = njs_str_value("errno"); - lhq.key_hash = NJS_ERRNO_HASH; - lhq.proto = &njs_object_hash_proto; - njs_set_number(&value, errn); - - prop = njs_object_prop_alloc(vm, &njs_fs_errno_string, &value, 1); - if (njs_slow_path(prop == NULL)) { - return NJS_ERROR; - } - - lhq.value = prop; - - ret = njs_lvlhsh_insert(&error->hash, &lhq); + ret = njs_value_property_set(vm, retval, njs_value_arg(&string_errno), + &value); if (njs_slow_path(ret != NJS_OK)) { - njs_internal_error(vm, "lvlhsh insert failed"); return NJS_ERROR; } } if (path != NULL) { - lhq.key = njs_str_value("path"); - lhq.key_hash = NJS_PATH_HASH; - lhq.proto = &njs_object_hash_proto; - - prop = njs_object_prop_alloc(vm, &njs_fs_path_string, path, 1); - if (njs_slow_path(prop == NULL)) { - return NJS_ERROR; - } - - lhq.value = prop; - - ret = njs_lvlhsh_insert(&error->hash, &lhq); + ret = njs_value_property_set(vm, retval, njs_value_arg(&string_path), + path); if (njs_slow_path(ret != NJS_OK)) { - njs_internal_error(vm, "lvlhsh insert failed"); return NJS_ERROR; } } if (syscall != NULL) { size = njs_strlen(syscall); - ret = njs_string_new(vm, &string, (u_char *) syscall, size, size); + ret = njs_string_new(vm, &value, (u_char *) syscall, size, size); if (njs_slow_path(ret != NJS_OK)) { return NJS_ERROR; } - lhq.key = njs_str_value("sycall"); - lhq.key_hash = NJS_SYSCALL_HASH; - lhq.proto = &njs_object_hash_proto; - - prop = njs_object_prop_alloc(vm, &njs_fs_syscall_string, &string, 1); - if (njs_slow_path(prop == NULL)) { - return NJS_ERROR; - } - - lhq.value = prop; - - ret = njs_lvlhsh_insert(&error->hash, &lhq); + ret = njs_value_property_set(vm, retval, njs_value_arg(&string_syscall), + &value); if (njs_slow_path(ret != NJS_OK)) { - njs_internal_error(vm, "lvlhsh insert failed"); return NJS_ERROR; } } - njs_set_object(retval, error); - return NJS_OK; } -static int -njs_fs_flags(njs_str_t *value) +static njs_int_t +ngx_fs_promise_trampoline(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t unused) { - njs_fs_entry_t *fl; + njs_value_t value; - for (fl = &njs_flags_table[0]; fl->name.length != 0; fl++) { - if (njs_strstr_eq(value, &fl->name)) { - return fl->value; - } - } - - return -1; + return njs_function_call(vm, njs_function(&args[1]), &njs_value_undefined, + &args[2], 1, &value); } -static mode_t -njs_fs_mode(njs_value_t *value) +static const njs_value_t promise_trampoline = + njs_native_function(ngx_fs_promise_trampoline, 2); + + +static njs_int_t +njs_fs_result(njs_vm_t *vm, njs_value_t *result, njs_index_t calltype, + const njs_value_t *callback, njs_uint_t nargs) { - switch (value->type) { - case NJS_OBJECT_NUMBER: - value = njs_object_value(value); - /* Fall through. */ + njs_int_t ret; + njs_value_t promise, callbacks[2], arguments[2]; + + switch (calltype) { + case NJS_FS_DIRECT: + vm->retval = *result; + return njs_is_error(result) ? NJS_ERROR : NJS_OK; + + case NJS_FS_PROMISE: + ret = njs_vm_promise_create(vm, &promise, &callbacks[0]); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + arguments[0] = njs_is_error(result) ? callbacks[1] : callbacks[0]; + arguments[1] = *result; - case NJS_NUMBER: - return (mode_t) njs_number(value); + ret = njs_fs_add_event(vm, njs_value_arg(&promise_trampoline), + njs_value_arg(&arguments), 2); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + vm->retval = promise; + + return NJS_OK; + + case NJS_FS_CALLBACK: + if (njs_is_error(result)) { + arguments[0] = *result; + njs_set_undefined(&arguments[1]); - case NJS_OBJECT_STRING: - value = njs_object_value(value); - /* Fall through. */ + } else { + njs_set_undefined(&arguments[0]); + arguments[1] = *result; + } - case NJS_STRING: - return (mode_t) njs_string_to_number(value, 0); + ret = njs_fs_add_event(vm, callback, njs_value_arg(&arguments), + nargs); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + njs_set_undefined(&vm->retval); + + return NJS_OK; default: - return (mode_t) 0; + njs_internal_error(vm, "invalid calltype"); + + return NJS_ERROR; } } static njs_int_t njs_fs_add_event(njs_vm_t *vm, const njs_value_t *callback, - const njs_value_t *err, const njs_value_t *result) + const njs_value_t *args, njs_uint_t nargs) { - njs_int_t nargs; njs_event_t *event; njs_vm_ops_t *ops; - nargs = (result == NULL) ? 1 : 2; - ops = vm->options.ops; if (njs_slow_path(ops == NULL)) { njs_internal_error(vm, "not supported by host environment"); @@ -813,12 +765,7 @@ njs_fs_add_event(njs_vm_t *vm, const njs goto memory_error; } - /* GC: retain */ - event->args[0] = *err; - - if (nargs == 2) { - event->args[1] = *result; - } + memcpy(event->args, args, sizeof(njs_value_t) * nargs); event->host_event = ops->set_timer(vm->external, 0, event); if (njs_slow_path(event->host_event == NULL)) { @@ -836,6 +783,50 @@ memory_error: } +static const njs_object_prop_t njs_fs_promises_properties[] = +{ + { + .type = NJS_PROPERTY, + .name = njs_string("readFile"), + .value = njs_native_function2(njs_fs_read_file, 0, NJS_FS_PROMISE), + .writable = 1, + .configurable = 1, + }, + + { From xeioex at nginx.com Fri Feb 7 14:35:06 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 07 Feb 2020 14:35:06 +0000 Subject: [njs] Introduced fs.access and friends. Message-ID: details: https://hg.nginx.org/njs/rev/1f3e045ea017 branches: changeset: 1322:1f3e045ea017 user: Artem S. Povalyukhin date: Sun Jan 26 21:56:23 2020 +0300 description: Introduced fs.access and friends. diffstat: src/njs_fs.c | 137 ++++++++++++++++++++++++++++++++++++++++ src/test/njs_interactive_test.c | 5 +- src/test/njs_unit_test.c | 48 ++++++++++++++ test/js/fs_promises_001.js | 4 +- test/js/fs_promises_002.js | 71 ++++++++++++++++++++ test/njs_expect_test.exp | 7 +- 6 files changed, 268 insertions(+), 4 deletions(-) diffs (374 lines): diff -r 03ec7b441da8 -r 1f3e045ea017 src/njs_fs.c --- a/src/njs_fs.c Sat Jan 25 21:55:50 2020 +0300 +++ b/src/njs_fs.c Sun Jan 26 21:56:23 2020 +0300 @@ -466,6 +466,68 @@ njs_fs_rename_sync(njs_vm_t *vm, njs_val static njs_int_t +njs_fs_access(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t calltype) +{ + int md; + njs_int_t ret; + const char *file_path; + njs_value_t retval, *path, *callback, *mode; + + path = njs_arg(args, nargs, 1); + ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path")); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + callback = NULL; + mode = njs_arg(args, nargs, 2); + + if (calltype == NJS_FS_CALLBACK) { + callback = njs_arg(args, nargs, njs_min(nargs - 1, 3)); + if (!njs_is_function(callback)) { + njs_type_error(vm, "\"callback\" must be a function"); + return NJS_ERROR; + } + + if (mode == callback) { + mode = njs_value_arg(&njs_value_undefined); + } + } + + switch (mode->type) { + case NJS_UNDEFINED: + md = F_OK; + break; + + case NJS_NUMBER: + md = njs_number(mode); + break; + + default: + njs_type_error(vm, "\"mode\" must be a number"); + return NJS_ERROR; + } + + ret = access(file_path, md); + if (njs_slow_path(ret != 0)) { + ret = njs_fs_error(vm, "access", strerror(errno), path, errno, &retval); + goto done; + } + + njs_set_undefined(&retval); + +done: + + if (ret == NJS_OK) { + return njs_fs_result(vm, &retval, calltype, callback, 1); + } + + return NJS_ERROR; +} + + +static njs_int_t njs_fs_fd_read(njs_vm_t *vm, int fd, njs_str_t *data) { u_char *p, *end, *start; @@ -810,6 +872,14 @@ static const njs_object_prop_t njs_fs_p .writable = 1, .configurable = 1, }, + + { + .type = NJS_PROPERTY, + .name = njs_string("access"), + .value = njs_native_function2(njs_fs_access, 0, NJS_FS_PROMISE), + .writable = 1, + .configurable = 1, + }, }; @@ -827,6 +897,50 @@ njs_fs_promises(njs_vm_t *vm, njs_object } +static const njs_object_prop_t njs_fs_constants_properties[] = +{ + { + .type = NJS_PROPERTY, + .name = njs_string("F_OK"), + .value = njs_value(NJS_NUMBER, 0, F_OK), + .enumerable = 1, + }, + { + .type = NJS_PROPERTY, + .name = njs_string("R_OK"), + .value = njs_value(NJS_NUMBER, 1, R_OK), + .enumerable = 1, + }, + { + .type = NJS_PROPERTY, + .name = njs_string("W_OK"), + .value = njs_value(NJS_NUMBER, 1, W_OK), + .enumerable = 1, + }, + { + .type = NJS_PROPERTY, + .name = njs_string("X_OK"), + .value = njs_value(NJS_NUMBER, 1, X_OK), + .enumerable = 1, + }, +}; + + +static const njs_object_init_t njs_fs_constants_init = { + njs_fs_constants_properties, + njs_nitems(njs_fs_constants_properties), +}; + + +static njs_int_t +njs_fs_constants(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, + njs_value_t *unused, njs_value_t *retval) +{ + return njs_object_prop_init(vm, &njs_fs_constants_init, prop, value, + retval); +} + + static const njs_object_prop_t njs_fs_object_properties[] = { { @@ -838,12 +952,35 @@ static const njs_object_prop_t njs_fs_o { .type = NJS_PROPERTY_HANDLER, + .name = njs_string("constants"), + .value = njs_prop_handler(njs_fs_constants), + .enumerable = 1, + }, + + { + .type = NJS_PROPERTY_HANDLER, .name = njs_string("promises"), .value = njs_prop_handler(njs_fs_promises), }, { .type = NJS_PROPERTY, + .name = njs_string("access"), + .value = njs_native_function2(njs_fs_access, 0, NJS_FS_CALLBACK), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("accessSync"), + .value = njs_native_function2(njs_fs_access, 0, NJS_FS_DIRECT), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, .name = njs_string("readFile"), .value = njs_native_function2(njs_fs_read_file, 0, NJS_FS_CALLBACK), .writable = 1, diff -r 03ec7b441da8 -r 1f3e045ea017 src/test/njs_interactive_test.c --- a/src/test/njs_interactive_test.c Sat Jan 25 21:55:50 2020 +0300 +++ b/src/test/njs_interactive_test.c Sun Jan 26 21:56:23 2020 +0300 @@ -233,7 +233,10 @@ static njs_interactive_test_t njs_test[ " at main (native)\n") }, { njs_str("var fs = require('fs');" - "['readFile'," + "[" + " 'access'," + " 'accessSync'," + " 'readFile'," " 'readFileSync'," " 'writeFile'," " 'writeFileSync'," diff -r 03ec7b441da8 -r 1f3e045ea017 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Sat Jan 25 21:55:50 2020 +0300 +++ b/src/test/njs_unit_test.c Sun Jan 26 21:56:23 2020 +0300 @@ -15991,9 +15991,39 @@ static njs_unit_test_t njs_test[] = ".every((x) => x === true)"), njs_str("true")}, + /* require('fs').access() */ + + { njs_str("var fs = require('fs');" + "fs.access()"), + njs_str("TypeError: \"path\" must be a string") }, + + { njs_str("var fs = require('fs');" + "fs.access('/njs_unknown_path')"), + njs_str("TypeError: \"callback\" must be a function") }, + + { njs_str("var fs = require('fs');" + "fs.access('/njs_unknown_path', fs.constants.F_OK)"), + njs_str("TypeError: \"callback\" must be a function") }, + + { njs_str("var fs = require('fs');" + "fs.access('/njs_unknown_path', 'fail', function () {})"), + njs_str("TypeError: \"mode\" must be a number") }, + + /* require('fs').accessSync() */ + + { njs_str("var fs = require('fs');" + "fs.accessSync()"), + njs_str("TypeError: \"path\" must be a string") }, + + { njs_str("var fs = require('fs');" + "fs.accessSync('/njs_unknown_path', 'fail')"), + njs_str("TypeError: \"mode\" must be a number") }, + { njs_str("var " "fs = require('fs')," "func = [" + "'access'," + "'accessSync'," "'readFile'," "'readFileSync'," "'writeFile'," @@ -16018,6 +16048,7 @@ static njs_unit_test_t njs_test[] = { njs_str("var " "fs = require('fs').promises," "func = [" + "'access'," "'readFile'," "'writeFile'," "'appendFile'," @@ -16025,6 +16056,23 @@ static njs_unit_test_t njs_test[] = "func.every((x) => typeof fs[x] == 'function')"), njs_str("true")}, + /* require('fs').constants */ + + { njs_str("var fs = require('fs');" + "typeof fs.constants"), + njs_str("object") }, + + { njs_str("var " + "fsc = require('fs').constants," + "items = [" + "'F_OK'," + "'R_OK'," + "'W_OK'," + "'X_OK'," + "];" + "items.every((x) => typeof fsc[x] == 'number')"), + njs_str("true")}, + /* require('crypto').createHash() */ { njs_str("var h = require('crypto').createHash('sha1');" diff -r 03ec7b441da8 -r 1f3e045ea017 test/js/fs_promises_001.js --- a/test/js/fs_promises_001.js Sat Jan 25 21:55:50 2020 +0300 +++ b/test/js/fs_promises_001.js Sun Jan 26 21:56:23 2020 +0300 @@ -15,10 +15,10 @@ Promise.resolve() return fs.readFile(fname).then(fs.readFile); }) .then((data) => { - console.log('short citcut ok', data == fname); + console.log('short circut ok', data == fname); }) .catch((e) => { - console.log('short citcut failed', e); + console.log('short circut failed', e); }) .then(() => { var read = fs.readFile.bind(fs, fname, 'utf8'); diff -r 03ec7b441da8 -r 1f3e045ea017 test/js/fs_promises_002.js --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test/js/fs_promises_002.js Sun Jan 26 21:56:23 2020 +0300 @@ -0,0 +1,71 @@ +var fs = require('fs'); +var fsp = fs.promises; +var fname = '/tmp/njs_fs_promises_002'; + +var testSync = new Promise((resolve, reject) => { + var failed = false; + try { + fs.writeFileSync(fname, fname); + + fs.accessSync(fname); + fs.accessSync(fname, fs.constants.R_OK | fs.constants.W_OK); + + try { + fs.accessSync(fname + '___'); + failed = true; + } catch(e) { + failed = (e.syscall != 'access'); + // TODO: e.code != 'ENOENT' + } + resolve(failed); + } catch (e) { + reject(e); + } +}); + +var testCallback = new Promise((resolve, reject) => { + var failed = false; + + fs.writeFileSync(fname, fname); + + fs.access(fname, (err) => { + failed = (err !== undefined); + fs.access(fname, fs.constants.R_OK | fs.constants.W_OK, (err) => { + failed |= (err !== undefined); + fs.access(fname + '___', (err) => { + failed |= ((err === undefined) || (err.syscall != 'access')); + resolve(failed); + }); + }); + }); +}); + +Promise.resolve() +.then(() => testSync) +.then((failed) => { + console.log('testSync ok', !failed); +}) +.catch((e) => { + console.log('testSync failed', e); +}) +.then(() => testCallback) +.then((failed) => { + console.log('testCallback ok', !failed); +}) +.catch((e) => { + console.log('testCallback failed', e); +}) +.then(() => { + fs.writeFileSync(fname, fname); + + return fsp.access(fname) + .then(() => fsp.access(fname, fs.constants.R_OK | fs.constants.W_OK)) + .then(() => fsp.access(fname + '___')); +}) +.then(() => { + console.log('testPromise failed'); +}) +.catch((e) => { + console.log('testPromise ok', (e.syscall == 'access') && (e.path == fname + '___')); +}) +; diff -r 03ec7b441da8 -r 1f3e045ea017 test/njs_expect_test.exp --- a/test/njs_expect_test.exp Sat Jan 25 21:55:50 2020 +0300 +++ b/test/njs_expect_test.exp Sun Jan 26 21:56:23 2020 +0300 @@ -1064,9 +1064,14 @@ PatchedPromise async done" njs_run {"./test/js/fs_promises_001.js"} \ "init ok true -short citcut ok true +short circut ok true chain ok true error 1 ok true error 2 ok true error 3 ok true errors ok" + +njs_run {"./test/js/fs_promises_002.js"} \ +"testSync ok true +testCallback ok true +testPromise ok true" From xeioex at nginx.com Mon Feb 10 16:01:59 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Mon, 10 Feb 2020 16:01:59 +0000 Subject: [njs] Improved Array object allocation after ccfa84cea2b3. Message-ID: details: https://hg.nginx.org/njs/rev/79c14715edc2 branches: changeset: 1323:79c14715edc2 user: Dmitry Volyntsev date: Mon Feb 10 17:09:53 2020 +0300 description: Improved Array object allocation after ccfa84cea2b3. Sometimes flat array alignment is desired, even if its "length" exceeds 32768. This patch introduces additional argument for njs_array_alloc() which enforced flat allocation when it is set. diffstat: src/njs_array.c | 75 +++++++++++++++++++++++++++++++++++++++++++---------- src/njs_array.h | 5 ++- src/njs_builtin.c | 2 +- src/njs_extern.c | 2 +- src/njs_function.c | 4 +- src/njs_json.c | 20 ++++++++++---- src/njs_object.c | 73 ++++++++++++++++++++++++++++++--------------------- src/njs_regexp.c | 2 +- src/njs_string.c | 4 +- src/njs_value.c | 4 +- src/njs_vm.c | 2 +- src/njs_vmcode.c | 2 +- 12 files changed, 133 insertions(+), 62 deletions(-) diffs (715 lines): diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_array.c --- a/src/njs_array.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_array.c Mon Feb 10 17:09:53 2020 +0300 @@ -32,7 +32,7 @@ static njs_int_t njs_array_prototype_sli njs_array_t * -njs_array_alloc(njs_vm_t *vm, uint64_t length, uint32_t spare) +njs_array_alloc(njs_vm_t *vm, njs_bool_t flat, uint64_t length, uint32_t spare) { uint64_t size; njs_int_t ret; @@ -50,7 +50,7 @@ njs_array_alloc(njs_vm_t *vm, uint64_t l size = length + spare; - if (size <= NJS_ARRAY_LARGE_OBJECT_LENGTH) { + if (flat || size <= NJS_ARRAY_LARGE_OBJECT_LENGTH) { array->data = njs_mp_align(vm->mem_pool, sizeof(njs_value_t), size * sizeof(njs_value_t)); if (njs_slow_path(array->data == NULL)) { @@ -102,6 +102,19 @@ overflow: } +void +njs_array_destroy(njs_vm_t *vm, njs_array_t *array) +{ + if (array->data != NULL) { + njs_mp_free(vm->mem_pool, array->data); + } + + /* TODO: destroy keys. */ + + njs_mp_free(vm->mem_pool, array); +} + + njs_int_t njs_array_convert_to_slow_array(njs_vm_t *vm, njs_array_t *array) { @@ -195,6 +208,8 @@ njs_array_length_set(njs_vm_t *vm, njs_v return ret; } + keys = NULL; + if (length < prev_length) { keys = njs_array_indices(vm, value); if (njs_slow_path(keys == NULL)) { @@ -210,7 +225,7 @@ njs_array_length_set(njs_vm_t *vm, njs_v ret = njs_value_property_delete(vm, value, &keys->start[i], NULL); if (njs_slow_path(ret == NJS_ERROR)) { - return ret; + goto done; } } } while (i-- != 0); @@ -222,7 +237,15 @@ njs_array_length_set(njs_vm_t *vm, njs_v return ret; } - return NJS_OK; + ret = NJS_OK; + +done: + + if (keys != NULL) { + njs_array_destroy(vm, keys); + } + + return ret; } @@ -341,7 +364,8 @@ njs_array_constructor(njs_vm_t *vm, njs_ args = NULL; } - array = njs_array_alloc(vm, size, NJS_ARRAY_SPARE); + array = njs_array_alloc(vm, size <= NJS_ARRAY_FLAT_MAX_LENGTH, + size, NJS_ARRAY_SPARE); if (njs_fast_path(array != NULL)) { @@ -401,7 +425,7 @@ njs_array_of(njs_vm_t *vm, njs_value_t * length = nargs > 1 ? nargs - 1 : 0; - array = njs_array_alloc(vm, length, NJS_ARRAY_SPARE); + array = njs_array_alloc(vm, 0, length, NJS_ARRAY_SPARE); if (njs_slow_path(array == NULL)) { return NJS_ERROR; } @@ -639,12 +663,14 @@ njs_array_prototype_slice_copy(njs_vm_t njs_slice_prop_t string_slice; njs_string_prop_t string; - array = njs_array_alloc(vm, length, NJS_ARRAY_SPARE); + keys = NULL; + array = njs_array_alloc(vm, 0, length, NJS_ARRAY_SPARE); if (njs_slow_path(array == NULL)) { return NJS_ERROR; } if (njs_slow_path(length == 0)) { + ret = NJS_OK; goto done; } @@ -744,6 +770,7 @@ njs_array_prototype_slice_copy(njs_vm_t } while (length != 0); } + ret = NJS_OK; goto done; } @@ -768,6 +795,7 @@ njs_array_prototype_slice_copy(njs_vm_t length--; } while (length != 0); + ret = NJS_OK; goto done; } @@ -779,21 +807,27 @@ njs_array_prototype_slice_copy(njs_vm_t for (n = 0; n < keys->length; n++) { ret = njs_value_property(vm, this, &keys->start[n], &retval); if (njs_slow_path(ret == NJS_ERROR)) { - return ret; + goto done; } ret = njs_value_property_set(vm, &array_value, &keys->start[n], &retval); if (njs_slow_path(ret == NJS_ERROR)) { - return ret; + goto done; } } + ret = NJS_OK; + done: + if (keys != NULL) { + njs_array_destroy(vm, keys); + } + njs_set_array(&vm->retval, array); - return NJS_OK; + return ret; } @@ -992,6 +1026,7 @@ njs_array_prototype_unshift(njs_vm_t *vm ret = njs_value_property_delete(vm, this, &keys->start[--from], &entry); if (njs_slow_path(ret == NJS_ERROR)) { + njs_array_destroy(vm, keys); return ret; } @@ -1002,11 +1037,14 @@ njs_array_prototype_unshift(njs_vm_t *vm ret = njs_value_property_set(vm, this, &index, &entry); if (njs_slow_path(ret == NJS_ERROR)) { + njs_array_destroy(vm, keys); return ret; } } } + njs_array_destroy(vm, keys); + length += nargs - 1; goto copy; @@ -1208,7 +1246,7 @@ njs_array_prototype_splice(njs_vm_t *vm, } } - deleted = njs_array_alloc(vm, delete, 0); + deleted = njs_array_alloc(vm, 0, delete, 0); if (njs_slow_path(deleted == NULL)) { return NJS_ERROR; } @@ -1711,10 +1749,13 @@ process_object: ret = njs_array_object_handler(vm, handler, args, &keys->start[i], idx); if (njs_slow_path(ret != NJS_OK)) { + njs_array_destroy(vm, keys); return ret; } } + njs_array_destroy(vm, keys); + return NJS_OK; } @@ -1867,10 +1908,13 @@ process_object: ret = njs_array_object_handler(vm, handler, args, &keys->start[i], idx); if (njs_slow_path(ret != NJS_OK)) { + njs_array_destroy(vm, keys); return ret; } } + njs_array_destroy(vm, keys); + return NJS_OK; } @@ -1934,7 +1978,7 @@ njs_array_prototype_concat(njs_vm_t *vm, /* TODO: ArraySpeciesCreate(). */ - array = njs_array_alloc(vm, 0, NJS_ARRAY_SPARE); + array = njs_array_alloc(vm, 0, 0, NJS_ARRAY_SPARE); if (njs_slow_path(array == NULL)) { return NJS_ERROR; } @@ -2035,11 +2079,14 @@ njs_array_prototype_concat(njs_vm_t *vm, ret = njs_value_property_set(vm, &this, &index, &retval); if (njs_slow_path(ret == NJS_ERROR)) { + njs_array_destroy(vm, keys); return ret; } } } + njs_array_destroy(vm, keys); + length += len; continue; @@ -2589,7 +2636,7 @@ njs_array_prototype_filter(njs_vm_t *vm, return ret; } - iargs.array = njs_array_alloc(vm, 0, NJS_ARRAY_SPARE); + iargs.array = njs_array_alloc(vm, 0, 0, NJS_ARRAY_SPARE); if (njs_slow_path(iargs.array == NULL)) { return NJS_ERROR; } @@ -2779,7 +2826,7 @@ njs_array_prototype_map(njs_vm_t *vm, nj goto unexpected_args; } - iargs.array = njs_array_alloc(vm, length, 0); + iargs.array = njs_array_alloc(vm, 0, length, 0); if (njs_slow_path(iargs.array == NULL)) { return NJS_ERROR; } diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_array.h --- a/src/njs_array.h Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_array.h Mon Feb 10 17:09:53 2020 +0300 @@ -16,8 +16,11 @@ #define NJS_ARRAY_MAX_LENGTH53 (0x1fffffffffffff) #define NJS_ARRAY_FAST_OBJECT_LENGTH (128) #define NJS_ARRAY_LARGE_OBJECT_LENGTH (32768) +#define NJS_ARRAY_FLAT_MAX_LENGTH (1048576) -njs_array_t *njs_array_alloc(njs_vm_t *vm, uint64_t length, uint32_t spare); +njs_array_t *njs_array_alloc(njs_vm_t *vm, njs_bool_t flat, uint64_t length, + uint32_t spare); +void njs_array_destroy(njs_vm_t *vm, njs_array_t *array); njs_int_t njs_array_add(njs_vm_t *vm, njs_array_t *array, njs_value_t *value); njs_int_t njs_array_convert_to_slow_array(njs_vm_t *vm, njs_array_t *array); njs_int_t njs_array_length_redefine(njs_vm_t *vm, njs_value_t *value, diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_builtin.c --- a/src/njs_builtin.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_builtin.c Mon Feb 10 17:09:53 2020 +0300 @@ -1511,7 +1511,7 @@ njs_process_object_argv(njs_vm_t *vm, nj static const njs_value_t argv_string = njs_string("argv"); - argv = njs_array_alloc(vm, vm->options.argc, 0); + argv = njs_array_alloc(vm, 1, vm->options.argc, 0); if (njs_slow_path(argv == NULL)) { return NJS_ERROR; } diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_extern.c --- a/src/njs_extern.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_extern.c Mon Feb 10 17:09:53 2020 +0300 @@ -234,7 +234,7 @@ njs_extern_keys_array(njs_vm_t *vm, cons keys_length++; } - keys = njs_array_alloc(vm, keys_length, NJS_ARRAY_SPARE); + keys = njs_array_alloc(vm, 1, keys_length, NJS_ARRAY_SPARE); if (njs_slow_path(keys == NULL)) { return NULL; } diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_function.c --- a/src/njs_function.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_function.c Mon Feb 10 17:09:53 2020 +0300 @@ -301,7 +301,7 @@ njs_function_rest_parameters_init(njs_vm n = frame->function->u.lambda->nargs; length = (nargs >= n) ? (nargs - n + 1) : 0; - array = njs_array_alloc(vm, length, 0); + array = njs_array_alloc(vm, 1, length, 0); if (njs_slow_path(array == NULL)) { return NJS_ERROR; } @@ -1125,7 +1125,7 @@ njs_function_prototype_apply(njs_vm_t *v return ret; } - arr = njs_array_alloc(vm, length, NJS_ARRAY_SPARE); + arr = njs_array_alloc(vm, 1, length, NJS_ARRAY_SPARE); if (njs_slow_path(arr == NULL)) { return NJS_ERROR; } diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_json.c --- a/src/njs_json.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_json.c Mon Feb 10 17:09:53 2020 +0300 @@ -31,6 +31,7 @@ typedef struct { uint32_t index; uint32_t length; njs_array_t *keys; + njs_value_t *key; njs_object_prop_t *prop; } njs_json_state_t; @@ -448,7 +449,7 @@ njs_json_parse_array(njs_json_parse_ctx_ return NULL; } - array = njs_array_alloc(ctx->vm, 0, 0); + array = njs_array_alloc(ctx->vm, 0, 0, NJS_ARRAY_SPARE); if (njs_slow_path(array == NULL)) { return NULL; } @@ -1027,6 +1028,8 @@ njs_json_push_stringify_state(njs_vm_t * state->value = *value; state->index = 0; state->written = 0; + state->keys = NULL; + state->key = NULL; if (njs_is_fast_array(value)) { state->type = NJS_JSON_ARRAY; @@ -1069,6 +1072,12 @@ njs_json_pop_stringify_state(njs_json_st { njs_json_state_t *state; + state = &stringify->states[stringify->depth - 1]; + if (!njs_is_array(&stringify->replacer) && state->keys != NULL) { + njs_array_destroy(stringify->vm, state->keys); + state->keys = NULL; + } + if (stringify->depth > 1) { stringify->depth--; state = &stringify->states[stringify->depth - 1]; @@ -1411,7 +1420,7 @@ njs_json_stringify_replacer(njs_json_str static njs_int_t -njs_json_stringify_array(njs_vm_t *vm, njs_json_stringify_t *stringify) +njs_json_stringify_array(njs_vm_t *vm, njs_json_stringify_t *stringify) { njs_int_t ret; uint32_t i, n, k, properties_length, array_length; @@ -1428,7 +1437,7 @@ njs_json_stringify_array(njs_vm_t *vm, n } } - properties = njs_array_alloc(vm, properties_length, NJS_ARRAY_SPARE); + properties = njs_array_alloc(vm, 1, properties_length, NJS_ARRAY_SPARE); if (njs_slow_path(properties == NULL)) { return NJS_ERROR; } @@ -2086,7 +2095,6 @@ njs_vm_value_dump(njs_vm_t *vm, njs_str_ goto memory_error; } - key = NULL; (void) njs_dump_visit(&visited, value); for ( ;; ) { @@ -2106,12 +2114,11 @@ njs_vm_value_dump(njs_vm_t *vm, njs_str_ njs_chb_append(&chain, state->array ? "[" : "{", 1); njs_json_stringify_indent(stringify, &chain, 1); - } if (state->index >= state->keys->length) { njs_dump_empty(stringify, state, &chain, state->length, - (state->index > 0) ? njs_key_to_index(key) : -1, 0); + (state->index > 0) ? njs_key_to_index(state->key) : -1, 0); njs_json_stringify_indent(stringify, &chain, 0); njs_chb_append(&chain, state->array ? "]" : "}", 1); @@ -2127,6 +2134,7 @@ njs_vm_value_dump(njs_vm_t *vm, njs_str_ njs_property_query_init(&pq, NJS_PROPERTY_QUERY_GET, 0); key = &state->keys->start[state->index++]; + state->key = key; ret = njs_property_query(vm, &pq, &state->value, key); if (njs_slow_path(ret != NJS_OK)) { diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_object.c --- a/src/njs_object.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_object.c Mon Feb 10 17:09:53 2020 +0300 @@ -584,16 +584,11 @@ njs_object_enumerate(njs_vm_t *vm, const length = njs_object_enumerate_length(object, type, all); - items = njs_array_alloc(vm, length, NJS_ARRAY_SPARE); + items = njs_array_alloc(vm, 1, length, NJS_ARRAY_SPARE); if (njs_slow_path(items == NULL)) { return NULL; } - if (njs_slow_path(!items->object.fast_array)) { - njs_internal_error(vm, "njs_object_enumerate() too many keys"); - return NULL; - } - ret = njs_object_enumerate_value(vm, object, items, kind, type, all); if (njs_slow_path(ret != NJS_OK)) { return NULL; @@ -615,16 +610,11 @@ njs_object_own_enumerate(njs_vm_t *vm, c length = njs_object_own_enumerate_length(object, object, type, all); - items = njs_array_alloc(vm, length, NJS_ARRAY_SPARE); + items = njs_array_alloc(vm, 1, length, NJS_ARRAY_SPARE); if (njs_slow_path(items == NULL)) { return NULL; } - if (njs_slow_path(!items->object.fast_array)) { - njs_internal_error(vm, "njs_object_own_enumerate() too many keys"); - return NULL; - } - ret = njs_object_own_enumerate_value(vm, object, object, items, kind, type, all); if (njs_slow_path(ret != NJS_OK)) { @@ -804,7 +794,7 @@ njs_object_enumerate_array(njs_vm_t *vm, for (i = 0; i < array->length; i++) { if (njs_is_valid(&array->start[i])) { - entry = njs_array_alloc(vm, 2, 0); + entry = njs_array_alloc(vm, 0, 2, 0); if (njs_slow_path(entry == NULL)) { return NJS_ERROR; } @@ -857,7 +847,7 @@ njs_object_enumerate_typed_array(njs_vm_ case NJS_ENUM_BOTH: for (i = 0; i < length; i++) { - entry = njs_array_alloc(vm, 2, 0); + entry = njs_array_alloc(vm, 0, 2, 0); if (njs_slow_path(entry == NULL)) { return NJS_ERROR; } @@ -938,7 +928,7 @@ njs_object_enumerate_string(njs_vm_t *vm for (i = 0; i < len; i++) { - entry = njs_array_alloc(vm, 2, 0); + entry = njs_array_alloc(vm, 0, 2, 0); if (njs_slow_path(entry == NULL)) { return NJS_ERROR; } @@ -965,7 +955,7 @@ njs_object_enumerate_string(njs_vm_t *vm i = 0; do { - entry = njs_array_alloc(vm, 2, 0); + entry = njs_array_alloc(vm, 0, 2, 0); if (njs_slow_path(entry == NULL)) { return NJS_ERROR; } @@ -1171,7 +1161,7 @@ njs_object_own_enumerate_object(njs_vm_t if (ext_prop == NULL && prop->type != NJS_WHITEOUT && (prop->enumerable || all)) { - entry = njs_array_alloc(vm, 2, 0); + entry = njs_array_alloc(vm, 0, 2, 0); if (njs_slow_path(entry == NULL)) { return NJS_ERROR; } @@ -1209,7 +1199,7 @@ njs_object_own_enumerate_object(njs_vm_t ext_prop = njs_object_exist_in_proto(parent, object, &lhq); if (ext_prop == NULL) { - entry = njs_array_alloc(vm, 2, 0); + entry = njs_array_alloc(vm, 0, 2, 0); if (njs_slow_path(entry == NULL)) { return NJS_ERROR; } @@ -1442,7 +1432,7 @@ njs_object_define_properties(njs_vm_t *v for (i = 0; i < length; i++) { ret = njs_property_query(vm, &pq, descs, &keys->start[i]); if (njs_slow_path(ret == NJS_ERROR)) { - return ret; + goto done; } prop = pq.lhq.value; @@ -1453,19 +1443,24 @@ njs_object_define_properties(njs_vm_t *v ret = njs_value_property(vm, descs, &keys->start[i], &desc); if (njs_slow_path(ret == NJS_ERROR)) { - return ret; + goto done; } ret = njs_object_prop_define(vm, value, &keys->start[i], &desc, NJS_OBJECT_PROP_DESCRIPTOR); if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; + goto done; } } + ret = NJS_OK; vm->retval = *value; - return NJS_OK; +done: + + njs_array_destroy(vm, keys); + + return ret; } @@ -1520,7 +1515,8 @@ njs_object_get_own_property_descriptors( descriptors = njs_object_alloc(vm); if (njs_slow_path(descriptors == NULL)) { - return NJS_ERROR; + ret = NJS_ERROR; + goto done; } lhq.replace = 0; @@ -1531,12 +1527,14 @@ njs_object_get_own_property_descriptors( key = &names->start[i]; ret = njs_object_prop_descriptor(vm, &descriptor, value, key); if (njs_slow_path(ret != NJS_OK)) { - return ret; + ret = NJS_ERROR; + goto done; } pr = njs_object_prop_alloc(vm, key, &descriptor, 1); if (njs_slow_path(pr == NULL)) { - return NJS_ERROR; + ret = NJS_ERROR; + goto done; } njs_object_property_key_set(&lhq, key, 0); @@ -1545,13 +1543,18 @@ njs_object_get_own_property_descriptors( ret = njs_lvlhsh_insert(&descriptors->hash, &lhq); if (njs_slow_path(ret != NJS_OK)) { njs_internal_error(vm, "lvlhsh insert failed"); - return NJS_ERROR; + goto done; } } + ret = NJS_OK; njs_set_object(&vm->retval, descriptors); - return NJS_OK; +done: + + njs_array_destroy(vm, names); + + return ret; } @@ -1912,6 +1915,8 @@ njs_object_assign(njs_vm_t *vm, njs_valu return ret; } + names = NULL; + for (i = 2; i < nargs; i++) { source = &args[i]; @@ -1930,7 +1935,7 @@ njs_object_assign(njs_vm_t *vm, njs_valu ret = njs_property_query(vm, &pq, source, key); if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; + goto exception; } prop = pq.lhq.value; @@ -1940,19 +1945,27 @@ njs_object_assign(njs_vm_t *vm, njs_valu ret = njs_value_property(vm, source, key, &setval); if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; + goto exception; } ret = njs_value_property_set(vm, value, key, &setval); if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; + goto exception; } } + + njs_array_destroy(vm, names); } vm->retval = *value; return NJS_OK; + +exception: + + njs_array_destroy(vm, names); + + return NJS_ERROR; } diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_regexp.c --- a/src/njs_regexp.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_regexp.c Mon Feb 10 17:09:53 2020 +0300 @@ -1072,7 +1072,7 @@ njs_regexp_exec_result(njs_vm_t *vm, njs static const njs_value_t string_input = njs_string("input"); static const njs_value_t string_groups = njs_string("groups"); - array = njs_array_alloc(vm, regexp->pattern->ncaptures, 0); + array = njs_array_alloc(vm, 0, regexp->pattern->ncaptures, 0); if (njs_slow_path(array == NULL)) { goto fail; } diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_string.c --- a/src/njs_string.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_string.c Mon Feb 10 17:09:53 2020 +0300 @@ -3237,7 +3237,7 @@ njs_string_match_multiple(njs_vm_t *vm, if (njs_regex_is_valid(&pattern->regex[type])) { - array = njs_array_alloc(vm, 0, NJS_ARRAY_SPARE); + array = njs_array_alloc(vm, 0, 0, NJS_ARRAY_SPARE); if (njs_slow_path(array == NULL)) { return NJS_ERROR; } @@ -3330,7 +3330,7 @@ njs_string_prototype_split(njs_vm_t *vm, return ret; } - array = njs_array_alloc(vm, 0, NJS_ARRAY_SPARE); + array = njs_array_alloc(vm, 0, 0, NJS_ARRAY_SPARE); if (njs_slow_path(array == NULL)) { return NJS_ERROR; } diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_value.c --- a/src/njs_value.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_value.c Mon Feb 10 17:09:53 2020 +0300 @@ -228,7 +228,7 @@ njs_value_enumerate(njs_vm_t *vm, const return njs_extern_keys_array(vm, ext_proto); } - return njs_array_alloc(vm, 0, NJS_ARRAY_SPARE); + return njs_array_alloc(vm, 1, 0, NJS_ARRAY_SPARE); } obj_val.object = vm->string_object; @@ -273,7 +273,7 @@ njs_value_own_enumerate(njs_vm_t *vm, co return njs_extern_keys_array(vm, ext_proto); } - return njs_array_alloc(vm, 0, NJS_ARRAY_SPARE); + return njs_array_alloc(vm, 1, 0, NJS_ARRAY_SPARE); } obj_val.object = vm->string_object; diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_vm.c --- a/src/njs_vm.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_vm.c Mon Feb 10 17:09:53 2020 +0300 @@ -805,7 +805,7 @@ njs_vm_array_alloc(njs_vm_t *vm, njs_val { njs_array_t *array; - array = njs_array_alloc(vm, 0, spare); + array = njs_array_alloc(vm, 0, 0, spare); if (njs_slow_path(array == NULL)) { return NJS_ERROR; diff -r 1f3e045ea017 -r 79c14715edc2 src/njs_vmcode.c --- a/src/njs_vmcode.c Sun Jan 26 21:56:23 2020 +0300 +++ b/src/njs_vmcode.c Mon Feb 10 17:09:53 2020 +0300 @@ -966,7 +966,7 @@ njs_vmcode_array(njs_vm_t *vm, u_char *p code = (njs_vmcode_array_t *) pc; - array = njs_array_alloc(vm, code->length, NJS_ARRAY_SPARE); + array = njs_array_alloc(vm, 0, code->length, NJS_ARRAY_SPARE); if (njs_fast_path(array != NULL)) { From xeioex at nginx.com Mon Feb 10 16:02:01 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Mon, 10 Feb 2020 16:02:01 +0000 Subject: [njs] Filtering out integer indices in njs_object_completions(). Message-ID: details: https://hg.nginx.org/njs/rev/5bd15bd3766c branches: changeset: 1324:5bd15bd3766c user: Dmitry Volyntsev date: Mon Feb 10 17:39:41 2020 +0300 description: Filtering out integer indices in njs_object_completions(). This closes #285 issue on Github. diffstat: src/njs_builtin.c | 123 +++++++++++++--------------------------------- test/njs_expect_test.exp | 21 ++++++++ 2 files changed, 57 insertions(+), 87 deletions(-) diffs (189 lines): diff -r 79c14715edc2 -r 5bd15bd3766c src/njs_builtin.c --- a/src/njs_builtin.c Mon Feb 10 17:09:53 2020 +0300 +++ b/src/njs_builtin.c Mon Feb 10 17:39:41 2020 +0300 @@ -24,7 +24,7 @@ typedef struct { static njs_arr_t *njs_vm_expression_completions(njs_vm_t *vm, njs_str_t *expression); -static njs_arr_t *njs_object_completions(njs_vm_t *vm, njs_object_t *object); +static njs_arr_t *njs_object_completions(njs_vm_t *vm, njs_value_t *object); static njs_int_t njs_env_hash_init(njs_vm_t *vm, njs_lvlhsh_t *hash, char **environment); @@ -633,110 +633,59 @@ njs_vm_expression_completions(njs_vm_t * value = &prop->value; } - return njs_object_completions(vm, njs_object(value)); + return njs_object_completions(vm, value); } static njs_arr_t * -njs_object_completions(njs_vm_t *vm, njs_object_t *object) +njs_object_completions(njs_vm_t *vm, njs_value_t *object) { - size_t size; - njs_str_t *compl; - njs_arr_t *completions; - njs_uint_t n, k; - njs_object_t *o; - njs_object_prop_t *prop; - njs_lvlhsh_each_t lhe; - - size = 0; - o = object; - - do { - njs_lvlhsh_each_init(&lhe, &njs_object_hash_proto); - - for ( ;; ) { - prop = njs_lvlhsh_each(&o->hash, &lhe); - if (prop == NULL) { - break; - } + double num; + njs_arr_t *array; + njs_str_t *completion; + njs_uint_t n; + njs_array_t *keys; + njs_value_type_t type; - size++; - } - - njs_lvlhsh_each_init(&lhe, &njs_object_hash_proto); - - for ( ;; ) { - prop = njs_lvlhsh_each(&o->shared_hash, &lhe); - if (prop == NULL) { - break; - } + array = NULL; + type = object->type; - size++; - } - - o = o->__proto__; + if (type == NJS_ARRAY || type == NJS_TYPED_ARRAY) { + object->type = NJS_OBJECT; + } - } while (o != NULL); - - completions = njs_arr_create(vm->mem_pool, size, sizeof(njs_str_t)); - if (njs_slow_path(completions == NULL)) { - return NULL; + keys = njs_value_enumerate(vm, object, NJS_ENUM_KEYS, NJS_ENUM_STRING, 1); + if (njs_slow_path(keys == NULL)) { + goto done; } - n = 0; - o = object; - compl = completions->start; - - do { - njs_lvlhsh_each_init(&lhe, &njs_object_hash_proto); + array = njs_arr_create(vm->mem_pool, 8, sizeof(njs_str_t)); + if (njs_slow_path(array == NULL)) { + goto done; + } - for ( ;; ) { - prop = njs_lvlhsh_each(&o->hash, &lhe); - if (prop == NULL) { - break; - } + for (n = 0; n < keys->length; n++) { + num = njs_key_to_index(&keys->start[n]); - njs_string_get(&prop->name, &compl[n]); - - for (k = 0; k < n; k++) { - if (njs_strstr_eq(&compl[k], &compl[n])) { - break; - } + if (!njs_key_is_integer_index(num, &keys->start[n])) { + completion = njs_arr_add(array); + if (njs_slow_path(completion == NULL)) { + njs_arr_destroy(array); + array = NULL; + goto done; } - if (k == n) { - n++; - } + njs_string_get(&keys->start[n], completion); } - - njs_lvlhsh_each_init(&lhe, &njs_object_hash_proto); + } - for ( ;; ) { - prop = njs_lvlhsh_each(&o->shared_hash, &lhe); - if (prop == NULL) { - break; - } - - njs_string_get(&prop->name, &compl[n]); +done: - for (k = 0; k < n; k++) { - if (njs_strstr_eq(&compl[k], &compl[n])) { - break; - } - } + if (type == NJS_ARRAY || type == NJS_TYPED_ARRAY) { + object->type = type; + } - if (k == n) { - n++; - } - } - - o = o->__proto__; - - } while (o != NULL); - - completions->items = n; - - return completions; + return array; } diff -r 79c14715edc2 -r 5bd15bd3766c test/njs_expect_test.exp --- a/test/njs_expect_test.exp Mon Feb 10 17:09:53 2020 +0300 +++ b/test/njs_expect_test.exp Mon Feb 10 17:39:41 2020 +0300 @@ -192,6 +192,27 @@ njs_test { "o.a.toDateString*o.a.toLocaleDateString*o.a.toString"} } +njs_test { + {"var o = {a:1,b:2,333:'t'}\r\n" + "var o = {a:1,b:2,333:'t'}\r\nundefined\r\n>> "} + {"o.3\t\t" + "o.3"} +} + +njs_test { + {"var a = Array(5000000); a.aab = 1; a.aac = 2\r\n" + "var a = Array(5000000); a.aab = 1; a.aac = 2\r\n2\r\n>> "} + {"a.\t\t" + "a.aab*"} +} + +njs_test { + {"var a = new Uint8Array([5,6,7,8,8]); a.aab = 1; a.aac = 2\r\n" + "var a = new Uint8Array(\\\[5,6,7,8,8]); a.aab = 1; a.aac = 2\r\n2\r\n>> "} + {"a.\t\t" + "a.aab*"} +} + # function declarations in interactive mode njs_test { {"function a() { return 1; }\r\n" From yshpakov at hotmail.com Wed Feb 12 02:01:57 2020 From: yshpakov at hotmail.com (Yury Shpakov) Date: Wed, 12 Feb 2020 02:01:57 +0000 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: Message-ID: Hi there, Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: http://nginx.org/en/docs/howto_build_on_win32.html But excluded (don't care about SSL at this point so don't want to install/configure Perl now): --with-openssl=objs/lib/openssl-master \ --with-openssl-opt=no-asm \ --with-http_ssl_module \ And added: --with-mail nmake was successful and nginx.exe was created. However nginx.exe keeps failing with the error: WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Windows API says the following about this error: WSAENOTCONN 10057 Socket is not connected. A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using sendto) no address was supplied. Any other type of operation might also return this error?for example, setsockopt setting SO_KEEPALIVE if the connection has been reset. https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. docs.microsoft.com Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. Any recommendation for me to make it work? Tried to play with config (commenting/uncommenting): #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. Thank you, Yury -------------- next part -------------- An HTML attachment was scrubbed... URL: From serg.brester at sebres.de Wed Feb 12 12:38:13 2020 From: serg.brester at sebres.de (Sergey Brester) Date: Wed, 12 Feb 2020 13:38:13 +0100 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: Message-ID: <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... try netstat (in cmd as admin): netstat /nabo netstat /nabo | grep -A 1 ":9000b" and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). additionally try to telnet or curl it: curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [7]). But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [8] Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH [9] - nginx.zip [10], where it works well). Regards, Sergey 12.02.2020 03:01, Yury Shpakov wrote: > Hi there, > > Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: > http://nginx.org/en/docs/howto_build_on_win32.html [2] > > But excluded (don't care about SSL at this point so don't want to install/configure Perl now): > --with-openssl=objs/lib/openssl-master > > --with-openssl-opt=no-asm > --with-http_ssl_module > And added: > --with-mail > > nmake was successful and nginx.exe was created. > > However nginx.exe keeps failing with the error: > WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 > > Windows API says the following about this error: > > WSAENOTCONN10057 > Socket is not connected.A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using SENDTO [3]) no address was supplied. Any other type of operation might also return this error--for example, SETSOCKOPT [4] setting SO_KEEPALIVE [5] if the connection has been reset. > > https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 [6] > > Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs [6] > Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. > docs.microsoft.com > > Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. > > Any recommendation for me to make it work? > > Tried to play with config (commenting/uncommenting): > > #user nobody; > worker_processes 1; > > #error_log logs/error.log; > #error_log logs/error.log notice; > #error_log logs/error.log info; > > #pid logs/nginx.pid; > > events { > worker_connections 1024; > } > > mail { > server_name localhost; > auth_http localhost:9000/cgi-bin/nginxauth.cgi; > # auth_http none; > > smtp_auth none; > # smtp_auth login plain cram-md5; > # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; > xclient off; > > server { > listen 8025; > protocol smtp; > proxy on; > proxy_pass_error_message on; > } > } > Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. > > Thank you, > Yury > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel [1] Links: ------ [1] http://mailman.nginx.org/mailman/listinfo/nginx-devel [2] http://nginx.org/en/docs/howto_build_on_win32.html [3] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-sendto [4] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-setsockopt [5] https://docs.microsoft.com/en-us/windows/desktop/winsock/so-keepalive [6] https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 [7] http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [8] https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [9] https://github.com/sebres/nginx/releases/tag/release-1.13.0 [10] https://github.com/sebres/nginx/files/2246440/nginx.zip -------------- next part -------------- An HTML attachment was scrubbed... URL: From vl at nginx.com Wed Feb 12 16:02:53 2020 From: vl at nginx.com (Vladimir Homutov) Date: Wed, 12 Feb 2020 16:02:53 +0000 Subject: [nginx] Made ngx_http_get_forwarded_addr_internal() non-recursive. Message-ID: details: https://hg.nginx.org/nginx/rev/1055e43e4fab branches: changeset: 7624:1055e43e4fab user: Vladimir Homutov date: Tue Feb 11 13:22:44 2020 +0300 description: Made ngx_http_get_forwarded_addr_internal() non-recursive. diffstat: src/http/ngx_http_core_module.c | 62 +++++++++++++++++++--------------------- 1 files changed, 30 insertions(+), 32 deletions(-) diffs (77 lines): diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -2667,43 +2667,41 @@ ngx_http_get_forwarded_addr_internal(ngx u_char *xff, size_t xfflen, ngx_array_t *proxies, int recursive) { u_char *p; - ngx_int_t rc; ngx_addr_t paddr; - - if (ngx_cidr_match(addr->sockaddr, proxies) != NGX_OK) { - return NGX_DECLINED; - } - - for (p = xff + xfflen - 1; p > xff; p--, xfflen--) { - if (*p != ' ' && *p != ',') { - break; + ngx_uint_t found; + + found = 0; + + do { + + if (ngx_cidr_match(addr->sockaddr, proxies) != NGX_OK) { + return found ? NGX_DONE : NGX_DECLINED; } - } - - for ( /* void */ ; p > xff; p--) { - if (*p == ' ' || *p == ',') { - p++; - break; + + for (p = xff + xfflen - 1; p > xff; p--, xfflen--) { + if (*p != ' ' && *p != ',') { + break; + } } - } - - if (ngx_parse_addr_port(r->pool, &paddr, p, xfflen - (p - xff)) != NGX_OK) { - return NGX_DECLINED; - } - - *addr = paddr; - - if (recursive && p > xff) { - rc = ngx_http_get_forwarded_addr_internal(r, addr, xff, p - 1 - xff, - proxies, 1); - - if (rc == NGX_DECLINED) { - return NGX_DONE; + + for ( /* void */ ; p > xff; p--) { + if (*p == ' ' || *p == ',') { + p++; + break; + } } - /* rc == NGX_OK || rc == NGX_DONE */ - return rc; - } + if (ngx_parse_addr_port(r->pool, &paddr, p, xfflen - (p - xff)) + != NGX_OK) + { + return found ? NGX_DONE : NGX_DECLINED; + } + + *addr = paddr; + found = 1; + xfflen = p - 1 - xff; + + } while (recursive && p > xff); return NGX_OK; } From yshpakov at hotmail.com Wed Feb 12 17:59:30 2020 From: yshpakov at hotmail.com (Yury Shpakov) Date: Wed, 12 Feb 2020 17:59:30 +0000 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> Message-ID: Hi Sergey, Thank you for you response. I tried netstat /nabo and I don't see any reference to port 9000 at all. So a problem is to make nginx to listen on port 9000 (as server)? Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 7:38 AM To: nginx-devel at nginx.org Cc: Yury Shpakov Subject: Re: nginx for Windows - WSASend() socket error 10057 It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... try netstat (in cmd as admin): netstat /nabo netstat /nabo | grep -A 1 ":9000\b" and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). additionally try to telnet or curl it: curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html). But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH - nginx.zip, where it works well). Regards, Sergey 12.02.2020 03:01, Yury Shpakov wrote: Hi there, Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: http://nginx.org/en/docs/howto_build_on_win32.html But excluded (don't care about SSL at this point so don't want to install/configure Perl now): --with-openssl=objs/lib/openssl-master \ --with-openssl-opt=no-asm \ --with-http_ssl_module \ And added: --with-mail nmake was successful and nginx.exe was created. However nginx.exe keeps failing with the error: WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Windows API says the following about this error: WSAENOTCONN 10057 Socket is not connected. A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using sendto) no address was supplied. Any other type of operation might also return this error?for example, setsockopt setting SO_KEEPALIVE if the connection has been reset. https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. docs.microsoft.com Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. Any recommendation for me to make it work? Tried to play with config (commenting/uncommenting): #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. Thank you, Yury _______________________________________________ nginx-devel mailing list nginx-devel at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From serg.brester at sebres.de Wed Feb 12 18:51:27 2020 From: serg.brester at sebres.de (Sergey Brester) Date: Wed, 12 Feb 2020 19:51:27 +0100 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> Message-ID: <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> I answered inline... 12.02.2020 18:59, Yury Shpakov wrote: > Hi Sergey, > > Thank you for you response. > > I tried netstat /nabo and I don't see any reference to port 9000 at all. > So a problem is to make nginx to listen on port 9000 (as server)? > Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ [11] for more examples. > Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy [12] - you can use some nginx location (and internal URL to same nginx instance) to specify that. Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. > At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. > And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. > > And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link [9]) But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). > Thank you, > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Wednesday, February 12, 2020 7:38 AM > TO: nginx-devel at nginx.org > CC: Yury Shpakov > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... > > try netstat (in cmd as admin): > > netstat /nabo > netstat /nabo | grep -A 1 ":9000b" > > and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). > > additionally try to telnet or curl it: > > curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi > > if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). > > If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [7]). > > But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. > It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). > > Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [8] > > Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH [9] - nginx.zip [10], where it works well). > > Regards, > Sergey > > 12.02.2020 03:01, Yury Shpakov wrote: > >> Hi there, >> >> Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: >> http://nginx.org/en/docs/howto_build_on_win32.html [2] >> But excluded (don't care about SSL at this point so don't want to install/configure Perl now): >> --with-openssl=objs/lib/openssl-master >> >> --with-openssl-opt=no-asm >> --with-http_ssl_module >> And added: >> --with-mail >> >> nmake was successful and nginx.exe was created. >> However nginx.exe keeps failing with the error: >> WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 >> Windows API says the following about this error: >> >> WSAENOTCONN10057 >> Socket is not connected.A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using SENDTO [3]) no address was supplied. Any other type of operation might also return this error--for example, SETSOCKOPT [4] setting SO_KEEPALIVE [5] if the connection has been reset. >> >> https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 [6] >> >> Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs [6] >> Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. >> docs.microsoft.com >> >> Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. >> Any recommendation for me to make it work? >> Tried to play with config (commenting/uncommenting): >> >> #user nobody; >> worker_processes 1; >> #error_log logs/error.log; >> #error_log logs/error.log notice; >> #error_log logs/error.log info; >> #pid logs/nginx.pid; >> events { >> worker_connections 1024; >> } >> mail { >> server_name localhost; >> auth_http localhost:9000/cgi-bin/nginxauth.cgi; >> # auth_http none; >> smtp_auth none; >> # smtp_auth login plain cram-md5; >> # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; >> xclient off; >> server { >> listen 8025; >> protocol smtp; >> proxy on; >> proxy_pass_error_message on; >> } >> } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. >> Thank you, >> Yury >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel [1] Links: ------ [1] http://mailman.nginx.org/mailman/listinfo/nginx-devel [2] http://nginx.org/en/docs/howto_build_on_win32.html [3] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-sendto [4] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-setsockopt [5] https://docs.microsoft.com/en-us/windows/desktop/winsock/so-keepalive [6] https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 [7] http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [8] https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [9] https://github.com/sebres/nginx/releases/tag/release-1.13.0 [10] https://github.com/sebres/nginx/files/2246440/nginx.zip [11] https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ [12] https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy -------------- next part -------------- An HTML attachment was scrubbed... URL: From maksim.yevmenkin at gmail.com Thu Feb 13 02:49:15 2020 From: maksim.yevmenkin at gmail.com (Maksim Yevmenkin) Date: Wed, 12 Feb 2020 18:49:15 -0800 Subject: Nginx tail module Message-ID: Hello What would be the best way to write a Nginx module that would essentially tail a file? Thanks Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From eran.kornblau at kaltura.com Thu Feb 13 07:07:36 2020 From: eran.kornblau at kaltura.com (Eran Kornblau) Date: Thu, 13 Feb 2020 07:07:36 +0000 Subject: Nginx tail module In-Reply-To: References: Message-ID: I don?t know what you?re trying to solve? but maybe you can just send a range request relative to the end, without any custom module. For example, ?Range: bytes=-1024? will return the last 1k of the resource. Eran From: nginx-devel On Behalf Of Maksim Yevmenkin Sent: Thursday, February 13, 2020 4:49 AM To: nginx-devel at nginx.org Subject: Nginx tail module Hello What would be the best way to write a Nginx module that would essentially tail a file? Thanks Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From maksim.yevmenkin at gmail.com Thu Feb 13 13:18:15 2020 From: maksim.yevmenkin at gmail.com (Maksim Yevmenkin) Date: Thu, 13 Feb 2020 05:18:15 -0800 Subject: Nginx tail module In-Reply-To: References: Message-ID: On Wed, Feb 12, 2020, 11:07 PM Eran Kornblau wrote: > I don?t know what you?re trying to solve? but maybe you can just send a > range request relative to the end, > without any custom module. > For example, ?Range: bytes=-1024? will return the last 1k of the resource. > Thanks but this is not what I was hoping for. The idea is to have Nginx return new data as soon as it is available in the tailed file. A client would issue one request and simply wait for the data to arrive in chunked transfer encoding. > What would be the best way to write a Nginx module that would essentially > tail a file? > Thanks Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.borisov at nginx.com Thu Feb 13 13:27:36 2020 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Thu, 13 Feb 2020 13:27:36 +0000 Subject: [njs] Fixed Object.getOwnPropertySymbols(). Message-ID: details: https://hg.nginx.org/njs/rev/974e6c195410 branches: changeset: 1325:974e6c195410 user: Alexander Borisov date: Thu Feb 13 16:25:37 2020 +0300 description: Fixed Object.getOwnPropertySymbols(). This closes #291 issue on GitHub. diffstat: src/njs_object.c | 4 ++-- src/test/njs_unit_test.c | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diffs (34 lines): diff -r 5bd15bd3766c -r 974e6c195410 src/njs_object.c --- a/src/njs_object.c Mon Feb 10 17:39:41 2020 +0300 +++ b/src/njs_object.c Thu Feb 13 16:25:37 2020 +0300 @@ -397,8 +397,6 @@ njs_object_exist_in_proto(const njs_obje njs_int_t ret; njs_object_prop_t *prop; - lhq->proto = &njs_object_hash_proto; - while (object != end) { ret = njs_lvlhsh_find(&object->hash, lhq); @@ -701,6 +699,8 @@ njs_object_own_enumerate_object_length(c njs_lvlhsh_each_init(&lhe, &njs_object_hash_proto); hash = &object->hash; + lhq.proto = &njs_object_hash_proto; + length = 0; for ( ;; ) { diff -r 5bd15bd3766c -r 974e6c195410 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Mon Feb 10 17:39:41 2020 +0300 +++ b/src/test/njs_unit_test.c Thu Feb 13 16:25:37 2020 +0300 @@ -11206,6 +11206,9 @@ static njs_unit_test_t njs_test[] = "delete obj[symA]"), njs_str("TypeError: Cannot delete property \"Symbol(A)\" of object") }, + { njs_str("typeof Object.getOwnPropertySymbols(globalThis);"), + njs_str("object") }, + { njs_str("[" " Object.prototype," " Symbol.prototype," From yshpakov at hotmail.com Thu Feb 13 21:45:39 2020 From: yshpakov at hotmail.com (Yury Shpakov) Date: Thu, 13 Feb 2020 21:45:39 +0000 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> , <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> Message-ID: Hi Sergey, I reconfigured the config file as follows: === === === #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.2; # backend ip add_header Auth-Port 143; # backend port return 204; } } } === === === And now it's responding on port 9000 as expected: === === === C:\WINDOWS\system32>curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi HTTP/1.1 204 No Content Server: nginx/1.17.9 Date: Thu, 13 Feb 2020 21:30:54 GMT Connection: keep-alive Auth-Status: OK Auth-Server: 127.0.0.2 Auth-Port: 143 === === === However I'm still experiencing the same issue (in log file): === === === 2020/02/13 16:29:24 [notice] 35048#26192: signal process started 2020/02/13 16:29:34 [error] 31732#22720: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 === === === Tried under both admin and regular user. Any further ideas how to get it fixed please? Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 1:51 PM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I answered inline... 12.02.2020 18:59, Yury Shpakov wrote: Hi Sergey, Thank you for you response. I tried netstat /nabo and I don't see any reference to port 9000 at all. So a problem is to make nginx to listen on port 9000 (as server)? Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ for more examples. Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy - you can use some nginx location (and internal URL to same nginx instance) to specify that. Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link) But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 7:38 AM To: nginx-devel at nginx.org Cc: Yury Shpakov Subject: Re: nginx for Windows - WSASend() socket error 10057 It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... try netstat (in cmd as admin): netstat /nabo netstat /nabo | grep -A 1 ":9000\b" and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). additionally try to telnet or curl it: curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html). But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH - nginx.zip, where it works well). Regards, Sergey 12.02.2020 03:01, Yury Shpakov wrote: Hi there, Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: http://nginx.org/en/docs/howto_build_on_win32.html But excluded (don't care about SSL at this point so don't want to install/configure Perl now): --with-openssl=objs/lib/openssl-master \ --with-openssl-opt=no-asm \ --with-http_ssl_module \ And added: --with-mail nmake was successful and nginx.exe was created. However nginx.exe keeps failing with the error: WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Windows API says the following about this error: WSAENOTCONN 10057 Socket is not connected. A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using sendto) no address was supplied. Any other type of operation might also return this error?for example, setsockopt setting SO_KEEPALIVE if the connection has been reset. https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. docs.microsoft.com Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. Any recommendation for me to make it work? Tried to play with config (commenting/uncommenting): #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. Thank you, Yury _______________________________________________ nginx-devel mailing list nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From yshpakov at hotmail.com Thu Feb 13 21:49:25 2020 From: yshpakov at hotmail.com (Yury Shpakov) Date: Thu, 13 Feb 2020 21:49:25 +0000 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> , <5528c341ee8af2cd16fc29fd9814eae5@sebres.de>, Message-ID: Extra info: C:\WINDOWS\system32>netstat /nabo | grep -A 1 ":9000\b" TCP 0.0.0.0:9000 0.0.0.0:0 LISTENING 34540 [nginx.exe] ________________________________ From: nginx-devel on behalf of Yury Shpakov Sent: Thursday, February 13, 2020 4:45 PM To: Sergey Brester Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 Hi Sergey, I reconfigured the config file as follows: === === === #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.2; # backend ip add_header Auth-Port 143; # backend port return 204; } } } === === === And now it's responding on port 9000 as expected: === === === C:\WINDOWS\system32>curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi HTTP/1.1 204 No Content Server: nginx/1.17.9 Date: Thu, 13 Feb 2020 21:30:54 GMT Connection: keep-alive Auth-Status: OK Auth-Server: 127.0.0.2 Auth-Port: 143 === === === However I'm still experiencing the same issue (in log file): === === === 2020/02/13 16:29:24 [notice] 35048#26192: signal process started 2020/02/13 16:29:34 [error] 31732#22720: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 === === === Tried under both admin and regular user. Any further ideas how to get it fixed please? Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 1:51 PM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I answered inline... 12.02.2020 18:59, Yury Shpakov wrote: Hi Sergey, Thank you for you response. I tried netstat /nabo and I don't see any reference to port 9000 at all. So a problem is to make nginx to listen on port 9000 (as server)? Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ for more examples. Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy - you can use some nginx location (and internal URL to same nginx instance) to specify that. Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link) But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 7:38 AM To: nginx-devel at nginx.org Cc: Yury Shpakov Subject: Re: nginx for Windows - WSASend() socket error 10057 It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... try netstat (in cmd as admin): netstat /nabo netstat /nabo | grep -A 1 ":9000\b" and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). additionally try to telnet or curl it: curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html). But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH - nginx.zip, where it works well). Regards, Sergey 12.02.2020 03:01, Yury Shpakov wrote: Hi there, Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: http://nginx.org/en/docs/howto_build_on_win32.html But excluded (don't care about SSL at this point so don't want to install/configure Perl now): --with-openssl=objs/lib/openssl-master \ --with-openssl-opt=no-asm \ --with-http_ssl_module \ And added: --with-mail nmake was successful and nginx.exe was created. However nginx.exe keeps failing with the error: WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Windows API says the following about this error: WSAENOTCONN 10057 Socket is not connected. A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using sendto) no address was supplied. Any other type of operation might also return this error?for example, setsockopt setting SO_KEEPALIVE if the connection has been reset. https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. docs.microsoft.com Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. Any recommendation for me to make it work? Tried to play with config (commenting/uncommenting): #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. Thank you, Yury _______________________________________________ nginx-devel mailing list nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From serg.brester at sebres.de Fri Feb 14 10:59:37 2020 From: serg.brester at sebres.de (Sergey Brester) Date: Fri, 14 Feb 2020 11:59:37 +0100 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> , <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> Message-ID: I don't know what is wrong with your config... I guess your smtp server does not answer properly. Is 127.0.0.2:143 really your SMTP-server? Because port 143 is mostly an IMAP port - but you've specified PROTOCOL SMTP in the server section. Anyway I tested your config with my settings (replaced name and smtp-server, here 192.0.2.222:25) and enabled debug: + error_log logs/error-mail.log DEBUG; mail { - server_name localhost; + server_name EXAMPLE.COM; ... http { ... - add_header Auth-Server 127.0.0.2; - add_header Auth-Port 143; + add_header Auth-Server 192.0.2.222; + add_header Auth-Port 25; it works well - I see the test incoming mail (I send to myself via 8025 port) and following output in the log (a lot of irrelevant messages are removed): >>>>>>>>> 2020/02/14 11:24:04 [debug] 121280#128244: *1 smtp mail from:"mail FROM:" ... 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request line: "GET /cgi-bin/nginxauth.cgi HTTP/1.0" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http uri: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http args: "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http exten: "cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http process request header line 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Host: localhost" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Method: none" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-User: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Pass: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Protocol: smtp" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Login-Attempt: 1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-IP: 127.0.0.1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-Host: [UNAVAILABLE]" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-Helo: myhost.example.com" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-From: mail FROM:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-To: rcpt TO:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header done 2020/02/14 11:24:04 [debug] 121280#128244: *3 event timer del: 512: 1127939767 2020/02/14 11:24:04 [debug] 121280#128244: *3 generic phase: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 1 2020/02/14 11:24:04 [debug] 121280#128244: *3 search through nested static locations of "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 test location: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 using configuration "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http cl:-1 max:1048576 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 3 2020/02/14 11:24:04 [debug] 121280#128244: *3 http set discard body 2020/02/14 11:24:04 [debug] 121280#128244: *3 HTTP/1.1 204 No Content Server: nginx/1.17.4 Date: Fri, 14 Feb 2020 10:24:04 GMT Connection: close Auth-Status: OK Auth-Server: 192.0.2.222 Auth-Port: 25 2020/02/14 11:24:04 [debug] 121280#128244: *3 write new buf t:1 f:0 008AD6A0, pos 008AD6A0, size: 164 file: 0, size: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter: l:1 f:0 s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter limit 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 WSASend: fd:512, s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter 00000000 2020/02/14 11:24:04 [debug] 121280#128244: *3 http finalize request: 0, "/cgi-bin/nginxauth.cgi?" a:1, c:1 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request count:1 blk:0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http close request 2020/02/14 11:24:04 [debug] 121280#128244: *3 http log handler 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008ACC50, unused: 1161 2020/02/14 11:24:04 [debug] 121280#128244: *3 close http connection: 512 2020/02/14 11:24:04 [debug] 121280#128244: *3 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008AC848 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 0039FDE0, unused: 28 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http read handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 164 of 1024 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process status line 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process headers 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Server: nginx/1.17.4" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Date: Fri, 14 Feb 2020 10:24:04 GMT" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Connection: close" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Status: OK" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Server: 192.0.2.222" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Port: 25" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header done 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer del: 496: 1127939764 2020/02/14 11:24:04 [debug] 121280#128244: *1 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *1 free: 008AC040, unused: 196 2020/02/14 11:24:04 [debug] 121280#128244: *1 stream socket 496 2020/02/14 11:24:04 [debug] 121280#128244: *1 connect to 192.0.2.222:25, fd:496 #4 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer add: 496: 60000:1127939769 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 008AC040:4096 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 post event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 delete posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy dummy handler 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 22 of 4096 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy send ehlo 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 0039FDE0:256 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSASend: fd:496, 0, 25 of 25 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 196 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send mail from 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 47 of 47 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 60 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send rcpt to 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 45 of 45 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 63 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer add: 492: 86400000:1214280441 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer del: 496: 1127939769 2020/02/14 11:24:05 [info] 121280#128244: *1 client logged in, client: 127.0.0.1, server: 0.0.0.0:8025 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 1, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 63 of 63 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 6 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 6 of 6 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 50 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 50 of 50 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 170 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 170 of 170 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280535 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 56 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 56 of 56 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle ... <<<<<<<<< Regards, Sergey 13.02.2020 22:45, Yury Shpakov wrote: > Hi Sergey, > > I reconfigured the config file as follows: > > === === === > > #user nobody; > worker_processes 1; > > #error_log logs/error.log; > #error_log logs/error.log notice; > #error_log logs/error.log info; > > #pid logs/nginx.pid; > > events { > worker_connections 1024; > } > > mail { > server_name localhost; > auth_http localhost:9000/cgi-bin/nginxauth.cgi; > # auth_http none; > > smtp_auth none; > # smtp_auth login plain cram-md5; > # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; > xclient off; > > server { > listen 8025; > protocol smtp; > proxy on; > proxy_pass_error_message on; > } > } > > http { > server { > listen 9000; > > location /cgi-bin/nginxauth.cgi { > add_header Auth-Status OK; > add_header Auth-Server 127.0.0.2; # backend ip > add_header Auth-Port 143; # backend port > return 204; > } > } > } > === === === > > And now it's responding on port 9000 as expected: > > === === === > C:WINDOWSsystem32>curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi > > HTTP/1.1 204 No Content > Server: nginx/1.17.9 > Date: Thu, 13 Feb 2020 21:30:54 GMT > Connection: keep-alive > Auth-Status: OK > Auth-Server: 127.0.0.2 Auth-Port: 143 > === === === > > However I'm still experiencing the same issue (in log file): > > === === === > 2020/02/13 16:29:24 [notice] 35048#26192: signal process started > > 2020/02/13 16:29:34 [error] 31732#22720: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 === === === > > Tried under both admin and regular user. > > Any further ideas how to get it fixed please? > > Thank you, > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Wednesday, February 12, 2020 1:51 PM > TO: Yury Shpakov > CC: nginx-devel at nginx.org > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > I answered inline... > > 12.02.2020 18:59, Yury Shpakov wrote: > >> Hi Sergey, >> >> Thank you for you response. >> >> I tried netstat /nabo and I don't see any reference to port 9000 at all. >> So a problem is to make nginx to listen on port 9000 (as server)? >> Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? > > With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). > Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). > > See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ [2] for more examples. > >> Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. > > I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. > > Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy [3] - you can use some nginx location (and internal URL to same nginx instance) to specify that. > > Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. > >> At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. > > Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. > >> And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. >> >> And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). > > Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link [4]) > But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). > > Thank you, > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Wednesday, February 12, 2020 7:38 AM > TO: nginx-devel at nginx.org > CC: Yury Shpakov > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... > > try netstat (in cmd as admin): > > netstat /nabo > netstat /nabo | grep -A 1 ":9000b" > > and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). > > additionally try to telnet or curl it: > > curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi > > if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). > > If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [5]). > > But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. > It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). > > Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [6] > > Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH [4] - nginx.zip [7], where it works well). > > Regards, > Sergey > > 12.02.2020 03:01, Yury Shpakov wrote: > Hi there, > > Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: > http://nginx.org/en/docs/howto_build_on_win32.html [8] > But excluded (don't care about SSL at this point so don't want to install/configure Perl now): > --with-openssl=objs/lib/openssl-master > > --with-openssl-opt=no-asm > --with-http_ssl_module > And added: > --with-mail > > nmake was successful and nginx.exe was created. > However nginx.exe keeps failing with the error: > WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 > Windows API says the following about this error: > > WSAENOTCONN10057 > Socket is not connected.A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using SENDTO [9]) no address was supplied. Any other type of operation might also return this error--for example, SETSOCKOPT [10] setting SO_KEEPALIVE [11] if the connection has been reset. > > https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 [12] > > Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs [12] > Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. > docs.microsoft.com > > Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. > Any recommendation for me to make it work? > Tried to play with config (commenting/uncommenting): > > #user nobody; > worker_processes 1; > #error_log logs/error.log; > #error_log logs/error.log notice; > #error_log logs/error.log info; > #pid logs/nginx.pid; > events { > worker_connections 1024; > } > mail { > server_name localhost; > auth_http localhost:9000/cgi-bin/nginxauth.cgi; > # auth_http none; > smtp_auth none; > # smtp_auth login plain cram-md5; > # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; > xclient off; > server { > listen 8025; > protocol smtp; > proxy on; > proxy_pass_error_message on; > } > } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. > Thank you, > Yury > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel [1] Links: ------ [1] http://mailman.nginx.org/mailman/listinfo/nginx-devel [2] https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ [3] https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy [4] https://github.com/sebres/nginx/releases/tag/release-1.13.0 [5] http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [6] https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [7] https://github.com/sebres/nginx/files/2246440/nginx.zip [8] http://nginx.org/en/docs/howto_build_on_win32.html [9] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-sendto [10] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-setsockopt [11] https://docs.microsoft.com/en-us/windows/desktop/winsock/so-keepalive [12] https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mstavrev at gmail.com Fri Feb 14 11:57:49 2020 From: mstavrev at gmail.com (Marin Stavrev) Date: Fri, 14 Feb 2020 13:57:49 +0200 Subject: [PATCH] Fix for the HT on request headers problem (#1752) In-Reply-To: References: <20200123192931.GI12894@mdounin.ru> Message-ID: Hello, Is there any update about this one, or it is a closed case for the Nginx team? Best Regards Marin On Fri, Jan 24, 2020 at 9:07 AM Marin Stavrev wrote: > Hello, > > I can understand your point, but still RFC 7230 defines OWS to allow HTAB > and the therm used to suggest a single SP is SHOULD (recommendation) not > MUST (mandatory). Thus, Nginx is not fully compliant. > I would never post such a change if it wasn't really needed - I am not > pushing for this change just out of love for RFC compliance.I have this > issue causing problems with some Chinese IP cameras and NVRs that are > generating such headers. I understand this is quite rare, and to be frank > this is the only case I had personally seen such a (lousy) HTTP > implementation. Unfortunately, I don't have any control over their FW and > thus needed this fix on the server side. > I know it does not matter much, but both Apache and Microsoft IIS handle > such headers as expected and do not treat the request as a bad one as Nginx > currently does. > > Best Regards > M. Stavrev > > On Thu, Jan 23, 2020 at 9:29 PM Maxim Dounin wrote: > >> Hello! >> >> On Mon, Jan 20, 2020 at 05:29:25PM +0200, mstavrev at gmail.com wrote: >> >> > # HG changeset patch >> > # User Marin Stavrev >> > # Date 1579526641 -7200 >> > # Mon Jan 20 15:24:01 2020 +0200 >> > # Node ID bf238762fdaf03383c2f3c3718c401e6141e3935 >> > # Parent 6439ef81e37dfccfc3a8c57fed278bf56014ef39 >> > Fix for the HT on request headers problem (#1752) >> > >> > When client send HTTP request with a header of Content-Length that >> starts with >> > horizontal tab character (HT=0x09), Nginx responds with HTTP 400 Bad >> Request. >> > According to HTTP RFC2616 section 4.2, "... The field value MAY be >> preceded by >> > any amount of LWS, though a single SP is preferred.". The difinition of >> LWS is: >> > >> > LWS = [CRLF] 1*( SP | HT ) >> > >> > So a header such as the following should be processed fine: >> > >> > Content-Length:<0x09>110\r\n >> >> Note that RFC 2616 you are quoting was obsoleted by RFC 7230. In >> particular, line folding (the "[CRLF]" part of the grammar) is >> obsolete and must not be generated. Modern syntax rules to refer >> to would be RFC 7230, section 3.2: >> >> header-field = field-name ":" OWS field-value OWS >> >> Where OWS is defined in section 3.2.3 as: >> >> OWS = *( SP / HTAB ) >> ; optional whitespace >> >> and text says that "a sender SHOULD generate the optional >> whitespace as a single SP" where an optional whitespace can >> improve readability. >> >> However, we haven't seen any interoperability problems due to no >> HTAB support in nginx. As such, instead of adding HTAB support it >> might be better to keep parsing strict. >> >> [...] >> >> -- >> Maxim Dounin >> http://mdounin.ru/ >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yshpakov at hotmail.com Fri Feb 14 23:08:51 2020 From: yshpakov at hotmail.com (Yury Shpakov) Date: Fri, 14 Feb 2020 23:08:51 +0000 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> , <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> , Message-ID: So what is the meaning of Auth-Server and Auth-Port headers? So it's relevant only when nginx works as SMTP Proxy (not SMTP Server)? And these are host/port where to redirect SMTP requests? Yeah, I was all the time surprised -- how come, it's set as Proxy but there is no setting where it redirects SMTP communication to. A little bit unexpected place for those setting. Well, let me try... I ran Fake SMTP Server on port 25.(I found on Internet some fake SMTP Server). I configured my test SMTP client to localhost:25 (later to 127.0.0.1:25). They send/receive successfully. So both SMTP Client and (fake) SMTP Server work fine. 127.0.0.1 works fine too. I re-configured my test SMTP client to localhost:8025 (tried 127.0.0.1:8025 too). As well, I changed this section of config as follows: http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } The same error: 2020/02/14 17:37:18 [error] 15260#3328: *5 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Update: Detailed logging with debug information helped a lot. This is what I noticed in there: 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp auth state 2020/02/14 17:40:28 [debug] 3940#22096: *1 WSARecv: fd:584 rc:0 24 of 4096 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp rcpt to:"RCPT TO:" 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer del: 584: 1172123084 2020/02/14 17:40:28 [debug] 3940#22096: *1 malloc: 02F8C260:2048 2020/02/14 17:40:28 [debug] 3940#22096: *1 stream socket 588 2020/02/14 17:40:28 [debug] 3940#22096: *1 connect to [::1]:9000, fd:588 #2 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:768 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:16 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 So it's trying to use IP6 rather than IP4. And below: 2020/02/14 17:40:29 [debug] 3940#22096: *1 delete posted event 03171170 2020/02/14 17:40:29 [debug] 3940#22096: *1 mail auth http write handler 2020/02/14 17:40:29 [debug] 3940#22096: *1 WSASend: fd:588, -1, 0 of 306 2020/02/14 17:40:29 [error] 3940#22096: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 So, I replaced localhost with 127.0.0.1 like this: auth_http 127.0.0.1:9000/cgi-bin/nginxauth.cgi; And it worked. Since I forced it to use IP4. Any idea how to use host name instead of IP address and still have it working? Update 2: I figured it out. Googled a little bit and ended up with the following change to my config: http { server { listen 9000; listen [::]:9000 ipv6only=on; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } Now it works. But why just "listen 9000" doesn't listen on both IP4 and IP6? Is it a bug? Yury ________________________________ From: Sergey Brester Sent: Friday, February 14, 2020 5:59 AM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I don't know what is wrong with your config... I guess your smtp server does not answer properly. Is 127.0.0.2:143 really your SMTP-server? Because port 143 is mostly an IMAP port - but you've specified protocol smtp in the server section. Anyway I tested your config with my settings (replaced name and smtp-server, here 192.0.2.222:25) and enabled debug: + error_log logs/error-mail.log debug; mail { - server_name localhost; + server_name example.com; ... http { ... - add_header Auth-Server 127.0.0.2; - add_header Auth-Port 143; + add_header Auth-Server 192.0.2.222; + add_header Auth-Port 25; it works well - I see the test incoming mail (I send to myself via 8025 port) and following output in the log (a lot of irrelevant messages are removed): >>>>>>>>> 2020/02/14 11:24:04 [debug] 121280#128244: *1 smtp mail from:"mail FROM:" ... 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request line: "GET /cgi-bin/nginxauth.cgi HTTP/1.0" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http uri: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http args: "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http exten: "cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http process request header line 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Host: localhost" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Method: none" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-User: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Pass: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Protocol: smtp" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Login-Attempt: 1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-IP: 127.0.0.1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-Host: [UNAVAILABLE]" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-Helo: myhost.example.com" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-From: mail FROM:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-To: rcpt TO:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header done 2020/02/14 11:24:04 [debug] 121280#128244: *3 event timer del: 512: 1127939767 2020/02/14 11:24:04 [debug] 121280#128244: *3 generic phase: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 1 2020/02/14 11:24:04 [debug] 121280#128244: *3 search through nested static locations of "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 test location: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 using configuration "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http cl:-1 max:1048576 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 3 2020/02/14 11:24:04 [debug] 121280#128244: *3 http set discard body 2020/02/14 11:24:04 [debug] 121280#128244: *3 HTTP/1.1 204 No Content Server: nginx/1.17.4 Date: Fri, 14 Feb 2020 10:24:04 GMT Connection: close Auth-Status: OK Auth-Server: 192.0.2.222 Auth-Port: 25 2020/02/14 11:24:04 [debug] 121280#128244: *3 write new buf t:1 f:0 008AD6A0, pos 008AD6A0, size: 164 file: 0, size: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter: l:1 f:0 s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter limit 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 WSASend: fd:512, s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter 00000000 2020/02/14 11:24:04 [debug] 121280#128244: *3 http finalize request: 0, "/cgi-bin/nginxauth.cgi?" a:1, c:1 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request count:1 blk:0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http close request 2020/02/14 11:24:04 [debug] 121280#128244: *3 http log handler 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008ACC50, unused: 1161 2020/02/14 11:24:04 [debug] 121280#128244: *3 close http connection: 512 2020/02/14 11:24:04 [debug] 121280#128244: *3 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008AC848 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 0039FDE0, unused: 28 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http read handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 164 of 1024 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process status line 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process headers 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Server: nginx/1.17.4" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Date: Fri, 14 Feb 2020 10:24:04 GMT" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Connection: close" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Status: OK" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Server: 192.0.2.222" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Port: 25" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header done 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer del: 496: 1127939764 2020/02/14 11:24:04 [debug] 121280#128244: *1 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *1 free: 008AC040, unused: 196 2020/02/14 11:24:04 [debug] 121280#128244: *1 stream socket 496 2020/02/14 11:24:04 [debug] 121280#128244: *1 connect to 192.0.2.222:25, fd:496 #4 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer add: 496: 60000:1127939769 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 008AC040:4096 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 post event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 delete posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy dummy handler 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 22 of 4096 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy send ehlo 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 0039FDE0:256 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSASend: fd:496, 0, 25 of 25 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 196 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send mail from 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 47 of 47 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 60 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send rcpt to 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 45 of 45 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 63 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer add: 492: 86400000:1214280441 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer del: 496: 1127939769 2020/02/14 11:24:05 [info] 121280#128244: *1 client logged in, client: 127.0.0.1, server: 0.0.0.0:8025 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 1, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 63 of 63 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 6 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 6 of 6 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 50 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 50 of 50 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 170 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 170 of 170 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280535 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 56 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 56 of 56 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle ... <<<<<<<<< Regards, Sergey 13.02.2020 22:45, Yury Shpakov wrote: Hi Sergey, I reconfigured the config file as follows: === === === #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.2; # backend ip add_header Auth-Port 143; # backend port return 204; } } } === === === And now it's responding on port 9000 as expected: === === === C:\WINDOWS\system32>curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi HTTP/1.1 204 No Content Server: nginx/1.17.9 Date: Thu, 13 Feb 2020 21:30:54 GMT Connection: keep-alive Auth-Status: OK Auth-Server: 127.0.0.2 Auth-Port: 143 === === === However I'm still experiencing the same issue (in log file): === === === 2020/02/13 16:29:24 [notice] 35048#26192: signal process started 2020/02/13 16:29:34 [error] 31732#22720: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 === === === Tried under both admin and regular user. Any further ideas how to get it fixed please? Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 1:51 PM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I answered inline... 12.02.2020 18:59, Yury Shpakov wrote: Hi Sergey, Thank you for you response. I tried netstat /nabo and I don't see any reference to port 9000 at all. So a problem is to make nginx to listen on port 9000 (as server)? Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ for more examples. Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy - you can use some nginx location (and internal URL to same nginx instance) to specify that. Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link) But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 7:38 AM To: nginx-devel at nginx.org Cc: Yury Shpakov Subject: Re: nginx for Windows - WSASend() socket error 10057 It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... try netstat (in cmd as admin): netstat /nabo netstat /nabo | grep -A 1 ":9000\b" and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). additionally try to telnet or curl it: curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html). But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH - nginx.zip, where it works well). Regards, Sergey 12.02.2020 03:01, Yury Shpakov wrote: Hi there, Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: http://nginx.org/en/docs/howto_build_on_win32.html But excluded (don't care about SSL at this point so don't want to install/configure Perl now): --with-openssl=objs/lib/openssl-master \ --with-openssl-opt=no-asm \ --with-http_ssl_module \ And added: --with-mail nmake was successful and nginx.exe was created. However nginx.exe keeps failing with the error: WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Windows API says the following about this error: WSAENOTCONN 10057 Socket is not connected. A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using sendto) no address was supplied. Any other type of operation might also return this error?for example, setsockopt setting SO_KEEPALIVE if the connection has been reset. https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. docs.microsoft.com Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. Any recommendation for me to make it work? Tried to play with config (commenting/uncommenting): #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. Thank you, Yury _______________________________________________ nginx-devel mailing list nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From erica at eff.org Sat Feb 15 00:44:16 2020 From: erica at eff.org (Erica Portnoy) Date: Fri, 14 Feb 2020 16:44:16 -0800 Subject: Building a Let's Encrypt module Message-ID: <385a8d17-5220-bc91-dbdd-29d92a5994f2@eff.org> Hello nginx-devel. I'm a developer on Certbot, EFF's Let's Encrypt client. I've been working on Certbot's NGINX plugin for a few years now. This is an extension of the standalone Certbot program that operates on a user's behalf to modify their configuration files to retrieve and install SSL certificates. For the obvious reasons, this approach has its drawbacks. For a while now, the Certbot team has been dreaming of a way to get certs automatically as part of the server, without a standalone application needed (similar to what Caddy does). From our research, it seems like the best approach would be to write an NGINX module. I've found some information about what this would entail and the best ways to go about doing so, but I still have some open questions. This is where you all come in. I would love any thoughts, feedback, proposed solutions, clarifications, pointing out of fundamental errors in understanding for the following: Basically our thinking so far is to write a plugin that will fetch and load certificates behind the scenes. Ideally it would do this decoupled from incoming requests. The problem we've been running into is about permissions and available actions -- we haven't found a clean way to either run scheduled tasks (can't modify cron from inside a module) or reload NGINX configurations from inside a plugin in vanilla open source NGINX. Are there ways to do this that we just haven't found? And if not, is there any chance of getting someone from the NGINX team to work with us to make that possible? If those are possible, we'd also have to decide if it's best to include an ACME library within the plugin itself, or shell out to certbot to fetch and/or manage certificates. Is calling out to an external process from inside an NGINX plugin possible? If we do get all of this working, we'd want to have the easiest possible solution for users to install the plugin. Obviously the easiest way would be for it to be built into NGINX itself :) Barring that, we could probably find some way to package and distribute it. We're currently in the process of moving our distribution to Snaps, so once we get that infrastructure set up there'll be the option of repackaging NGINX with the plugin included and distributing the whole thing. The other thing we were considering was to build on top of OpenResty instead, which seems to have a bit more functionality available. Obviously that isn't ideal, but if it's the only way forward we might start looking into performance and distribution options for that. Looking forward to hearing thoughts. Best, Erica Portnoy Senior Staff Technologist Electronic Frontier Foundation -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Mon Feb 17 14:01:43 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Mon, 17 Feb 2020 14:01:43 +0000 Subject: [njs] Introduced ToLength() according to ES6. Message-ID: details: https://hg.nginx.org/njs/rev/13dbdff9b76f branches: changeset: 1326:13dbdff9b76f user: Dmitry Volyntsev date: Mon Feb 17 16:13:43 2020 +0300 description: Introduced ToLength() according to ES6. Since ES6, according to the spec maximum length value is 2**53 - 1 not 2**32 - 1, so uint64_t data type is required. diffstat: src/njs_array.c | 418 +++++++++++++++++++++++--------------------- src/njs_array.h | 4 +- src/njs_chb.h | 8 +- src/njs_function.c | 8 +- src/njs_json.c | 10 +- src/njs_number.c | 24 ++ src/njs_number.h | 14 +- src/njs_object.c | 2 +- src/njs_object.h | 4 +- src/njs_regexp.c | 4 +- src/njs_string.c | 5 +- src/njs_typed_array.c | 24 +- src/njs_value.c | 18 +- src/njs_value.h | 63 ++++++- src/njs_value_conversion.h | 2 +- src/test/njs_unit_test.c | 126 ++++++++++++- 16 files changed, 475 insertions(+), 259 deletions(-) diffs (truncated from 1794 to 1000 lines): diff -r 974e6c195410 -r 13dbdff9b76f src/njs_array.c --- a/src/njs_array.c Thu Feb 13 16:25:37 2020 +0300 +++ b/src/njs_array.c Mon Feb 17 16:13:43 2020 +0300 @@ -18,13 +18,13 @@ typedef struct { njs_array_t *array; - uint32_t from; - uint32_t to; + int64_t from; + int64_t to; } njs_array_iterator_args_t; typedef njs_int_t (*njs_array_iterator_handler_t)(njs_vm_t *vm, - njs_array_iterator_args_t *args, njs_value_t *entry, uint32_t n); + njs_array_iterator_args_t *args, njs_value_t *entry, uint64_t n); static njs_int_t njs_array_prototype_slice_copy(njs_vm_t *vm, @@ -128,9 +128,8 @@ njs_array_convert_to_slow_array(njs_vm_t length = array->length; for (i = 0; i < length; i++) { - njs_uint32_to_string(&index, i); - if (njs_is_valid(&array->start[i])) { + njs_uint32_to_string(&index, i); prop = njs_object_property_add(vm, &value, &index, 0); if (njs_slow_path(prop == NULL)) { return NJS_ERROR; @@ -183,7 +182,8 @@ njs_array_length_set(njs_vm_t *vm, njs_v njs_object_prop_t *prev, njs_value_t *setval) { double num, idx; - uint32_t i, length, prev_length; + uint32_t i, length; + uint64_t prev_length; njs_int_t ret; njs_array_t *array, *keys; @@ -197,7 +197,7 @@ njs_array_length_set(njs_vm_t *vm, njs_v return ret; } - length = njs_number_to_length(num); + length = (uint32_t) njs_number_to_length(num); if ((double) length != num) { njs_range_error(vm, "Invalid array length"); return NJS_ERROR; @@ -306,7 +306,7 @@ njs_array_expand(njs_vm_t *vm, njs_array size += size / 2; } - if (njs_slow_path(size > NJS_ARRAY_MAX_LENGTH)) { + if (njs_slow_path(size > (UINT32_MAX / sizeof(njs_value_t)))) { goto memory_error; } @@ -354,7 +354,7 @@ njs_array_constructor(njs_vm_t *vm, njs_ if (size == 1 && njs_is_number(&args[0])) { num = njs_number(&args[0]); - size = (uint32_t) num; + size = (uint32_t) njs_number_to_length(num); if ((double) size != num) { njs_range_error(vm, "Invalid array length"); @@ -529,7 +529,7 @@ njs_array_length(njs_vm_t *vm,njs_object return ret; } - length = njs_number_to_length(num); + length = (uint32_t) njs_number_to_length(num); if ((double) length != num) { njs_range_error(vm, "Invalid array length"); return NJS_ERROR; @@ -582,7 +582,7 @@ njs_array_prototype_slice(njs_vm_t *vm, njs_index_t unused) { int64_t start, end, length; - uint32_t object_length; + uint64_t object_length; njs_int_t ret; njs_value_t *this; @@ -658,7 +658,7 @@ njs_array_prototype_slice_copy(njs_vm_t uint32_t n; njs_int_t ret; njs_array_t *array, *keys; - njs_value_t *value, index, retval, array_value; + njs_value_t *value, retval, self; const u_char *src, *end; njs_slice_prop_t string_slice; njs_string_prop_t string; @@ -688,10 +688,8 @@ njs_array_prototype_slice_copy(njs_vm_t /* src value may be in Array.prototype object. */ - njs_uint32_to_string(&index, start++); - value = &array->start[n++]; - ret = njs_value_property(vm, this, &index, value); + ret = njs_value_property_i64(vm, this, start++, value); if (njs_slow_path(ret == NJS_ERROR)) { return NJS_ERROR; } @@ -746,10 +744,8 @@ njs_array_prototype_slice_copy(njs_vm_t } else if (njs_is_object(this)) { do { - njs_uint32_to_string(&index, start++); - value = &array->start[n++]; - ret = njs_value_property(vm, this, &index, value); + ret = njs_value_property_i64(vm, this, start++, value); if (ret != NJS_OK) { njs_set_invalid(value); @@ -774,19 +770,17 @@ njs_array_prototype_slice_copy(njs_vm_t goto done; } - njs_set_array(&array_value, array); + njs_set_array(&self, array); if (njs_fast_object(length)) { do { - njs_uint32_to_string(&index, start++); - - ret = njs_value_property(vm, this, &index, &retval); + ret = njs_value_property_i64(vm, this, start++, &retval); if (njs_slow_path(ret == NJS_ERROR)) { return NJS_ERROR; } if (ret == NJS_OK) { - ret = njs_value_property_set(vm, &array_value, &index, &retval); + ret = njs_value_property_i64_set(vm, &self, start, &retval); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -810,8 +804,7 @@ njs_array_prototype_slice_copy(njs_vm_t goto done; } - ret = njs_value_property_set(vm, &array_value, &keys->start[n], - &retval); + ret = njs_value_property_set(vm, &self, &keys->start[n], &retval); if (njs_slow_path(ret == NJS_ERROR)) { goto done; } @@ -835,11 +828,11 @@ static njs_int_t njs_array_prototype_push(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { - uint32_t length; + uint64_t length; njs_int_t ret; njs_uint_t i; njs_array_t *array; - njs_value_t *this, index; + njs_value_t *this; length = 0; this = njs_argument(args, 0); @@ -874,10 +867,13 @@ njs_array_prototype_push(njs_vm_t *vm, n return ret; } + if (njs_slow_path((length + nargs - 1) > NJS_MAX_LENGTH)) { + njs_type_error(vm, "Invalid length"); + return NJS_ERROR; + } + for (i = 1; i < nargs; i++) { - njs_uint32_to_string(&index, length++); - - ret = njs_value_property_set(vm, this, &index, &args[i]); + ret = njs_value_property_i64_set(vm, this, length++, &args[i]); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -898,10 +894,10 @@ static njs_int_t njs_array_prototype_pop(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { - uint32_t length; + uint64_t length; njs_int_t ret; njs_array_t *array; - njs_value_t *this, *entry, index; + njs_value_t *this, *entry; this = njs_argument(args, 0); @@ -921,6 +917,15 @@ njs_array_prototype_pop(njs_vm_t *vm, nj if (njs_is_valid(entry)) { vm->retval = *entry; + + } else { + /* src value may be in Array.prototype object. */ + + ret = njs_value_property_i64(vm, this, array->length, + &vm->retval); + if (njs_slow_path(ret == NJS_ERROR)) { + return NJS_ERROR; + } } } @@ -932,15 +937,23 @@ njs_array_prototype_pop(njs_vm_t *vm, nj return ret; } - if (length != 0) { - njs_uint32_to_string(&index, --length); - - ret = njs_value_property_delete(vm, this, &index, &vm->retval); - if (njs_slow_path(ret == NJS_ERROR)) { - return ret; - } + if (length == 0) { + njs_set_undefined(&vm->retval); + goto done; } + ret = njs_value_property_i64(vm, this, --length, &vm->retval); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + + ret = njs_value_property_i64_delete(vm, this, length, NULL); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + +done: + ret = njs_object_length_set(vm, this, length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; @@ -955,11 +968,11 @@ njs_array_prototype_unshift(njs_vm_t *vm njs_index_t unused) { double idx; - uint32_t from, to, length; + uint64_t from, to, length; njs_int_t ret; njs_uint_t n; njs_array_t *array, *keys; - njs_value_t *this, entry, index; + njs_value_t *this, entry; this = njs_argument(args, 0); length = 0; @@ -970,14 +983,9 @@ njs_array_prototype_unshift(njs_vm_t *vm return ret; } - if (njs_is_fast_array(this)) { + if (njs_fast_path(njs_is_fast_array(this))) { array = njs_array(this); - if (array->length > (UINT32_MAX - n)) { - njs_type_error(vm, "Invalid length"); - return NJS_ERROR; - } - if (n != 0) { ret = njs_array_expand(vm, array, n, 0); if (njs_slow_path(ret != NJS_OK)) { @@ -1009,7 +1017,7 @@ njs_array_prototype_unshift(njs_vm_t *vm goto done; } - if (length > (UINT32_MAX - n)) { + if (njs_slow_path((length + n) > NJS_MAX_LENGTH)) { njs_type_error(vm, "Invalid length"); return NJS_ERROR; } @@ -1031,11 +1039,9 @@ njs_array_prototype_unshift(njs_vm_t *vm } if (ret == NJS_OK) { - idx = njs_string_to_index(&keys->start[from]); - - njs_uint32_to_string(&index, (uint32_t) idx + nargs - 1); - - ret = njs_value_property_set(vm, this, &index, &entry); + idx = njs_string_to_index(&keys->start[from]) + n; + + ret = njs_value_property_i64_set(vm, this, idx, &entry); if (njs_slow_path(ret == NJS_ERROR)) { njs_array_destroy(vm, keys); return ret; @@ -1045,7 +1051,7 @@ njs_array_prototype_unshift(njs_vm_t *vm njs_array_destroy(vm, keys); - length += nargs - 1; + length += n; goto copy; } @@ -1055,9 +1061,7 @@ njs_array_prototype_unshift(njs_vm_t *vm to = length; while (from > 0) { - njs_uint32_to_string(&index, --from); - - ret = njs_value_property_delete(vm, this, &index, &entry); + ret = njs_value_property_i64_delete(vm, this, --from, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -1065,9 +1069,7 @@ njs_array_prototype_unshift(njs_vm_t *vm to--; if (ret == NJS_OK) { - njs_uint32_to_string(&index, to); - - ret = njs_value_property_set(vm, this, &index, &entry); + ret = njs_value_property_i64_set(vm, this, to, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -1077,9 +1079,7 @@ njs_array_prototype_unshift(njs_vm_t *vm copy: for (n = 1; n < nargs; n++) { - njs_uint32_to_string(&index, n - 1); - - ret = njs_value_property_set(vm, this, &index, &args[n]); + ret = njs_value_property_i64_set(vm, this, n - 1, &args[n]); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -1102,10 +1102,10 @@ static njs_int_t njs_array_prototype_shift(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { - uint32_t i, length; + uint64_t i, length; njs_int_t ret; njs_array_t *array; - njs_value_t *this, *item, entry, index; + njs_value_t *this, *item, entry; this = njs_argument(args, 0); length = 0; @@ -1122,13 +1122,21 @@ njs_array_prototype_shift(njs_vm_t *vm, if (array->length != 0) { array->length--; - item = &array->start[0]; - array->start++; if (njs_is_valid(item)) { vm->retval = *item; + + } else { + /* src value may be in Array.prototype object. */ + + ret = njs_value_property_i64(vm, this, 0, &vm->retval); + if (njs_slow_path(ret == NJS_ERROR)) { + return NJS_ERROR; + } } + + array->start++; } return NJS_OK; @@ -1143,25 +1151,19 @@ njs_array_prototype_shift(njs_vm_t *vm, goto done; } - njs_uint32_to_string(&index, 0); - - ret = njs_value_property_delete(vm, this, &index, &vm->retval); + ret = njs_value_property_i64_delete(vm, this, 0, &vm->retval); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } for (i = 1; i < length; i++) { - njs_uint32_to_string(&index, i); - - ret = njs_value_property_delete(vm, this, &index, &entry); + ret = njs_value_property_i64_delete(vm, this, i, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } if (ret == NJS_OK) { - njs_uint32_to_string(&index, i - 1); - - ret = njs_value_property_set(vm, this, &index, &entry); + ret = njs_value_property_i64_set(vm, this, i - 1, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -1304,7 +1306,7 @@ static njs_int_t njs_array_prototype_reverse(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { - uint32_t length; + uint64_t length; njs_int_t ret; njs_uint_t i, n; njs_value_t value, *this; @@ -1385,15 +1387,13 @@ njs_array_prototype_join(njs_vm_t *vm, n njs_index_t unused) { u_char *p, *last; - size_t size; - ssize_t length; - uint32_t len; + int64_t length; + uint64_t i, len, size; njs_int_t ret; njs_chb_t chain; njs_utf8_t utf8; - njs_uint_t i; njs_array_t *array; - njs_value_t *value, *this, index, entry; + njs_value_t *value, *this, entry; njs_string_prop_t separator, string; this = njs_argument(args, 0); @@ -1447,13 +1447,13 @@ njs_array_prototype_join(njs_vm_t *vm, n njs_chb_init(&chain, vm->mem_pool); for (i = 0; i < len; i++) { - if (njs_fast_path(array != NULL)) { + if (njs_fast_path(njs_object(this)->fast_array + && njs_is_valid(&array->start[i]))) + { value = &array->start[i]; } else { - njs_uint32_to_string(&index, i); - - ret = njs_value_property(vm, this, &index, &entry); + ret = njs_value_property_i64(vm, this, i, &entry); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -1493,6 +1493,11 @@ njs_array_prototype_join(njs_vm_t *vm, n length += separator.length; njs_chb_append(&chain, separator.start, separator.size); + + if (njs_slow_path(length > NJS_STRING_MAX_LENGTH)) { + njs_range_error(vm, "invalid string length"); + return NJS_ERROR; + } } njs_chb_drop(&chain, separator.size); @@ -1597,14 +1602,22 @@ njs_array_indices(njs_vm_t *vm, const nj njs_inline njs_int_t njs_array_object_handler(njs_vm_t *vm, njs_array_iterator_handler_t handler, - njs_array_iterator_args_t *args, njs_value_t *key, uint32_t i) + njs_array_iterator_args_t *args, njs_value_t *key, uint64_t i) { njs_int_t ret; njs_value_t prop, *entry; - ret = njs_value_property(vm, args->value, key, &prop); - if (njs_slow_path(ret == NJS_ERROR)) { - return ret; + if (key != NULL) { + ret = njs_value_property(vm, args->value, key, &prop); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + + } else { + ret = njs_value_property_i64(vm, args->value, i, &prop); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } } entry = (ret == NJS_OK) ? &prop : njs_value_arg(&njs_value_invalid); @@ -1627,10 +1640,10 @@ njs_array_iterator(njs_vm_t *vm, njs_arr njs_array_iterator_handler_t handler) { double idx; - uint32_t length, i, from, to; + uint64_t length, i, from, to; njs_int_t ret; - njs_array_t *keys; - njs_value_t *value, character, index, string_obj; + njs_array_t *array, *keys; + njs_value_t *value, *entry, prop, character, string_obj; njs_object_t *object; const u_char *p, *end, *pos; njs_string_prop_t string_prop; @@ -1640,16 +1653,30 @@ njs_array_iterator(njs_vm_t *vm, njs_arr to = args->to; if (njs_is_array(value)) { - if (njs_slow_path(!njs_is_fast_array(value))) { - goto process_object; - } - - for (i = from; i < to; i++) { - if (i < njs_array_len(value)) { - ret = handler(vm, args, &njs_array_start(value)[i], i); + array = njs_array(value); + + for (; from < to; from++) { + if (njs_slow_path(!array->object.fast_array)) { + goto process_object; + } + + if (njs_fast_path(from < array->length + && njs_is_valid(&array->start[from]))) + { + ret = handler(vm, args, &array->start[from], from); } else { - ret = handler(vm, args, njs_value_arg(&njs_value_invalid), i); + entry = njs_value_arg(&njs_value_invalid); + ret = njs_value_property_i64(vm, value, from, &prop); + if (njs_slow_path(ret != NJS_DECLINED)) { + if (ret == NJS_ERROR) { + return NJS_ERROR; + } + + entry = ∝ + } + + ret = handler(vm, args, entry, from); } if (njs_slow_path(ret != NJS_OK)) { @@ -1680,7 +1707,7 @@ njs_array_iterator(njs_vm_t *vm, njs_arr value = njs_object_value(value); } - length = (uint32_t) njs_string_prop(&string_prop, value); + length = njs_string_prop(&string_prop, value); p = string_prop.start; end = p + string_prop.size; @@ -1760,9 +1787,7 @@ process_object: } for (i = from; i < to; i++) { - njs_uint32_to_string(&index, i); - - ret = njs_array_object_handler(vm, handler, args, &index, i); + ret = njs_array_object_handler(vm, handler, args, NULL, i); if (njs_slow_path(ret != NJS_OK)) { return ret; } @@ -1777,10 +1802,10 @@ njs_array_reverse_iterator(njs_vm_t *vm, njs_array_iterator_handler_t handler) { double idx; - uint32_t i, from, to, length; + uint64_t i, from, to, length; njs_int_t ret; - njs_array_t *keys; - njs_value_t *entry, *value, character, index, string_obj; + njs_array_t *array, *keys; + njs_value_t *entry, *value, prop, character, string_obj; njs_object_t *object; const u_char *p, *end, *pos; njs_string_prop_t string_prop; @@ -1790,16 +1815,34 @@ njs_array_reverse_iterator(njs_vm_t *vm, to = args->to; if (njs_is_array(value)) { - if (njs_slow_path(!njs_is_fast_array(value))) { - goto process_object; - } - - i = from + 1; - - while (i-- > to) { - entry = &njs_array_start(value)[i]; - - ret = handler(vm, args, entry, i); + array = njs_array(value); + + from += 1; + + while (from-- > to) { + if (njs_slow_path(!array->object.fast_array)) { + goto process_object; + } + + if (njs_fast_path(from < array->length + && njs_is_valid(&array->start[from]))) + { + ret = handler(vm, args, &array->start[from], from); + + } else { + entry = njs_value_arg(&njs_value_invalid); + ret = njs_value_property_i64(vm, value, from, &prop); + if (njs_slow_path(ret != NJS_DECLINED)) { + if (ret == NJS_ERROR) { + return NJS_ERROR; + } + + entry = ∝ + } + + ret = handler(vm, args, entry, from); + } + if (njs_slow_path(ret != NJS_OK)) { if (ret > 0) { return NJS_DECLINED; @@ -1828,7 +1871,7 @@ njs_array_reverse_iterator(njs_vm_t *vm, value = njs_object_value(value); } - length = (uint32_t) njs_string_prop(&string_prop, value); + length = njs_string_prop(&string_prop, value); end = string_prop.start + string_prop.size; if (length == string_prop.size) { @@ -1921,9 +1964,7 @@ process_object: i = from + 1; while (i-- > to) { - njs_uint32_to_string(&index, i); - - ret = njs_array_object_handler(vm, handler, args, &index, i); + ret = njs_array_object_handler(vm, handler, args, NULL, i); if (njs_slow_path(ret != NJS_OK)) { return ret; } @@ -1964,11 +2005,9 @@ njs_array_prototype_concat(njs_vm_t *vm, njs_index_t unused) { double idx; - uint32_t len; - uint64_t length; + uint64_t i, k, len, length; njs_int_t ret; - njs_uint_t i, k; - njs_value_t this, index, retval, *value, *e; + njs_value_t this, retval, *value, *e; njs_array_t *array, *keys; ret = njs_value_to_object(vm, &args[0]); @@ -2002,7 +2041,7 @@ njs_array_prototype_concat(njs_vm_t *vm, return ret; } - if (njs_slow_path((length + len) >= NJS_ARRAY_MAX_LENGTH53)) { + if (njs_slow_path((length + len) > NJS_MAX_LENGTH)) { njs_type_error(vm, "Invalid length"); return NJS_ERROR; } @@ -2014,9 +2053,7 @@ njs_array_prototype_concat(njs_vm_t *vm, value = &njs_array_start(e)[k]; if (njs_slow_path(!njs_is_valid(value))) { - njs_uint32_to_string(&index, k); - ret = njs_value_property(vm, e, &index, - &retval); + ret = njs_value_property_i64(vm, e, k, &retval); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -2039,9 +2076,7 @@ njs_array_prototype_concat(njs_vm_t *vm, if (njs_fast_object(len)) { for (k = 0; k < len; k++, length++) { - njs_uint32_to_string(&index, k); - - ret = njs_value_property(vm, e, &index, &retval); + ret = njs_value_property_i64(vm, e, k, &retval); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -2050,9 +2085,8 @@ njs_array_prototype_concat(njs_vm_t *vm, continue; } - njs_uint32_to_string(&index, length); - - ret = njs_value_property_set(vm, &this, &index, &retval); + ret = njs_value_property_i64_set(vm, &this, length, + &retval); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -2072,12 +2106,10 @@ njs_array_prototype_concat(njs_vm_t *vm, return ret; } - idx = njs_string_to_index(&keys->start[k]); + idx = njs_string_to_index(&keys->start[k]) + length; if (ret == NJS_OK) { - njs_uint32_to_string(&index, length + idx); - - ret = njs_value_property_set(vm, &this, &index, &retval); + ret = njs_value_property_i64_set(vm, &this, idx, &retval); if (njs_slow_path(ret == NJS_ERROR)) { njs_array_destroy(vm, keys); return ret; @@ -2092,7 +2124,7 @@ njs_array_prototype_concat(njs_vm_t *vm, continue; } - if (njs_slow_path((length + len) >= NJS_ARRAY_MAX_LENGTH53)) { + if (njs_slow_path((length + len) >= NJS_MAX_LENGTH)) { njs_type_error(vm, "Invalid length"); return NJS_ERROR; } @@ -2104,9 +2136,7 @@ njs_array_prototype_concat(njs_vm_t *vm, } } else { - njs_uint32_to_string(&index, length); - - ret = njs_value_property_set(vm, &this, &index, e); + ret = njs_value_property_i64_set(vm, &this, length, e); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -2128,7 +2158,7 @@ njs_array_prototype_concat(njs_vm_t *vm, static njs_int_t njs_array_handler_index_of(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { if (njs_values_strict_equal(args->argument, entry)) { njs_set_number(&vm->retval, n); @@ -2145,7 +2175,7 @@ njs_array_prototype_index_of(njs_vm_t *v njs_index_t unused) { int64_t from; - uint32_t length; + uint64_t length; njs_int_t ret; njs_array_iterator_args_t iargs; @@ -2180,12 +2210,12 @@ njs_array_prototype_index_of(njs_vm_t *v } } - iargs.from = (uint32_t) from; + iargs.from = from; iargs.to = length; ret = njs_array_iterator(vm, &iargs, njs_array_handler_index_of); - if (njs_fast_path(ret == NJS_DECLINED)) { - return NJS_OK; + if (njs_fast_path(ret != NJS_OK)) { + return (ret == NJS_DECLINED) ? NJS_OK : NJS_ERROR; } not_found: @@ -2201,7 +2231,7 @@ njs_array_prototype_last_index_of(njs_vm njs_uint_t nargs, njs_index_t unused) { int64_t from; - uint32_t length; + uint64_t length; njs_int_t ret; njs_array_iterator_args_t iargs; @@ -2234,7 +2264,7 @@ njs_array_prototype_last_index_of(njs_vm } if (from >= 0) { - from = njs_min(from, length - 1); + from = njs_min((uint64_t) from, length - 1); } else if (from < 0) { from += length; @@ -2248,8 +2278,8 @@ njs_array_prototype_last_index_of(njs_vm iargs.to = 0; ret = njs_array_reverse_iterator(vm, &iargs, njs_array_handler_index_of); - if (njs_fast_path(ret == NJS_DECLINED)) { - return NJS_OK; + if (njs_fast_path(ret != NJS_OK)) { + return (ret == NJS_DECLINED) ? NJS_OK : NJS_ERROR; } not_found: @@ -2262,7 +2292,7 @@ not_found: static njs_int_t njs_array_handler_includes(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { if (!njs_is_valid(entry)) { entry = njs_value_arg(&njs_value_undefined); @@ -2283,7 +2313,7 @@ njs_array_prototype_includes(njs_vm_t *v njs_index_t unused) { int64_t from; - uint32_t length; + uint64_t length; njs_int_t ret; njs_array_iterator_args_t iargs; @@ -2318,12 +2348,12 @@ njs_array_prototype_includes(njs_vm_t *v } } - iargs.from = (uint32_t) from; + iargs.from = from; iargs.to = length; ret = njs_array_iterator(vm, &iargs, njs_array_handler_includes); - if (njs_fast_path(ret == NJS_DECLINED)) { - return NJS_OK; + if (njs_fast_path(ret != NJS_OK)) { + return (ret == NJS_DECLINED) ? NJS_OK : NJS_ERROR; } not_found: @@ -2338,11 +2368,10 @@ static njs_int_t njs_array_prototype_fill(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { - int64_t start, end; - uint32_t length; - njs_int_t i, ret; + int64_t i, length, start, end; + njs_int_t ret; njs_array_t *array; - njs_value_t name, *this, *value; + njs_value_t *this, *value; this = njs_argument(args, 0); @@ -2358,7 +2387,7 @@ njs_array_prototype_fill(njs_vm_t *vm, n length = array->length; } else { - ret = njs_object_length(vm, this, &length); + ret = njs_object_length(vm, this, (uint64_t *) &length); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -2398,9 +2427,7 @@ njs_array_prototype_fill(njs_vm_t *vm, n value = njs_arg(args, nargs, 1); while (start < end) { - njs_uint32_to_string(&name, start++); - - ret = njs_value_property_set(vm, this, &name, value); + ret = njs_value_property_i64_set(vm, this, start++, value); if (njs_slow_path(ret == NJS_ERROR)) { return ret; } @@ -2442,7 +2469,7 @@ njs_array_validate_args(njs_vm_t *vm, nj return ret; } - ret = njs_value_length(vm, iargs->value, &iargs->to); + ret = njs_value_length(vm, iargs->value, (uint64_t *) &iargs->to); if (njs_slow_path(ret != NJS_OK)) { return ret; } @@ -2467,7 +2494,7 @@ failed: static njs_int_t njs_array_handler_for_each(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { if (njs_is_valid(entry)) { return njs_array_iterator_call(vm, args, entry, n); @@ -2502,7 +2529,7 @@ njs_array_prototype_for_each(njs_vm_t *v static njs_int_t njs_array_handler_some(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { njs_int_t ret; @@ -2550,7 +2577,7 @@ njs_array_prototype_some(njs_vm_t *vm, n static njs_int_t njs_array_handler_every(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { njs_int_t ret; @@ -2598,7 +2625,7 @@ njs_array_prototype_every(njs_vm_t *vm, static njs_int_t njs_array_handler_filter(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { njs_int_t ret; njs_value_t copy; @@ -2654,7 +2681,7 @@ njs_array_prototype_filter(njs_vm_t *vm, static njs_int_t njs_array_handler_find(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { njs_int_t ret; njs_value_t copy; @@ -2708,7 +2735,7 @@ njs_array_prototype_find(njs_vm_t *vm, n static njs_int_t njs_array_handler_find_index(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { njs_int_t ret; njs_value_t copy; @@ -2762,11 +2789,11 @@ njs_array_prototype_find_index(njs_vm_t static njs_int_t njs_array_handler_map(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { njs_int_t ret; njs_array_t *retval; - njs_value_t this, key; + njs_value_t this; retval = args->array; @@ -2786,9 +2813,8 @@ njs_array_handler_map(njs_vm_t *vm, njs_ } else { njs_set_array(&this, retval); - njs_uint32_to_string(&key, n); - - ret = njs_value_property_set(vm, &this, &key, &vm->retval); + + ret = njs_value_property_i64_set(vm, &this, n, &vm->retval); if (njs_slow_path(ret != NJS_OK)) { return ret; } @@ -2804,7 +2830,7 @@ static njs_int_t njs_array_prototype_map(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { - uint32_t length, i; + uint64_t length, i; njs_int_t ret; njs_array_t *array; njs_value_t *this; @@ -2870,7 +2896,7 @@ unexpected_args: njs_inline njs_int_t njs_array_iterator_reduce(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { njs_value_t arguments[5]; @@ -2888,7 +2914,7 @@ njs_array_iterator_reduce(njs_vm_t *vm, static njs_int_t njs_array_handler_reduce(njs_vm_t *vm, njs_array_iterator_args_t *args, - njs_value_t *entry, uint32_t n) + njs_value_t *entry, uint64_t n) { njs_int_t ret; @@ -2960,7 +2986,7 @@ njs_array_prototype_reduce_right(njs_vm_ return ret; } - ret = njs_value_length(vm, iargs.value, &iargs.from); + ret = njs_value_length(vm, iargs.value, (uint64_t *) &iargs.from); if (njs_slow_path(ret != NJS_OK)) { return ret; } @@ -3019,8 +3045,8 @@ unexpected_args: static njs_int_t -njs_array_string_sort(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) +njs_array_string_sort(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t unused) { From xeioex at nginx.com Mon Feb 17 14:01:45 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Mon, 17 Feb 2020 14:01:45 +0000 Subject: [njs] Style: removing excessive commentaries. Message-ID: details: https://hg.nginx.org/njs/rev/cbc09d7edfc8 branches: changeset: 1327:cbc09d7edfc8 user: Dmitry Volyntsev date: Mon Feb 17 16:18:18 2020 +0300 description: Style: removing excessive commentaries. diffstat: src/njs_array.c | 10 ------- src/njs_array_buffer.c | 5 --- src/njs_boolean.c | 3 -- src/njs_date.c | 3 -- src/njs_dtoa_fixed.c | 10 ------- src/njs_error.c | 27 ------------------- src/njs_function.c | 10 ------- src/njs_math.c | 17 ------------ src/njs_number.c | 12 -------- src/njs_object.c | 26 ------------------ src/njs_regexp.c | 3 -- src/njs_string.c | 69 -------------------------------------------------- src/njs_symbol.c | 18 ------------- 13 files changed, 0 insertions(+), 213 deletions(-) diffs (truncated from 1324 to 1000 lines): diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_array.c --- a/src/njs_array.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_array.c Mon Feb 17 16:18:18 2020 +0300 @@ -444,7 +444,6 @@ njs_array_of(njs_vm_t *vm, njs_value_t * static const njs_object_prop_t njs_array_constructor_properties[] = { - /* Array.name == "Array". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -452,7 +451,6 @@ static const njs_object_prop_t njs_arra .configurable = 1, }, - /* Array.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -460,14 +458,12 @@ static const njs_object_prop_t njs_arra .configurable = 1, }, - /* Array.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), .value = njs_prop_handler(njs_object_prototype_create), }, - /* Array.isArray(). */ { .type = NJS_PROPERTY, .name = njs_string("isArray"), @@ -476,8 +472,6 @@ static const njs_object_prop_t njs_arra .configurable = 1, }, - /* ES6. */ - /* Array.of(). */ { .type = NJS_PROPERTY, .name = njs_string("of"), @@ -3403,7 +3397,6 @@ static const njs_object_prop_t njs_arra .configurable = 1, }, - /* ES7. */ { .type = NJS_PROPERTY, .name = njs_string("includes"), @@ -3436,7 +3429,6 @@ static const njs_object_prop_t njs_arra .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("fill"), @@ -3453,7 +3445,6 @@ static const njs_object_prop_t njs_arra .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("find"), @@ -3462,7 +3453,6 @@ static const njs_object_prop_t njs_arra .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("findIndex"), diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_array_buffer.c --- a/src/njs_array_buffer.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_array_buffer.c Mon Feb 17 16:18:18 2020 +0300 @@ -112,7 +112,6 @@ njs_array_buffer_is_view(njs_vm_t *vm, n static const njs_object_prop_t njs_array_buffer_constructor_properties[] = { - /* ArrayBuffer.name == "ArrayBuffer". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -120,7 +119,6 @@ static const njs_object_prop_t njs_arra .configurable = 1, }, - /* ArrayBuffer.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -128,14 +126,12 @@ static const njs_object_prop_t njs_arra .configurable = 1, }, - /* ArrayBuffer.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), .value = njs_prop_handler(njs_object_prototype_create), }, - /* ArrayBuffer[Symbol.species] */ { .type = NJS_PROPERTY, .name = njs_wellknown_symbol(NJS_SYMBOL_SPECIES), @@ -147,7 +143,6 @@ static const njs_object_prop_t njs_arra .enumerable = 0, }, - /* ArrayBuffer.isView(new Uint8Array()) === true */ { .type = NJS_PROPERTY, .name = njs_string("isView"), diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_boolean.c --- a/src/njs_boolean.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_boolean.c Mon Feb 17 16:18:18 2020 +0300 @@ -40,7 +40,6 @@ njs_boolean_constructor(njs_vm_t *vm, nj static const njs_object_prop_t njs_boolean_constructor_properties[] = { - /* Boolean.name == "Boolean". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -48,7 +47,6 @@ static const njs_object_prop_t njs_bool .configurable = 1, }, - /* Boolean.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -56,7 +54,6 @@ static const njs_object_prop_t njs_bool .configurable = 1, }, - /* Boolean.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_date.c --- a/src/njs_date.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_date.c Mon Feb 17 16:18:18 2020 +0300 @@ -1057,7 +1057,6 @@ njs_date_number_parse(int64_t *value, co static const njs_object_prop_t njs_date_constructor_properties[] = { - /* Date.name == "Date". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -1065,7 +1064,6 @@ static const njs_object_prop_t njs_date .configurable = 1, }, - /* Date.length == 7. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -1073,7 +1071,6 @@ static const njs_object_prop_t njs_date .configurable = 1, }, - /* Date.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_dtoa_fixed.c --- a/src/njs_dtoa_fixed.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_dtoa_fixed.c Mon Feb 17 16:18:18 2020 +0300 @@ -378,16 +378,6 @@ njs_fixed_dtoa(double value, njs_uint_t dividend = significand; - /* - * Let v = f * 2^e with f == significand and e == exponent. - * Then need q (quotient) and r (remainder) as follows: - * f * 2^e = q * 5^17 * 2^17 + r - * If e > 17 then - * f * 2^(e-17) = q * 5^17 + r/2^17 - * else - * f = q * 5^17 * 2^(17-e) + r/2^e - */ - if (exponent > 17) { /* (e - 17) <= 3. */ dividend <<= exponent - 17; diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_error.c --- a/src/njs_error.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_error.c Mon Feb 17 16:18:18 2020 +0300 @@ -297,7 +297,6 @@ njs_error_constructor(njs_vm_t *vm, njs_ static const njs_object_prop_t njs_error_constructor_properties[] = { - /* Error.name == "Error". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -305,7 +304,6 @@ static const njs_object_prop_t njs_erro .configurable = 1, }, - /* Error.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -313,7 +311,6 @@ static const njs_object_prop_t njs_erro .configurable = 1, }, - /* Error.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), @@ -338,7 +335,6 @@ njs_eval_error_constructor(njs_vm_t *vm, static const njs_object_prop_t njs_eval_error_constructor_properties[] = { - /* EvalError.name == "EvalError". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -346,7 +342,6 @@ static const njs_object_prop_t njs_eval .configurable = 1, }, - /* EvalError.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -354,7 +349,6 @@ static const njs_object_prop_t njs_eval .configurable = 1, }, - /* EvalError.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), @@ -379,7 +373,6 @@ njs_internal_error_constructor(njs_vm_t static const njs_object_prop_t njs_internal_error_constructor_properties[] = { - /* InternalError.name == "InternalError". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -387,7 +380,6 @@ static const njs_object_prop_t njs_inte .configurable = 1, }, - /* InternalError.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -395,7 +387,6 @@ static const njs_object_prop_t njs_inte .configurable = 1, }, - /* InternalError.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), @@ -420,7 +411,6 @@ njs_range_error_constructor(njs_vm_t *vm static const njs_object_prop_t njs_range_error_constructor_properties[] = { - /* RangeError.name == "RangeError". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -428,7 +418,6 @@ static const njs_object_prop_t njs_rang .configurable = 1, }, - /* RangeError.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -436,7 +425,6 @@ static const njs_object_prop_t njs_rang .configurable = 1, }, - /* RangeError.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), @@ -461,7 +449,6 @@ njs_reference_error_constructor(njs_vm_t static const njs_object_prop_t njs_reference_error_constructor_properties[] = { - /* ReferenceError.name == "ReferenceError". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -469,7 +456,6 @@ static const njs_object_prop_t njs_refe .configurable = 1, }, - /* ReferenceError.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -477,7 +463,6 @@ static const njs_object_prop_t njs_refe .configurable = 1, }, - /* ReferenceError.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), @@ -502,7 +487,6 @@ njs_syntax_error_constructor(njs_vm_t *v static const njs_object_prop_t njs_syntax_error_constructor_properties[] = { - /* SyntaxError.name == "SyntaxError". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -510,7 +494,6 @@ static const njs_object_prop_t njs_synt .configurable = 1, }, - /* SyntaxError.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -518,7 +501,6 @@ static const njs_object_prop_t njs_synt .configurable = 1, }, - /* SyntaxError.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), @@ -543,7 +525,6 @@ njs_type_error_constructor(njs_vm_t *vm, static const njs_object_prop_t njs_type_error_constructor_properties[] = { - /* TypeError.name == "TypeError". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -551,7 +532,6 @@ static const njs_object_prop_t njs_type .configurable = 1, }, - /* TypeError.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -559,7 +539,6 @@ static const njs_object_prop_t njs_type .configurable = 1, }, - /* TypeError.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), @@ -584,7 +563,6 @@ njs_uri_error_constructor(njs_vm_t *vm, static const njs_object_prop_t njs_uri_error_constructor_properties[] = { - /* URIError.name == "URIError". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -592,7 +570,6 @@ static const njs_object_prop_t njs_uri_ .configurable = 1, }, - /* URIError.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -600,7 +577,6 @@ static const njs_object_prop_t njs_uri_ .configurable = 1, }, - /* URIError.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), @@ -686,7 +662,6 @@ njs_memory_error_prototype_create(njs_vm static const njs_object_prop_t njs_memory_error_constructor_properties[] = { - /* MemoryError.name == "MemoryError". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -694,7 +669,6 @@ static const njs_object_prop_t njs_memo .configurable = 1, }, - /* MemoryError.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -702,7 +676,6 @@ static const njs_object_prop_t njs_memo .configurable = 1, }, - /* MemoryError.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_function.c --- a/src/njs_function.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_function.c Mon Feb 17 16:18:18 2020 +0300 @@ -217,9 +217,6 @@ njs_function_copy(njs_vm_t *vm, njs_func } -/* - * ES5.1, 10.6: CreateArgumentsObject. - */ njs_int_t njs_function_arguments_object_init(njs_vm_t *vm, njs_native_frame_t *frame) { @@ -973,7 +970,6 @@ njs_function_constructor(njs_vm_t *vm, n static const njs_object_prop_t njs_function_constructor_properties[] = { - /* Function.name == "Function". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -981,7 +977,6 @@ static const njs_object_prop_t njs_func .configurable = 1, }, - /* Function.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -989,7 +984,6 @@ static const njs_object_prop_t njs_func .configurable = 1, }, - /* Function.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), @@ -1004,10 +998,6 @@ const njs_object_init_t njs_function_co }; -/* - * ES5.1, 15.3.5.1 length - * the typical number of arguments expected by the function. - */ static njs_int_t njs_function_instance_length(njs_vm_t *vm, njs_object_prop_t *prop, njs_value_t *value, njs_value_t *setval, njs_value_t *retval) diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_math.c --- a/src/njs_math.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_math.c Mon Feb 17 16:18:18 2020 +0300 @@ -1063,7 +1063,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("acosh"), @@ -1080,7 +1079,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("asinh"), @@ -1105,7 +1103,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("atanh"), @@ -1114,7 +1111,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("cbrt"), @@ -1131,7 +1127,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("clz32"), @@ -1148,7 +1143,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("cosh"), @@ -1165,7 +1159,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("expm1"), @@ -1182,7 +1175,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("fround"), @@ -1191,7 +1183,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("hypot"), @@ -1200,7 +1191,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("imul"), @@ -1217,7 +1207,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("log10"), @@ -1226,7 +1215,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("log1p"), @@ -1235,7 +1223,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("log2"), @@ -1284,7 +1271,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("sign"), @@ -1301,7 +1287,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("sinh"), @@ -1326,7 +1311,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("tanh"), @@ -1335,7 +1319,6 @@ static const njs_object_prop_t njs_math .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("trunc"), diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_number.c --- a/src/njs_number.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_number.c Mon Feb 17 16:18:18 2020 +0300 @@ -420,7 +420,6 @@ njs_number_is_finite(njs_vm_t *vm, njs_v static const njs_object_prop_t njs_number_constructor_properties[] = { - /* Number.name == "Number". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -428,7 +427,6 @@ static const njs_object_prop_t njs_numb .configurable = 1, }, - /* Number.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -436,28 +434,24 @@ static const njs_object_prop_t njs_numb .configurable = 1, }, - /* Number.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), .value = njs_prop_handler(njs_object_prototype_create), }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("EPSILON"), .value = njs_value(NJS_NUMBER, 1, DBL_EPSILON), }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_long_string("MAX_SAFE_INTEGER"), .value = njs_value(NJS_NUMBER, 1, NJS_MAX_SAFE_INTEGER), }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_long_string("MIN_SAFE_INTEGER"), @@ -494,7 +488,6 @@ static const njs_object_prop_t njs_numb .value = njs_value(NJS_NUMBER, 1, -INFINITY), }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("isFinite"), @@ -503,7 +496,6 @@ static const njs_object_prop_t njs_numb .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("isInteger"), @@ -512,7 +504,6 @@ static const njs_object_prop_t njs_numb .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("isSafeInteger"), @@ -521,7 +512,6 @@ static const njs_object_prop_t njs_numb .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("isNaN"), @@ -530,7 +520,6 @@ static const njs_object_prop_t njs_numb .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("parseFloat"), @@ -539,7 +528,6 @@ static const njs_object_prop_t njs_numb .configurable = 1, }, - /* ES6. */ { .type = NJS_PROPERTY, .name = njs_string("parseInt"), diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_object.c --- a/src/njs_object.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_object.c Mon Feb 17 16:18:18 2020 +0300 @@ -2084,7 +2084,6 @@ njs_property_prototype_create(njs_vm_t * static const njs_object_prop_t njs_object_constructor_properties[] = { - /* Object.name == "Object". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -2092,7 +2091,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -2100,14 +2098,12 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), .value = njs_prop_handler(njs_object_prototype_create), }, - /* Object.create(). */ { .type = NJS_PROPERTY, .name = njs_string("create"), @@ -2116,7 +2112,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.keys(). */ { .type = NJS_PROPERTY, .name = njs_string("keys"), @@ -2125,7 +2120,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* ES8: Object.values(). */ { .type = NJS_PROPERTY, .name = njs_string("values"), @@ -2134,7 +2128,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* ES8: Object.entries(). */ { .type = NJS_PROPERTY, .name = njs_string("entries"), @@ -2143,7 +2136,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.defineProperty(). */ { .type = NJS_PROPERTY, .name = njs_string("defineProperty"), @@ -2152,7 +2144,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.defineProperties(). */ { .type = NJS_PROPERTY, .name = njs_long_string("defineProperties"), @@ -2161,7 +2152,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.getOwnPropertyDescriptor(). */ { .type = NJS_PROPERTY, .name = njs_long_string("getOwnPropertyDescriptor"), @@ -2170,7 +2160,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.getOwnPropertyDescriptors(). */ { .type = NJS_PROPERTY, .name = njs_long_string("getOwnPropertyDescriptors"), @@ -2180,7 +2169,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.getOwnPropertyNames(). */ { .type = NJS_PROPERTY, .name = njs_long_string("getOwnPropertyNames"), @@ -2190,7 +2178,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.getOwnPropertySymbols(). */ { .type = NJS_PROPERTY, .name = njs_long_string("getOwnPropertySymbols"), @@ -2200,7 +2187,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.getPrototypeOf(). */ { .type = NJS_PROPERTY, .name = njs_string("getPrototypeOf"), @@ -2209,7 +2195,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.setPrototypeOf(). */ { .type = NJS_PROPERTY, .name = njs_string("setPrototypeOf"), @@ -2218,7 +2203,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.freeze(). */ { .type = NJS_PROPERTY, .name = njs_string("freeze"), @@ -2227,7 +2211,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.isFrozen(). */ { .type = NJS_PROPERTY, .name = njs_string("isFrozen"), @@ -2236,7 +2219,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.seal(). */ { .type = NJS_PROPERTY, .name = njs_string("seal"), @@ -2245,7 +2227,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.isSealed(). */ { .type = NJS_PROPERTY, .name = njs_string("isSealed"), @@ -2254,7 +2235,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.preventExtensions(). */ { .type = NJS_PROPERTY, .name = njs_long_string("preventExtensions"), @@ -2263,7 +2243,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.isExtensible(). */ { .type = NJS_PROPERTY, .name = njs_string("isExtensible"), @@ -2272,7 +2251,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.assign(). */ { .type = NJS_PROPERTY, .name = njs_string("assign"), @@ -2281,7 +2259,6 @@ static const njs_object_prop_t njs_obje .configurable = 1, }, - /* Object.is(). */ { .type = NJS_PROPERTY, .name = njs_string("is"), @@ -2298,9 +2275,6 @@ const njs_object_init_t njs_object_cons }; -/* - * ES6, 9.1.2: [[SetPrototypeOf]]. - */ static njs_int_t njs_object_set_prototype(njs_vm_t *vm, njs_object_t *object, const njs_value_t *value) diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_regexp.c --- a/src/njs_regexp.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_regexp.c Mon Feb 17 16:18:18 2020 +0300 @@ -1224,7 +1224,6 @@ njs_regexp_string_create(njs_vm_t *vm, n static const njs_object_prop_t njs_regexp_constructor_properties[] = { - /* RegExp.name == "RegExp". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -1232,7 +1231,6 @@ static const njs_object_prop_t njs_rege .configurable = 1, }, - /* RegExp.length == 2. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -1240,7 +1238,6 @@ static const njs_object_prop_t njs_rege .configurable = 1, }, - /* RegExp.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), diff -r 13dbdff9b76f -r cbc09d7edfc8 src/njs_string.c --- a/src/njs_string.c Mon Feb 17 16:13:43 2020 +0300 +++ b/src/njs_string.c Mon Feb 17 16:18:18 2020 +0300 @@ -581,7 +581,6 @@ njs_string_constructor(njs_vm_t *vm, njs static const njs_object_prop_t njs_string_constructor_properties[] = { - /* String.name == "String". */ { .type = NJS_PROPERTY, .name = njs_string("name"), @@ -589,7 +588,6 @@ static const njs_object_prop_t njs_stri .configurable = 1, }, - /* String.length == 1. */ { .type = NJS_PROPERTY, .name = njs_string("length"), @@ -597,14 +595,12 @@ static const njs_object_prop_t njs_stri .configurable = 1, }, - /* String.prototype. */ { .type = NJS_PROPERTY_HANDLER, .name = njs_string("prototype"), .value = njs_prop_handler(njs_object_prototype_create), }, - /* String.bytesFrom(). */ { .type = NJS_PROPERTY, .name = njs_string("bytesFrom"), @@ -613,7 +609,6 @@ static const njs_object_prop_t njs_stri .configurable = 1, }, - /* String.fromCharCode(). */ { .type = NJS_PROPERTY, .name = njs_string("fromCharCode"), @@ -622,7 +617,6 @@ static const njs_object_prop_t njs_stri .configurable = 1, }, - /* String.fromCodePoint(), ECMAScript 6. */ { .type = NJS_PROPERTY, .name = njs_string("fromCodePoint"), @@ -869,11 +863,6 @@ njs_string_prototype_to_string(njs_vm_t } -/* - * String.concat(string2[, ..., stringN]). - * JavaScript 1.2, ECMAScript 3. - */ - njs_int_t njs_string_prototype_concat(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) @@ -1168,11 +1157,6 @@ njs_string_prototype_to_bytes(njs_vm_t * } -/* - * String.slice(start[, end]). - * JavaScript 1.2, ECMAScript 3. - */ - static njs_int_t njs_string_prototype_slice(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) @@ -1195,11 +1179,6 @@ njs_string_prototype_slice(njs_vm_t *vm, } -/* - * String.substring(start[, end]). - * JavaScript 1.0, ECMAScript 1. - */ - static njs_int_t From xeioex at nginx.com Mon Feb 17 14:01:47 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Mon, 17 Feb 2020 14:01:47 +0000 Subject: [njs] Simplified code using magic arguments introduced in 6df48738a043. Message-ID: details: https://hg.nginx.org/njs/rev/db3a66bd71c1 branches: changeset: 1328:db3a66bd71c1 user: Dmitry Volyntsev date: Mon Feb 17 16:18:38 2020 +0300 description: Simplified code using magic arguments introduced in 6df48738a043. This reduces code duplication and the binary size. diffstat: src/njs_error.c | 92 +--- src/njs_math.c | 1136 ++++++++++++----------------------------------------- src/njs_string.c | 118 +---- 3 files changed, 306 insertions(+), 1040 deletions(-) diffs (truncated from 1853 to 1000 lines): diff -r cbc09d7edfc8 -r db3a66bd71c1 src/njs_error.c --- a/src/njs_error.c Mon Feb 17 16:18:18 2020 +0300 +++ b/src/njs_error.c Mon Feb 17 16:18:38 2020 +0300 @@ -257,8 +257,8 @@ memory_error: static njs_int_t -njs_error_create(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_object_type_t type) +njs_error_constructor(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t type) { njs_int_t ret; njs_value_t *value; @@ -287,14 +287,6 @@ njs_error_create(njs_vm_t *vm, njs_value } -static njs_int_t -njs_error_constructor(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - return njs_error_create(vm, args, nargs, NJS_OBJ_TYPE_ERROR); -} - - static const njs_object_prop_t njs_error_constructor_properties[] = { { @@ -325,14 +317,6 @@ const njs_object_init_t njs_error_const }; -static njs_int_t -njs_eval_error_constructor(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - return njs_error_create(vm, args, nargs, NJS_OBJ_TYPE_EVAL_ERROR); -} - - static const njs_object_prop_t njs_eval_error_constructor_properties[] = { { @@ -363,14 +347,6 @@ const njs_object_init_t njs_eval_error_ }; -static njs_int_t -njs_internal_error_constructor(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) -{ - return njs_error_create(vm, args, nargs, NJS_OBJ_TYPE_INTERNAL_ERROR); -} - - static const njs_object_prop_t njs_internal_error_constructor_properties[] = { { @@ -401,14 +377,6 @@ const njs_object_init_t njs_internal_er }; -static njs_int_t -njs_range_error_constructor(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) -{ - return njs_error_create(vm, args, nargs, NJS_OBJ_TYPE_RANGE_ERROR); -} - - static const njs_object_prop_t njs_range_error_constructor_properties[] = { { @@ -439,14 +407,6 @@ const njs_object_init_t njs_range_error }; -static njs_int_t -njs_reference_error_constructor(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) -{ - return njs_error_create(vm, args, nargs, NJS_OBJ_TYPE_REF_ERROR); -} - - static const njs_object_prop_t njs_reference_error_constructor_properties[] = { { @@ -477,14 +437,6 @@ const njs_object_init_t njs_reference_e }; -static njs_int_t -njs_syntax_error_constructor(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) -{ - return njs_error_create(vm, args, nargs, NJS_OBJ_TYPE_SYNTAX_ERROR); -} - - static const njs_object_prop_t njs_syntax_error_constructor_properties[] = { { @@ -515,14 +467,6 @@ const njs_object_init_t njs_syntax_erro }; -static njs_int_t -njs_type_error_constructor(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) -{ - return njs_error_create(vm, args, nargs, NJS_OBJ_TYPE_TYPE_ERROR); -} - - static const njs_object_prop_t njs_type_error_constructor_properties[] = { { @@ -553,14 +497,6 @@ const njs_object_init_t njs_type_error_ }; -static njs_int_t -njs_uri_error_constructor(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) -{ - return njs_error_create(vm, args, nargs, NJS_OBJ_TYPE_URI_ERROR); -} - - static const njs_object_prop_t njs_uri_error_constructor_properties[] = { { @@ -874,7 +810,8 @@ const njs_object_init_t njs_error_proto const njs_object_type_init_t njs_error_type_init = { - .constructor = njs_native_ctor(njs_error_constructor, 1, 0), + .constructor = njs_native_ctor(njs_error_constructor, 1, + NJS_OBJ_TYPE_ERROR), .constructor_props = &njs_error_constructor_init, .prototype_props = &njs_error_prototype_init, .prototype_value = { .object = { .type = NJS_OBJECT } }, @@ -916,7 +853,8 @@ const njs_object_init_t njs_eval_error_ const njs_object_type_init_t njs_eval_error_type_init = { - .constructor = njs_native_ctor(njs_eval_error_constructor, 1, 0), + .constructor = njs_native_ctor(njs_error_constructor, 1, + NJS_OBJ_TYPE_EVAL_ERROR), .constructor_props = &njs_eval_error_constructor_init, .prototype_props = &njs_eval_error_prototype_init, .prototype_value = { .object = { .type = NJS_OBJECT } }, @@ -978,7 +916,8 @@ const njs_object_init_t njs_internal_er const njs_object_type_init_t njs_internal_error_type_init = { - .constructor = njs_native_ctor(njs_internal_error_constructor, 1, 0), + .constructor = njs_native_ctor(njs_error_constructor, 1, + NJS_OBJ_TYPE_INTERNAL_ERROR), .constructor_props = &njs_internal_error_constructor_init, .prototype_props = &njs_internal_error_prototype_init, .prototype_value = { .object = { .type = NJS_OBJECT } }, @@ -1028,7 +967,8 @@ const njs_object_init_t njs_range_error const njs_object_type_init_t njs_range_error_type_init = { - .constructor = njs_native_ctor(njs_range_error_constructor, 1, 0), + .constructor = njs_native_ctor(njs_error_constructor, 1, + NJS_OBJ_TYPE_RANGE_ERROR), .constructor_props = &njs_range_error_constructor_init, .prototype_props = &njs_range_error_prototype_init, .prototype_value = { .object = { .type = NJS_OBJECT } }, @@ -1070,7 +1010,8 @@ const njs_object_init_t njs_reference_e const njs_object_type_init_t njs_reference_error_type_init = { - .constructor = njs_native_ctor(njs_reference_error_constructor, 1, 0), + .constructor = njs_native_ctor(njs_error_constructor, 1, + NJS_OBJ_TYPE_REF_ERROR), .constructor_props = &njs_reference_error_constructor_init, .prototype_props = &njs_reference_error_prototype_init, .prototype_value = { .object = { .type = NJS_OBJECT } }, @@ -1112,7 +1053,8 @@ const njs_object_init_t njs_syntax_erro const njs_object_type_init_t njs_syntax_error_type_init = { - .constructor = njs_native_ctor(njs_syntax_error_constructor, 1, 0), + .constructor = njs_native_ctor(njs_error_constructor, 1, + NJS_OBJ_TYPE_SYNTAX_ERROR), .constructor_props = &njs_syntax_error_constructor_init, .prototype_props = &njs_syntax_error_prototype_init, .prototype_value = { .object = { .type = NJS_OBJECT } }, @@ -1154,7 +1096,8 @@ const njs_object_init_t njs_type_error_ const njs_object_type_init_t njs_type_error_type_init = { - .constructor = njs_native_ctor(njs_type_error_constructor, 1, 0), + .constructor = njs_native_ctor(njs_error_constructor, 1, + NJS_OBJ_TYPE_TYPE_ERROR), .constructor_props = &njs_type_error_constructor_init, .prototype_props = &njs_type_error_prototype_init, .prototype_value = { .object = { .type = NJS_OBJECT } }, @@ -1196,7 +1139,8 @@ const njs_object_init_t njs_uri_error_p const njs_object_type_init_t njs_uri_error_type_init = { - .constructor = njs_native_ctor(njs_uri_error_constructor, 1, 0), + .constructor = njs_native_ctor(njs_error_constructor, 1, + NJS_OBJ_TYPE_URI_ERROR), .constructor_props = &njs_uri_error_constructor_init, .prototype_props = &njs_uri_error_prototype_init, .prototype_value = { .object = { .type = NJS_OBJECT } }, diff -r cbc09d7edfc8 -r db3a66bd71c1 src/njs_math.c --- a/src/njs_math.c Mon Feb 17 16:18:18 2020 +0300 +++ b/src/njs_math.c Mon Feb 17 16:18:38 2020 +0300 @@ -8,444 +8,267 @@ #include -static njs_int_t -njs_object_math_abs(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, fabs(njs_number(&args[1]))); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_acos(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - double num; - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - num = njs_number(&args[1]); - -#if (NJS_SOLARIS) - /* On Solaris acos(x) returns 0 for x > 1. */ - if (fabs(num) > 1.0) { - num = NAN; - } -#endif - - njs_set_number(&vm->retval, acos(num)); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_acosh(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, acosh(njs_number(&args[1]))); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_asin(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - double num; - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - num = njs_number(&args[1]); - -#if (NJS_SOLARIS) - /* On Solaris asin(x) returns 0 for x > 1. */ - if (fabs(num) > 1.0) { - num = NAN; - } -#endif - - njs_set_number(&vm->retval, asin(num)); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_asinh(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, asinh(njs_number(&args[1]))); - - return NJS_OK; -} +typedef enum { + NJS_MATH_ABS, + NJS_MATH_ACOS, + NJS_MATH_ACOSH, + NJS_MATH_ASIN, + NJS_MATH_ASINH, + NJS_MATH_ATAN, + NJS_MATH_ATAN2, + NJS_MATH_ATANH, + NJS_MATH_CBRT, + NJS_MATH_CEIL, + NJS_MATH_CLZ32, + NJS_MATH_COS, + NJS_MATH_COSH, + NJS_MATH_EXP, + NJS_MATH_EXPM1, + NJS_MATH_FLOOR, + NJS_MATH_FROUND, + NJS_MATH_IMUL, + NJS_MATH_LOG, + NJS_MATH_LOG10, + NJS_MATH_LOG1P, + NJS_MATH_LOG2, + NJS_MATH_POW, + NJS_MATH_ROUND, + NJS_MATH_SIGN, + NJS_MATH_SIN, + NJS_MATH_SINH, + NJS_MATH_SQRT, + NJS_MATH_TAN, + NJS_MATH_TANH, + NJS_MATH_TRUNC, +} njs_math_func_t; static njs_int_t -njs_object_math_atan(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) +njs_object_math_func(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t magic) { - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, atan(njs_number(&args[1]))); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_atan2(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - double y, x; - njs_int_t ret; + double num, num2; + uint8_t sign; + uint32_t u32; + uint64_t one, fraction_mask; + njs_int_t ret, ep; + njs_math_func_t func; + njs_diyfp_conv_t conv; - if (njs_slow_path(nargs < 3)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - if (njs_slow_path(!njs_is_number(&args[2]))) { - ret = njs_value_to_numeric(vm, &args[2], &args[2]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } + func = magic; - y = njs_number(&args[1]); - x = njs_number(&args[2]); - - njs_set_number(&vm->retval, atan2(y, x)); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_atanh(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; + ret = njs_value_to_number(vm, njs_arg(args, nargs, 1), &num); + if (njs_slow_path(ret != NJS_OK)) { + return ret; } - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; + switch (func) { + case NJS_MATH_ABS: + num = fabs(num); + break; + + case NJS_MATH_ACOS: +#if (NJS_SOLARIS) + /* On Solaris acos(x) returns 0 for x > 1. */ + if (fabs(num) > 1.0) { + num = NAN; } - } +#endif - njs_set_number(&vm->retval, atanh(njs_number(&args[1]))); + num = acos(num); + break; + + case NJS_MATH_ACOSH: + num = acosh(num); + break; - return NJS_OK; -} + case NJS_MATH_ASIN: +#if (NJS_SOLARIS) + /* On Solaris asin(x) returns 0 for x > 1. */ + if (fabs(num) > 1.0) { + num = NAN; + } +#endif + + num = asin(num); + break; + case NJS_MATH_ASINH: + num = asinh(num); + break; + + case NJS_MATH_ATAN: + num = atan(num); + break; + + case NJS_MATH_ATANH: + num = atanh(num); + break; -static njs_int_t -njs_object_math_cbrt(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; + case NJS_MATH_CBRT: + num = cbrt(num); + break; + + case NJS_MATH_CEIL: + num = ceil(num); + break; - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } + case NJS_MATH_CLZ32: + u32 = njs_number_to_uint32(num); + num = njs_leading_zeros(u32); + break; + + case NJS_MATH_COS: + num = cos(num); + break; + + case NJS_MATH_COSH: + num = cosh(num); + break; - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } + case NJS_MATH_EXP: + num = exp(num); + break; + + case NJS_MATH_EXPM1: + num = expm1(num); + break; + + case NJS_MATH_FLOOR: + num = floor(num); + break; - njs_set_number(&vm->retval, cbrt(njs_number(&args[1]))); + case NJS_MATH_FROUND: + num = (float) num; + break; - return NJS_OK; -} + case NJS_MATH_LOG: + num = log(num); + break; + case NJS_MATH_LOG10: + num = log10(num); + break; -static njs_int_t -njs_object_math_ceil(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; + case NJS_MATH_LOG1P: + num = log1p(num); + break; + + case NJS_MATH_LOG2: +#if (NJS_SOLARIS) + /* On Solaris 10 log(-1) returns -Infinity. */ + if (num < 0) { + num = NAN; + } +#endif + + num = log2(num); + break; + + case NJS_MATH_SIGN: + if (!isnan(num) && num != 0) { + num = signbit(num) ? -1 : 1; + } - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } + break; + + case NJS_MATH_SIN: + num = sin(num); + break; + + case NJS_MATH_SINH: + num = sinh(num); + break; - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } + case NJS_MATH_SQRT: + num = sqrt(num); + break; + + case NJS_MATH_ROUND: + conv.d = num; + ep = (conv.u64 & NJS_DBL_EXPONENT_MASK) >> NJS_DBL_SIGNIFICAND_SIZE; + + if (ep < NJS_DBL_EXPONENT_OFFSET) { + + /* |v| < 1. */ - njs_set_number(&vm->retval, ceil(njs_number(&args[1]))); + if (ep == (NJS_DBL_EXPONENT_OFFSET - 1) + && conv.u64 != njs_uint64(0xbfe00000, 0x00000000)) + { + /* (|v| > 0.5 || v == 0.5) => +-1.0 */ + + conv.u64 = conv.u64 & NJS_DBL_SIGN_MASK; + conv.u64 |= NJS_DBL_EXPONENT_OFFSET << NJS_DBL_SIGNIFICAND_SIZE; + + } else { - return NJS_OK; -} + /* (|v| < 0.5 || v == -0.5) => +-0. */ + + conv.u64 &= ((uint64_t) 1) << 63; + } + } else if (ep < NJS_DBL_EXPONENT_BIAS) { + + /* |v| <= 2^52 - 1 (largest safe integer). */ + + one = ((uint64_t) 1) << (NJS_DBL_EXPONENT_BIAS - ep); + fraction_mask = one - 1; -static njs_int_t -njs_object_math_clz32(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - uint32_t ui32; - njs_int_t ret; + /* truncation. */ + + sign = conv.u64 >> 63; + conv.u64 += (one >> 1) - sign; + conv.u64 &= ~fraction_mask; + } + + num = conv.d; + break; - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, 32); - return NJS_OK; - } + case NJS_MATH_TAN: + num = tan(num); + break; - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_uint32(vm, &args[1], &ui32); + case NJS_MATH_TANH: + num = tanh(num); + break; + + case NJS_MATH_TRUNC: + num = trunc(num); + break; + + default: + ret = njs_value_to_number(vm, njs_arg(args, nargs, 2), &num2); if (njs_slow_path(ret != NJS_OK)) { return ret; } - } else { - ui32 = njs_number_to_uint32(njs_number(&args[1])); - } - - njs_set_number(&vm->retval, njs_leading_zeros(ui32)); - - return NJS_OK; -} - + switch (func) { + case NJS_MATH_ATAN2: + num = atan2(num, num2); + break; -static njs_int_t -njs_object_math_cos(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } + case NJS_MATH_IMUL: + u32 = njs_number_to_uint32(num); + num = (int32_t) (u32 * njs_number_to_uint32(num2)); + break; - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, cos(njs_number(&args[1]))); - - return NJS_OK; -} - + default: + /* + * According to ECMA-262: + * 1. If exponent is NaN, the result should be NaN; + * 2. The result of Math.pow(+/-1, +/-Infinity) should be NaN. + */ -static njs_int_t -njs_object_math_cosh(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; + if (fabs(num) != 1 || (!isnan(num2) && !isinf(num2))) { + num = pow(num, num2); - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; + } else { + num = NAN; + } } } - njs_set_number(&vm->retval, cosh(njs_number(&args[1]))); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_exp(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, exp(njs_number(&args[1]))); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_expm1(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, expm1(njs_number(&args[1]))); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_floor(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, floor(njs_number(&args[1]))); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_fround(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, (float) njs_number(&args[1])); + njs_set_number(&vm->retval, num); return NJS_OK; } @@ -459,21 +282,22 @@ njs_object_math_hypot(njs_vm_t *vm, njs_ njs_int_t ret; njs_uint_t i; - for (i = 1; i < nargs; i++) { - if (!njs_is_numeric(&args[i])) { - ret = njs_value_to_numeric(vm, &args[i], &args[i]); - if (ret != NJS_OK) { - return ret; - } - } + ret = njs_value_to_number(vm, njs_arg(args, nargs, 1), &num); + if (njs_slow_path(ret != NJS_OK)) { + return ret; } - num = (nargs > 1) ? fabs(njs_number(&args[1])) : 0; + num = (nargs > 1) ? fabs(num) : 0; for (i = 2; i < nargs; i++) { + ret = njs_value_to_numeric(vm, &args[i], &args[i]); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + num = hypot(num, njs_number(&args[i])); - if (num == INFINITY) { + if (njs_slow_path(isinf(num))) { break; } } @@ -484,150 +308,6 @@ njs_object_math_hypot(njs_vm_t *vm, njs_ } -static njs_int_t -njs_object_math_imul(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - uint32_t a, b; - njs_int_t ret; - - if (njs_slow_path(nargs < 3)) { - njs_set_number(&vm->retval, 0); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_uint32(vm, &args[1], &a); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - } else { - a = njs_number_to_uint32(njs_number(&args[1])); - } - - if (njs_slow_path(!njs_is_number(&args[2]))) { - ret = njs_value_to_uint32(vm, &args[2], &b); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - - } else { - b = njs_number_to_uint32(njs_number(&args[2])); - } - - njs_set_number(&vm->retval, (int32_t) (a * b)); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_log(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - njs_set_number(&vm->retval, log(njs_number(&args[1]))); - - return NJS_OK; -} - - -static njs_int_t -njs_object_math_log10(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) -{ - njs_int_t ret; - - if (njs_slow_path(nargs < 2)) { - njs_set_number(&vm->retval, NAN); - return NJS_OK; - } - - if (njs_slow_path(!njs_is_number(&args[1]))) { - ret = njs_value_to_numeric(vm, &args[1], &args[1]); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } From xeioex at nginx.com Mon Feb 17 14:01:49 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Mon, 17 Feb 2020 14:01:49 +0000 Subject: [njs] Fixed potential integer-overflow in String.prototype.replace(). Message-ID: details: https://hg.nginx.org/njs/rev/d2877d602d39 branches: changeset: 1329:d2877d602d39 user: Dmitry Volyntsev date: Mon Feb 17 16:18:40 2020 +0300 description: Fixed potential integer-overflow in String.prototype.replace(). diffstat: src/njs_string.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diffs (20 lines): diff -r db3a66bd71c1 -r d2877d602d39 src/njs_string.c --- a/src/njs_string.c Mon Feb 17 16:18:38 2020 +0300 +++ b/src/njs_string.c Mon Feb 17 16:18:40 2020 +0300 @@ -3672,10 +3672,16 @@ njs_string_replace_regexp_function(njs_v njs_value_t *arguments; njs_string_prop_t string; + if (njs_slow_path((n + 3) >= UINT32_MAX / sizeof(njs_value_t))) { + njs_memory_error(vm); + return NJS_ERROR; + } + njs_set_invalid(&r->retval); arguments = njs_mp_alloc(vm->mem_pool, (n + 3) * sizeof(njs_value_t)); if (njs_slow_path(arguments == NULL)) { + njs_memory_error(vm); return NJS_ERROR; } From xeioex at nginx.com Tue Feb 18 15:56:36 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 18 Feb 2020 15:56:36 +0000 Subject: [njs] Detecting memory error in njs_chb_*() functions. Message-ID: details: https://hg.nginx.org/njs/rev/aa9ac79f7fd9 branches: changeset: 1330:aa9ac79f7fd9 user: Dmitry Volyntsev date: Tue Feb 18 18:41:38 2020 +0300 description: Detecting memory error in njs_chb_*() functions. diffstat: src/njs_array.c | 9 +++++++-- src/njs_chb.c | 17 ++++++++++++----- src/njs_chb.h | 12 ++++++++++-- src/njs_json.c | 14 ++++++++------ 4 files changed, 37 insertions(+), 15 deletions(-) diffs (158 lines): diff -r d2877d602d39 -r aa9ac79f7fd9 src/njs_array.c --- a/src/njs_array.c Mon Feb 17 16:18:40 2020 +0300 +++ b/src/njs_array.c Tue Feb 18 18:41:38 2020 +0300 @@ -1381,8 +1381,8 @@ njs_array_prototype_join(njs_vm_t *vm, n njs_index_t unused) { u_char *p, *last; - int64_t length; - uint64_t i, len, size; + int64_t size, length; + uint64_t i, len; njs_int_t ret; njs_chb_t chain; njs_utf8_t utf8; @@ -1497,6 +1497,11 @@ njs_array_prototype_join(njs_vm_t *vm, n njs_chb_drop(&chain, separator.size); size = njs_chb_size(&chain); + if (njs_slow_path(size < 0)) { + njs_memory_error(vm); + return NJS_ERROR; + } + length -= separator.length; p = njs_string_alloc(vm, &vm->retval, size, utf8 ? length : 0); diff -r d2877d602d39 -r aa9ac79f7fd9 src/njs_chb.c --- a/src/njs_chb.c Mon Feb 17 16:18:40 2020 +0300 +++ b/src/njs_chb.c Tue Feb 18 18:41:38 2020 +0300 @@ -134,9 +134,13 @@ njs_chb_drain(njs_chb_t *chain, size_t d void njs_chb_drop(njs_chb_t *chain, size_t drop) { - size_t size; + uint64_t size; njs_chb_node_t *n, *next; + if (njs_slow_path(chain->error)) { + return; + } + n = chain->last; if (njs_fast_path(n != NULL && (njs_chb_node_size(n) > drop))) { @@ -145,7 +149,7 @@ njs_chb_drop(njs_chb_t *chain, size_t dr } n = chain->nodes; - size = njs_chb_size(chain); + size = (uint64_t) njs_chb_size(chain); if (drop >= size) { njs_chb_destroy(chain); @@ -181,10 +185,10 @@ njs_int_t njs_chb_join(njs_chb_t *chain, njs_str_t *str) { u_char *start; - size_t size; + uint64_t size; njs_chb_node_t *n; - if (chain->error) { + if (njs_slow_path(chain->error)) { return NJS_DECLINED; } @@ -196,7 +200,10 @@ njs_chb_join(njs_chb_t *chain, njs_str_t return NJS_OK; } - size = njs_chb_size(chain); + size = (uint64_t) njs_chb_size(chain); + if (njs_slow_path(size >= UINT32_MAX)) { + return NJS_ERROR; + } start = njs_mp_alloc(chain->pool, size); if (njs_slow_path(start == NULL)) { diff -r d2877d602d39 -r aa9ac79f7fd9 src/njs_chb.h --- a/src/njs_chb.h Mon Feb 17 16:18:40 2020 +0300 +++ b/src/njs_chb.h Tue Feb 18 18:41:38 2020 +0300 @@ -61,12 +61,16 @@ njs_chb_init(njs_chb_t *chain, njs_mp_t } -njs_inline uint64_t +njs_inline int64_t njs_chb_size(njs_chb_t *chain) { uint64_t size; njs_chb_node_t *n; + if (njs_slow_path(chain->error)) { + return -1; + } + n = chain->nodes; size = 0; @@ -86,6 +90,10 @@ njs_chb_utf8_length(njs_chb_t *chain) int64_t len, length; njs_chb_node_t *n; + if (njs_slow_path(chain->error)) { + return -1; + } + n = chain->nodes; length = 0; @@ -93,7 +101,7 @@ njs_chb_utf8_length(njs_chb_t *chain) while (n != NULL) { len = njs_utf8_length(n->start, njs_chb_node_size(n)); if (njs_slow_path(len < 0)) { - return len; + return 0; } length += len; diff -r d2877d602d39 -r aa9ac79f7fd9 src/njs_json.c --- a/src/njs_json.c Mon Feb 17 16:18:40 2020 +0300 +++ b/src/njs_json.c Tue Feb 18 18:41:38 2020 +0300 @@ -1120,8 +1120,7 @@ njs_json_stringify_iterator(njs_vm_t *vm njs_value_t *object) { u_char *p; - int64_t length; - uint64_t size; + int64_t size, length; njs_int_t ret; njs_chb_t chain; njs_value_t *key, *value, index, wrapper; @@ -1295,18 +1294,21 @@ done: } size = njs_chb_size(&chain); - if (njs_slow_path(size == 0)) { + if (njs_slow_path(size < 0)) { + njs_chb_destroy(&chain); + goto memory_error; + } + + if (size == 0) { njs_set_undefined(&vm->retval); goto release; } length = njs_chb_utf8_length(&chain); - if (njs_slow_path(length < 0)) { - length = 0; - } p = njs_string_alloc(vm, &vm->retval, size, length); if (njs_slow_path(p == NULL)) { + njs_chb_destroy(&chain); goto memory_error; } From xeioex at nginx.com Tue Feb 18 15:56:38 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 18 Feb 2020 15:56:38 +0000 Subject: [njs] Fixed njs_json_append_string(). Message-ID: details: https://hg.nginx.org/njs/rev/9b70f038abfa branches: changeset: 1331:9b70f038abfa user: Dmitry Volyntsev date: Tue Feb 18 18:42:37 2020 +0300 description: Fixed njs_json_append_string(). diffstat: src/njs_json.c | 91 ++++++++++++++++++++++++++------------------------------- 1 files changed, 42 insertions(+), 49 deletions(-) diffs (128 lines): diff -r aa9ac79f7fd9 -r 9b70f038abfa src/njs_json.c --- a/src/njs_json.c Tue Feb 18 18:41:38 2020 +0300 +++ b/src/njs_json.c Tue Feb 18 18:42:37 2020 +0300 @@ -1545,37 +1545,50 @@ njs_json_append_value(njs_chb_t *chain, static void njs_json_append_string(njs_chb_t *chain, const njs_value_t *value, char quote) { + size_t size; u_char c, *dst, *dst_end; - size_t length; + njs_bool_t utf8; const u_char *p, *end; - njs_string_prop_t str; - - static char hex2char[16] = { '0', '1', '2', '3', '4', '5', '6', '7', - '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' }; - - (void) njs_string_prop(&str, value); - - p = str.start; - end = p + str.size; - length = str.length; - - dst = njs_chb_reserve(chain, length + 2); + njs_string_prop_t string; + + static char hex2char[16] = { '0', '1', '2', '3', '4', '5', '6', '7', + '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' }; + + (void) njs_string_prop(&string, value); + + p = string.start; + end = p + string.size; + utf8 = (string.length != 0 && string.length != string.size); + + size = njs_max(string.size + 2, 7); + dst = njs_chb_reserve(chain, string.size + 2); if (njs_slow_path(dst == NULL)) { return; } - dst_end = dst + length + 2; + dst_end = dst + size; *dst++ = quote; + njs_chb_written(chain, 1); while (p < end) { - - if (*p < ' ' - || *p == '\\' - || (*p == '\"' && quote == '\"')) + if (njs_slow_path(dst_end <= dst + njs_length("\\uXXXX"))) { + size = njs_max(end - p + 1, 6); + dst = njs_chb_reserve(chain, size); + if (njs_slow_path(dst == NULL)) { + return; + } + + dst_end = dst + size; + } + + if (njs_slow_path(*p < ' ' + || *p == '\\' + || (*p == '\"' && quote == '\"'))) { c = (u_char) *p++; *dst++ = '\\'; + njs_chb_written(chain, 2); switch (c) { case '\\': @@ -1605,44 +1618,24 @@ njs_json_append_string(njs_chb_t *chain, *dst++ = '0'; *dst++ = hex2char[(c & 0xf0) >> 4]; *dst++ = hex2char[c & 0x0f]; - } - } - - /* - * Control characters less than space are encoded using 6 bytes - * "\uXXXX". Checking there is at least 6 bytes of destination storage - * space. - */ - - while (p < end && (dst_end - dst) > 6) { - if (*p < ' ' || (*p == '\"' && quote == '\"') || *p == '\\') { - break; + njs_chb_written(chain, 4); } - if (length != 0) { - /* UTF-8 or ASCII string. */ - dst = njs_utf8_copy(dst, &p, end); - - } else { - /* Byte string. */ - *dst++ = *p++; - } + continue; } - if (dst_end - dst <= 6) { - njs_chb_written(chain, dst - chain->last->pos); - - dst = njs_chb_reserve(chain, 64); - if (njs_slow_path(dst == NULL)) { - return; - } - - dst_end = dst + 64; + if (utf8) { + /* UTF-8 string. */ + dst = njs_utf8_copy(dst, &p, end); + + } else { + /* Byte or ASCII string. */ + *dst++ = *p++; } + + njs_chb_written(chain, dst - chain->last->pos); } - njs_chb_written(chain, dst - chain->last->pos); - njs_chb_append(chain, "e, 1); } From xeioex at nginx.com Tue Feb 18 15:56:40 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 18 Feb 2020 15:56:40 +0000 Subject: [njs] Fixed encodeURI() and decodeURI() according to the spec. Message-ID: details: https://hg.nginx.org/njs/rev/c60911765952 branches: changeset: 1332:c60911765952 user: Dmitry Volyntsev date: Tue Feb 18 18:55:50 2020 +0300 description: Fixed encodeURI() and decodeURI() according to the spec. diffstat: src/njs_builtin.c | 8 +- src/njs_json.c | 8 +- src/njs_parser_terminal.c | 12 +- src/njs_string.c | 509 +++++++++++++++++++++++++-------------------- src/njs_string.h | 14 +- src/test/njs_unit_test.c | 54 +++- 6 files changed, 342 insertions(+), 263 deletions(-) diffs (840 lines): diff -r 9b70f038abfa -r c60911765952 src/njs_builtin.c --- a/src/njs_builtin.c Tue Feb 18 18:42:37 2020 +0300 +++ b/src/njs_builtin.c Tue Feb 18 18:55:50 2020 +0300 @@ -1018,7 +1018,7 @@ static const njs_object_prop_t njs_glob { .type = NJS_PROPERTY, .name = njs_string("encodeURI"), - .value = njs_native_function(njs_string_encode_uri, 1), + .value = njs_native_function2(njs_string_encode_uri, 1, 0), .writable = 1, .configurable = 1, }, @@ -1026,7 +1026,7 @@ static const njs_object_prop_t njs_glob { .type = NJS_PROPERTY, .name = njs_long_string("encodeURIComponent"), - .value = njs_native_function(njs_string_encode_uri_component, 1), + .value = njs_native_function2(njs_string_encode_uri, 1, 1), .writable = 1, .configurable = 1, }, @@ -1034,7 +1034,7 @@ static const njs_object_prop_t njs_glob { .type = NJS_PROPERTY, .name = njs_string("decodeURI"), - .value = njs_native_function(njs_string_decode_uri, 1), + .value = njs_native_function2(njs_string_decode_uri, 1, 0), .writable = 1, .configurable = 1, }, @@ -1042,7 +1042,7 @@ static const njs_object_prop_t njs_glob { .type = NJS_PROPERTY, .name = njs_long_string("decodeURIComponent"), - .value = njs_native_function(njs_string_decode_uri_component, 1), + .value = njs_native_function2(njs_string_decode_uri, 1, 1), .writable = 1, .configurable = 1, }, diff -r 9b70f038abfa -r c60911765952 src/njs_json.c --- a/src/njs_json.c Tue Feb 18 18:42:37 2020 +0300 +++ b/src/njs_json.c Tue Feb 18 18:55:50 2020 +0300 @@ -684,9 +684,7 @@ njs_json_parse_string(njs_json_parse_ctx utf = njs_json_unicode(p); p += 4; - if (utf >= 0xd800 && utf <= 0xdfff) { - - /* Surrogate pair. */ + if (njs_surrogate_any(utf)) { if (utf > 0xdbff || p[0] != '\\' || p[1] != 'u') { s = njs_utf8_encode(s, NJS_UTF8_REPLACEMENT); @@ -698,10 +696,10 @@ njs_json_parse_string(njs_json_parse_ctx utf_low = njs_json_unicode(p); p += 4; - if (njs_fast_path(utf_low >= 0xdc00 && utf_low <= 0xdfff)) { + if (njs_fast_path(njs_surrogate_trailing(utf_low))) { utf = njs_string_surrogate_pair(utf, utf_low); - } else if (utf_low >= 0xd800 && utf_low <= 0xdbff) { + } else if (njs_surrogate_leading(utf_low)) { utf = NJS_UTF8_REPLACEMENT; s = njs_utf8_encode(s, NJS_UTF8_REPLACEMENT); diff -r 9b70f038abfa -r c60911765952 src/njs_parser_terminal.c --- a/src/njs_parser_terminal.c Tue Feb 18 18:42:37 2020 +0300 +++ b/src/njs_parser_terminal.c Tue Feb 18 18:55:50 2020 +0300 @@ -1111,10 +1111,10 @@ njs_parser_escape_string_create(njs_vm_t } if (cp_pair != 0) { - if (njs_fast_path(cp >= 0xdc00 && cp <= 0xdfff)) { + if (njs_fast_path(njs_surrogate_trailing(cp))) { cp = njs_string_surrogate_pair(cp_pair, cp); - } else if (njs_slow_path(cp >= 0xd800 && cp <= 0xdbff)) { + } else if (njs_slow_path(njs_surrogate_leading(cp))) { cp = NJS_UTF8_REPLACEMENT; dst = njs_utf8_encode(dst, (uint32_t) cp); @@ -1125,7 +1125,7 @@ njs_parser_escape_string_create(njs_vm_t cp_pair = 0; - } else if (cp >= 0xd800 && cp <= 0xdfff) { + } else if (njs_surrogate_any(cp)) { if (cp <= 0xdbff && src[0] == '\\' && src[1] == 'u') { cp_pair = cp; continue; @@ -1256,10 +1256,10 @@ njs_parser_escape_string_calc_length(njs } if (cp_pair != 0) { - if (njs_fast_path(cp >= 0xdc00 && cp <= 0xdfff)) { + if (njs_fast_path(njs_surrogate_trailing(cp))) { cp = njs_string_surrogate_pair(cp_pair, cp); - } else if (njs_slow_path(cp >= 0xd800 && cp <= 0xdbff)) { + } else if (njs_slow_path(njs_surrogate_leading(cp))) { cp = NJS_UTF8_REPLACEMENT; size += njs_utf8_size(cp); @@ -1272,7 +1272,7 @@ njs_parser_escape_string_calc_length(njs cp_pair = 0; - } else if (cp >= 0xd800 && cp <= 0xdfff) { + } else if (njs_surrogate_any(cp)) { if (cp <= 0xdbff && src[0] == '\\' && src[1] == 'u') { cp_pair = cp; continue; diff -r 9b70f038abfa -r c60911765952 src/njs_string.c --- a/src/njs_string.c Tue Feb 18 18:42:37 2020 +0300 +++ b/src/njs_string.c Tue Feb 18 18:55:50 2020 +0300 @@ -91,10 +91,6 @@ static njs_int_t njs_string_replace_subs static njs_int_t njs_string_replace_join(njs_vm_t *vm, njs_string_replace_t *r); static void njs_string_replacement_copy(njs_string_replace_part_t *string, const njs_value_t *value); -static njs_int_t njs_string_encode(njs_vm_t *vm, njs_value_t *value, - const uint32_t *escape); -static njs_int_t njs_string_decode(njs_vm_t *vm, njs_value_t *value, - const uint32_t *reserve); #define njs_base64_encoded_length(len) (((len + 2) / 3) * 4) @@ -4698,14 +4694,55 @@ const njs_object_init_t njs_string_inst }; +njs_inline njs_bool_t +njs_need_escape(const uint32_t *escape, uint32_t byte) +{ + return ((escape[byte >> 5] & ((uint32_t) 1 << (byte & 0x1f))) != 0); +} + + +njs_inline u_char * +njs_string_encode(const uint32_t *escape, size_t size, const u_char *src, + u_char *dst) +{ + uint8_t byte; + static const u_char hex[16] = "0123456789ABCDEF"; + + do { + byte = *src++; + + if (njs_need_escape(escape, byte)) { + *dst++ = '%'; + *dst++ = hex[byte >> 4]; + *dst++ = hex[byte & 0xf]; + + } else { + *dst++ = byte; + } + + size--; + + } while (size != 0); + + return dst; +} + + njs_int_t njs_string_encode_uri(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) + njs_index_t component) { - njs_int_t ret; - njs_value_t *value; - - static const uint32_t escape[] = { + u_char byte, *dst; + uint64_t size; + uint32_t cp, cp_low; + njs_int_t ret; + njs_value_t *value; + const u_char *src, *end; + const uint32_t *escape; + njs_string_prop_t string; + u_char encode[4]; + + static const uint32_t escape_uri[] = { 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ /* ?>=< ;:98 7654 3210 /.-, +*)( '&%$ #"! */ @@ -4723,33 +4760,7 @@ njs_string_encode_uri(njs_vm_t *vm, njs_ 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ }; - if (nargs < 2) { - njs_set_undefined(&vm->retval); - - return NJS_OK; - } - - value = njs_argument(args, 1); - - if (!njs_is_string(value)) { - ret = njs_value_to_string(vm, value, value); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - return njs_string_encode(vm, value, escape); -} - - -njs_int_t -njs_string_encode_uri_component(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) -{ - njs_int_t ret; - njs_value_t *value; - - static const uint32_t escape[] = { + static const uint32_t escape_uri_component[] = { 0xffffffff, /* 1111 1111 1111 1111 1111 1111 1111 1111 */ /* ?>=< ;:98 7654 3210 /.-, +*)( '&%$ #"! */ @@ -4768,91 +4779,172 @@ njs_string_encode_uri_component(njs_vm_t }; if (nargs < 2) { - njs_set_undefined(&vm->retval); - + vm->retval = njs_string_undefined; return NJS_OK; } value = njs_argument(args, 1); - - if (!njs_is_string(value)) { - ret = njs_value_to_string(vm, value, value); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - return njs_string_encode(vm, value, escape); -} - - -static njs_int_t -njs_string_encode(njs_vm_t *vm, njs_value_t *value, const uint32_t *escape) -{ - u_char byte, *src, *dst; - size_t n, size; - njs_str_t string; - static const u_char hex[16] = "0123456789ABCDEF"; + ret = njs_value_to_string(vm, value, value); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + escape = (component) ? escape_uri_component : escape_uri; njs_prefetch(escape); - njs_string_get(value, &string); - + (void) njs_string_prop(&string, value); + + size = 0; src = string.start; - n = 0; - - for (size = string.length; size != 0; size--) { - byte = *src++; - - if ((escape[byte >> 5] & ((uint32_t) 1 << (byte & 0x1f))) != 0) { - n += 2; + end = src + string.size; + + if (string.length == 0 || string.length == string.size) { + /* Byte or ASCII string. */ + + while (src < end) { + byte = *src++; + size += njs_need_escape(escape, byte) ? 3 : 1; } - } - - if (n == 0) { + + } else { + /* UTF-8 string. */ + + while (src < end) { + cp = njs_utf8_decode(&src, end); + + if (cp < 0x80 && !njs_need_escape(escape, cp)) { + size++; + continue; + } + + if (njs_slow_path(njs_surrogate_any(cp))) { + if (src == end) { + goto uri_error; + } + + if (njs_surrogate_leading(cp)) { + cp_low = njs_utf8_decode(&src, end); + + if (njs_slow_path(!njs_surrogate_trailing(cp_low))) { + goto uri_error; + } + + cp = njs_string_surrogate_pair(cp, cp_low); + size += njs_utf8_size(cp) * 3; + continue; + } + + goto uri_error; + } + + size += njs_utf8_size(cp) * 3; + } + } + + if (size == 0) { /* GC: retain src. */ vm->retval = *value; return NJS_OK; } - size = string.length + n; - dst = njs_string_alloc(vm, &vm->retval, size, size); if (njs_slow_path(dst == NULL)) { return NJS_ERROR; } - size = string.length; src = string.start; - do { - byte = *src++; - - if ((escape[byte >> 5] & ((uint32_t) 1 << (byte & 0x1f))) != 0) { - *dst++ = '%'; - *dst++ = hex[byte >> 4]; - *dst++ = hex[byte & 0xf]; - - } else { - *dst++ = byte; + if (string.length == 0 || string.length == string.size) { + /* Byte or ASCII string. */ + (void) njs_string_encode(escape, string.size, src, dst); + return NJS_OK; + } + + /* UTF-8 string. */ + + while (src < end) { + cp = njs_utf8_decode(&src, end); + + if (njs_slow_path(njs_surrogate_leading(cp))) { + cp_low = njs_utf8_decode(&src, end); + cp = njs_string_surrogate_pair(cp, cp_low); } - size--; - - } while (size != 0); + njs_utf8_encode(encode, cp); + + dst = njs_string_encode(escape, njs_utf8_size(cp), encode, dst); + } return NJS_OK; + +uri_error: + + njs_uri_error(vm, "malformed URI"); + + return NJS_ERROR; +} + + +njs_inline uint32_t +njs_string_decode_uri_cp(const int8_t *hex, const u_char **start, + const u_char *end, njs_bool_t expect_percent) +{ + int8_t d0, d1; + uint32_t cp; + const u_char *p; + + cp = njs_utf8_decode(start, end); + if (njs_fast_path(cp != '%')) { + return expect_percent ? 0xFFFFFFFF: cp; + } + + p = *start; + + if (njs_slow_path((p + 1) >= end)) { + return 0xFFFFFFFF; + } + + d0 = hex[*p++]; + if (njs_slow_path(d0 < 0)) { + return 0xFFFFFFFF; + } + + d1 = hex[*p++]; + if (njs_slow_path(d1 < 0)) { + return 0xFFFFFFFF; + } + + *start += 2; + return (d0 << 4) + d1; +} + + +njs_inline njs_bool_t +njs_reserved(const uint32_t *reserve, uint32_t byte) +{ + return ((reserve[byte >> 5] & ((uint32_t) 1 << (byte & 0x1f))) != 0); } njs_int_t njs_string_decode_uri(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, - njs_index_t unused) + njs_index_t component) { - njs_int_t ret; - njs_value_t *value; - - static const uint32_t reserve[] = { + u_char *dst; + int64_t size, length; + uint32_t cp; + njs_int_t ret; + njs_chb_t chain; + njs_uint_t i, n; + njs_bool_t percent; + njs_value_t *value; + const u_char *src, *p, *end; + const uint32_t *reserve; + njs_string_prop_t string; + u_char encode[4]; + + static const uint32_t reserve_uri[] = { 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ /* ?>=< ;:98 7654 3210 /.-, +*)( '&%$ #"! */ @@ -4870,33 +4962,7 @@ njs_string_decode_uri(njs_vm_t *vm, njs_ 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ }; - if (nargs < 2) { - njs_set_undefined(&vm->retval); - - return NJS_OK; - } - - value = njs_argument(args, 1); - - if (njs_slow_path(!njs_is_string(value))) { - ret = njs_value_to_string(vm, value, value); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - return njs_string_decode(vm, value, reserve); -} - - -njs_int_t -njs_string_decode_uri_component(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused) -{ - njs_int_t ret; - njs_value_t *value; - - static const uint32_t reserve[] = { + static const uint32_t reserve_uri_component[] = { 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ /* ?>=< ;:98 7654 3210 /.-, +*)( '&%$ #"! */ @@ -4914,35 +4980,6 @@ njs_string_decode_uri_component(njs_vm_t 0x00000000, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ }; - if (nargs < 2) { - njs_set_undefined(&vm->retval); - - return NJS_OK; - } - - value = njs_argument(args, 1); - - if (njs_slow_path(!njs_is_string(value))) { - ret = njs_value_to_string(vm, value, value); - if (njs_slow_path(ret != NJS_OK)) { - return ret; - } - } - - return njs_string_decode(vm, &args[1], reserve); -} - - -static njs_int_t -njs_string_decode(njs_vm_t *vm, njs_value_t *value, const uint32_t *reserve) -{ - int8_t d0, d1; - u_char byte, *start, *src, *dst; - size_t n; - ssize_t size, length; - njs_str_t string; - njs_bool_t utf8; - static const int8_t hex[256] njs_aligned(32) = { @@ -4964,104 +5001,126 @@ njs_string_decode(njs_vm_t *vm, njs_valu -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, }; - njs_prefetch(&hex['0']); + if (nargs < 2) { + vm->retval = njs_string_undefined; + return NJS_OK; + } + + value = njs_argument(args, 1); + ret = njs_value_to_string(vm, value, value); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + reserve = component ? reserve_uri_component : reserve_uri; + njs_prefetch(reserve); - - njs_string_get(value, &string); - + njs_prefetch(&hex['0']); + + (void) njs_string_prop(&string, value); + + length = 0; src = string.start; - n = 0; - - for (size = string.length; size != 0; size--) { - byte = *src++; - - if (byte == '%') { - size -= 2; - - if (size <= 0) { + end = string.start + string.size; + + njs_chb_init(&chain, vm->mem_pool); + + while (src < end) { + percent = (src[0] == '%'); + cp = njs_string_decode_uri_cp(hex, &src, end, 0); + if (njs_slow_path(cp == 0xFFFFFFFF)) { + goto uri_error; + } + + if (!percent) { + length += 1; + dst = njs_chb_reserve(&chain, 4); + if (dst != NULL) { + njs_utf8_encode(dst, cp); + njs_chb_written(&chain, njs_utf8_size(cp)); + } + + continue; + } + + if (cp < 0x80) { + if (njs_reserved(reserve, cp)) { + length += 3; + njs_chb_append(&chain, &src[-3], 3); + + } else { + length += 1; + dst = njs_chb_reserve(&chain, 1); + if (dst != NULL) { + *dst = cp; + njs_chb_written(&chain, 1); + } + } + + continue; + } + + n = 1; + + do { + n++; + } while (((cp << n) & 0x80)); + + if (njs_slow_path(n > 4)) { + goto uri_error; + } + + encode[0] = cp; + + for (i = 1; i < n; i++) { + cp = njs_string_decode_uri_cp(hex, &src, end, 1); + if (njs_slow_path(cp == 0xFFFFFFFF)) { goto uri_error; } - d0 = hex[*src++]; - if (d0 < 0) { - goto uri_error; - } - - d1 = hex[*src++]; - if (d1 < 0) { - goto uri_error; - } - - byte = (d0 << 4) + d1; - - if ((reserve[byte >> 5] & ((uint32_t) 1 << (byte & 0x1f))) == 0) { - n += 2; - } + encode[i] = cp; + } + + p = encode; + cp = njs_utf8_decode(&p, p + n); + if (njs_slow_path(cp == 0xFFFFFFFF)) { + goto uri_error; } - } - - if (n == 0) { + + dst = njs_chb_reserve(&chain, 4); + if (dst != NULL) { + njs_utf8_encode(dst, cp); + njs_chb_written(&chain, njs_utf8_size(cp)); + } + + length += 1; + } + + size = njs_chb_size(&chain); + if (njs_slow_path(size < 0)) { + njs_memory_error(vm); + return NJS_ERROR; + } + + if (size == 0) { /* GC: retain src. */ vm->retval = *value; return NJS_OK; } - n = string.length - n; - - start = njs_string_alloc(vm, &vm->retval, n, n); - if (njs_slow_path(start == NULL)) { + dst = njs_string_alloc(vm, &vm->retval, size, length); + if (njs_slow_path(dst == NULL)) { return NJS_ERROR; } - utf8 = 0; - dst = start; - size = string.length; - src = string.start; - - do { - byte = *src++; - - if (byte == '%') { - size -= 2; - - d0 = hex[*src++]; - d1 = hex[*src++]; - byte = (d0 << 4) + d1; - - utf8 |= (byte >= 0x80); - - if ((reserve[byte >> 5] & ((uint32_t) 1 << (byte & 0x1f))) != 0) { - *dst++ = '%'; - *dst++ = src[-2]; - byte = src[-1]; - } - } - - *dst++ = byte; - size--; - - } while (size != 0); - - if (utf8) { - length = njs_utf8_length(start, n); - - if (length < 0) { - length = 0; - } - - if (vm->retval.short_string.size != NJS_STRING_LONG) { - vm->retval.short_string.length = length; - - } else { - vm->retval.long_string.data->length = length; - } - } + njs_chb_join_to(&chain, dst); + njs_chb_destroy(&chain); return NJS_OK; uri_error: - njs_uri_error(vm, NULL); + njs_uri_error(vm, "malformed URI"); return NJS_ERROR; } diff -r 9b70f038abfa -r c60911765952 src/njs_string.h --- a/src/njs_string.h Tue Feb 18 18:42:37 2020 +0300 +++ b/src/njs_string.h Tue Feb 18 18:55:50 2020 +0300 @@ -26,6 +26,12 @@ /* The maximum signed int32_t. */ #define NJS_STRING_MAX_LENGTH 0x7fffffff +#define njs_surrogate_leading(cp) ((cp) >= 0xd800 && (cp) <= 0xdbff) + +#define njs_surrogate_trailing(cp) ((cp) >= 0xdc00 && (cp) <= 0xdfff) + +#define njs_surrogate_any(cp) ((cp) >= 0xd800 && (cp) <= 0xdfff) + /* Converting surrogate pair to code point. */ #define njs_string_surrogate_pair(high, low) \ (0x10000 + ((high - 0xd800) << 10) + (low - 0xdc00)) @@ -184,13 +190,9 @@ void njs_string_offset_map_init(const u_ double njs_string_to_index(const njs_value_t *value); const char *njs_string_to_c_string(njs_vm_t *vm, njs_value_t *value); njs_int_t njs_string_encode_uri(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused); -njs_int_t njs_string_encode_uri_component(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused); + njs_uint_t nargs, njs_index_t component); njs_int_t njs_string_decode_uri(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused); -njs_int_t njs_string_decode_uri_component(njs_vm_t *vm, njs_value_t *args, - njs_uint_t nargs, njs_index_t unused); + njs_uint_t nargs, njs_index_t component); njs_index_t njs_value_index(njs_vm_t *vm, const njs_value_t *src, njs_uint_t runtime); diff -r 9b70f038abfa -r c60911765952 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Tue Feb 18 18:42:37 2020 +0300 +++ b/src/test/njs_unit_test.c Tue Feb 18 18:55:50 2020 +0300 @@ -7979,12 +7979,20 @@ static njs_unit_test_t njs_test[] = { njs_str("encodeURI.length"), njs_str("1")}, - { njs_str("encodeURI()"), - njs_str("undefined")}, - { njs_str("encodeURI('012???')"), njs_str("012%D0%B0%D0%B1%D0%B2")}, + { njs_str("[" + " String.fromCharCode(0xD800)," + " String.fromCharCode(0xD800) + 'a'," + " String.fromCharCode(0xDC00)," + " String.fromCharCode(0xDC00) + 'a'," + "].every(v=>{try { encodeURI(v)} catch(e) {return e.name == 'URIError'}})"), + njs_str("true")}, + + { njs_str("encodeURI(String.fromCharCode(0xD800)+String.fromCharCode(0xDC00))"), + njs_str("%F0%90%80%80")}, + { njs_str("encodeURI('~}|{`_^]\\\\[@?>=<;:/.-,+*)(\\\'&%$#\"! ')"), njs_str("~%7D%7C%7B%60_%5E%5D%5C%5B@?%3E=%3C;:/.-,+*)('&%25$#%22!%20")}, @@ -8003,18 +8011,6 @@ static njs_unit_test_t njs_test[] = { njs_str("decodeURI.length"), njs_str("1")}, - { njs_str("decodeURI()"), - njs_str("undefined")}, - - { njs_str("decodeURI('%QQ')"), - njs_str("URIError")}, - - { njs_str("decodeURI('%')"), - njs_str("URIError")}, - - { njs_str("decodeURI('%0')"), - njs_str("URIError")}, - { njs_str("decodeURI('%00')"), njs_str("\0")}, @@ -8039,8 +8035,32 @@ static njs_unit_test_t njs_test[] = { njs_str("decodeURI('%D0%B0%D0%B1%D0%B2').length"), njs_str("3")}, - { njs_str("decodeURI('%80%81%82').length"), - njs_str("3")}, + { njs_str("[" + " '%'," + " '%0'," + " '%QQ'," + " '%C0%10'," + " '%DC%C7'," + " '%80%81%82'," + " '%EF%5C%A0'," + " '%EF%A0%5E'," + " '%E0%EF%A0'," + " '%E0%A0%EF'," + " '%FF%A2%95%BB'," + "].every(v=>{try { decodeURI(v)} catch(e) {return e.name == 'URIError'}})"), + njs_str("true")}, + + { njs_str("[" + " 'abc'," + " '???'," + " '????'," + " String.fromCodePoint(0x20000)," + "].every(v=>decodeURI(encodeURI(v)) === v)"), + njs_str("true")}, + + { njs_str("[encodeURI, encodeURIComponent, decodeURI, decodeURIComponent]" + ".every(v=>{var r = v(); return (typeof r === 'string') && r === 'undefined';})"), + njs_str("true")}, /* Functions. */ From mdounin at mdounin.ru Tue Feb 18 16:03:47 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 18 Feb 2020 19:03:47 +0300 Subject: [PATCH] Fix for the HT on request headers problem (#1752) In-Reply-To: References: <20200123192931.GI12894@mdounin.ru> Message-ID: <20200218160347.GU12894@mdounin.ru> Hello! On Fri, Feb 14, 2020 at 01:57:49PM +0200, Marin Stavrev wrote: > Is there any update about this one, or it is a closed case for the Nginx > team? In this particular case I tend to think that fixing the camera to use SP instead of HTAB is probably a better option. Alternatively, consider maintaining a local patch if you want to maintain interoperability with a particular IP camera. We might consider adding HTAB support if more interoperability problems will be reported. > > Best Regards > Marin > > On Fri, Jan 24, 2020 at 9:07 AM Marin Stavrev wrote: > > > Hello, > > > > I can understand your point, but still RFC 7230 defines OWS to allow HTAB > > and the therm used to suggest a single SP is SHOULD (recommendation) not > > MUST (mandatory). Thus, Nginx is not fully compliant. > > I would never post such a change if it wasn't really needed - I am not > > pushing for this change just out of love for RFC compliance.I have this > > issue causing problems with some Chinese IP cameras and NVRs that are > > generating such headers. I understand this is quite rare, and to be frank > > this is the only case I had personally seen such a (lousy) HTTP > > implementation. Unfortunately, I don't have any control over their FW and > > thus needed this fix on the server side. > > I know it does not matter much, but both Apache and Microsoft IIS handle > > such headers as expected and do not treat the request as a bad one as Nginx > > currently does. > > > > Best Regards > > M. Stavrev > > > > On Thu, Jan 23, 2020 at 9:29 PM Maxim Dounin wrote: > > > >> Hello! > >> > >> On Mon, Jan 20, 2020 at 05:29:25PM +0200, mstavrev at gmail.com wrote: > >> > >> > # HG changeset patch > >> > # User Marin Stavrev > >> > # Date 1579526641 -7200 > >> > # Mon Jan 20 15:24:01 2020 +0200 > >> > # Node ID bf238762fdaf03383c2f3c3718c401e6141e3935 > >> > # Parent 6439ef81e37dfccfc3a8c57fed278bf56014ef39 > >> > Fix for the HT on request headers problem (#1752) > >> > > >> > When client send HTTP request with a header of Content-Length that > >> starts with > >> > horizontal tab character (HT=0x09), Nginx responds with HTTP 400 Bad > >> Request. > >> > According to HTTP RFC2616 section 4.2, "... The field value MAY be > >> preceded by > >> > any amount of LWS, though a single SP is preferred.". The difinition of > >> LWS is: > >> > > >> > LWS = [CRLF] 1*( SP | HT ) > >> > > >> > So a header such as the following should be processed fine: > >> > > >> > Content-Length:<0x09>110\r\n > >> > >> Note that RFC 2616 you are quoting was obsoleted by RFC 7230. In > >> particular, line folding (the "[CRLF]" part of the grammar) is > >> obsolete and must not be generated. Modern syntax rules to refer > >> to would be RFC 7230, section 3.2: > >> > >> header-field = field-name ":" OWS field-value OWS > >> > >> Where OWS is defined in section 3.2.3 as: > >> > >> OWS = *( SP / HTAB ) > >> ; optional whitespace > >> > >> and text says that "a sender SHOULD generate the optional > >> whitespace as a single SP" where an optional whitespace can > >> improve readability. > >> > >> However, we haven't seen any interoperability problems due to no > >> HTAB support in nginx. As such, instead of adding HTAB support it > >> might be better to keep parsing strict. > >> > >> [...] > >> > >> -- > >> Maxim Dounin > >> http://mdounin.ru/ > >> _______________________________________________ > >> nginx-devel mailing list > >> nginx-devel at nginx.org > >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > >> > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Tue Feb 18 16:14:35 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 18 Feb 2020 19:14:35 +0300 Subject: Nginx tail module In-Reply-To: References: Message-ID: <20200218161435.GV12894@mdounin.ru> Hello! On Thu, Feb 13, 2020 at 05:18:15AM -0800, Maksim Yevmenkin wrote: > On Wed, Feb 12, 2020, 11:07 PM Eran Kornblau > wrote: > > > I don?t know what you?re trying to solve? but maybe you can just send a > > range request relative to the end, > > without any custom module. > > For example, ?Range: bytes=-1024? will return the last 1k of the resource. > > > > Thanks but this is not what I was hoping for. The idea is to have Nginx > return new data as soon as it is available in the tailed file. > > A client would issue one request and simply wait for the data to arrive in > chunked transfer encoding. The tricky part is to be notified on file modifications. It should be possible to do so with low overhead using EVFILT_VNODE on FreeBSD and inotify on Linux, but infrastructure in nginx for this isn't yet complete. Some EVFILT_VNODE code is there though, you may try using it. -- Maxim Dounin http://mdounin.ru/ From xeioex at nginx.com Tue Feb 18 17:39:24 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 18 Feb 2020 17:39:24 +0000 Subject: [njs] Added fs.symlink(), fs.unlink(), fs.realpath() and friends. Message-ID: details: https://hg.nginx.org/njs/rev/f5adc2ea2d53 branches: changeset: 1333:f5adc2ea2d53 user: Artem S. Povalyukhin date: Tue Feb 11 01:48:24 2020 +0300 description: Added fs.symlink(), fs.unlink(), fs.realpath() and friends. diffstat: src/njs_fs.c | 266 ++++++++++++++++++++++++++++++++++++++++ src/test/njs_interactive_test.c | 9 +- src/test/njs_unit_test.c | 9 + test/js/fs_promises_002.js | 2 +- test/js/fs_promises_003.js | 108 ++++++++++++++++ test/js/fs_promises_004.js | 200 ++++++++++++++++++++++++++++++ test/njs_expect_test.exp | 10 + 7 files changed, 602 insertions(+), 2 deletions(-) diffs (679 lines): diff -r c60911765952 -r f5adc2ea2d53 src/njs_fs.c --- a/src/njs_fs.c Tue Feb 18 18:55:50 2020 +0300 +++ b/src/njs_fs.c Tue Feb 11 01:48:24 2020 +0300 @@ -528,6 +528,201 @@ done: static njs_int_t +njs_fs_symlink(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t calltype) +{ + njs_int_t ret; + const char *target_path, *file_path; + njs_value_t retval, *target, *path, *callback, *type; + + target = njs_arg(args, nargs, 1); + ret = njs_fs_path_arg(vm, &target_path, target, &njs_str_value("target")); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + path = njs_arg(args, nargs, 2); + ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path")); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + callback = NULL; + type = njs_arg(args, nargs, 3); + + if (calltype == NJS_FS_CALLBACK) { + callback = njs_arg(args, nargs, njs_min(nargs - 1, 4)); + if (!njs_is_function(callback)) { + njs_type_error(vm, "\"callback\" must be a function"); + return NJS_ERROR; + } + + if (type == callback) { + type = njs_value_arg(&njs_value_undefined); + } + } + + if (njs_slow_path(!njs_is_undefined(type) && !njs_is_string(type))) { + njs_type_error(vm, "\"type\" must be a string"); + return NJS_ERROR; + } + + ret = symlink(target_path, file_path); + if (njs_slow_path(ret != 0)) { + ret = njs_fs_error(vm, "symlink", strerror(errno), path, errno, + &retval); + goto done; + } + + njs_set_undefined(&retval); + +done: + + if (ret == NJS_OK) { + return njs_fs_result(vm, &retval, calltype, callback, 1); + } + + return NJS_ERROR; +} + + +static njs_int_t +njs_fs_unlink(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t calltype) +{ + njs_int_t ret; + const char *file_path; + njs_value_t retval, *path, *callback; + + path = njs_arg(args, nargs, 1); + ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path")); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + callback = NULL; + + if (calltype == NJS_FS_CALLBACK) { + callback = njs_arg(args, nargs, 2); + if (!njs_is_function(callback)) { + njs_type_error(vm, "\"callback\" must be a function"); + return NJS_ERROR; + } + } + + ret = unlink(file_path); + if (njs_slow_path(ret != 0)) { + ret = njs_fs_error(vm, "unlink", strerror(errno), path, errno, &retval); + goto done; + } + + njs_set_undefined(&retval); + +done: + + if (ret == NJS_OK) { + return njs_fs_result(vm, &retval, calltype, callback, 1); + } + + return NJS_ERROR; +} + + +static njs_int_t +njs_fs_realpath(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, + njs_index_t calltype) +{ + u_char *resolved_path; + size_t size; + ssize_t length; + njs_int_t ret; + const char *file_path; + njs_value_t encoding, retval, *path, *callback, *options; + njs_fs_encoding_t enc; + char path_buf[MAXPATHLEN]; + + static const njs_value_t string_encoding = njs_string("encoding"); + + path = njs_arg(args, nargs, 1); + ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path")); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + callback = NULL; + options = njs_arg(args, nargs, 2); + + if (calltype == NJS_FS_CALLBACK) { + callback = njs_arg(args, nargs, njs_min(nargs - 1, 3)); + if (!njs_is_function(callback)) { + njs_type_error(vm, "\"callback\" must be a function"); + return NJS_ERROR; + } + + if (options == callback) { + options = njs_value_arg(&njs_value_undefined); + } + } + + njs_set_undefined(&encoding); + + switch (options->type) { + case NJS_STRING: + encoding = *options; + break; + + case NJS_UNDEFINED: + break; + + default: + if (!njs_is_object(options)) { + njs_type_error(vm, "Unknown options type: \"%s\" " + "(a string or object required)", + njs_type_string(options->type)); + return NJS_ERROR; + } + + ret = njs_value_property(vm, options, njs_value_arg(&string_encoding), + &encoding); + if (njs_slow_path(ret == NJS_ERROR)) { + return ret; + } + } + + enc = njs_fs_encoding(vm, &encoding); + if (njs_slow_path(enc == NJS_FS_ENC_INVALID)) { + return NJS_ERROR; + } + + resolved_path = (u_char *) realpath(file_path, path_buf); + if (njs_slow_path(resolved_path == NULL)) { + ret = njs_fs_error(vm, "realpath", strerror(errno), path, errno, + &retval); + goto done; + } + + size = njs_strlen(resolved_path); + length = njs_utf8_length(resolved_path, size); + if (njs_slow_path(length < 0)) { + length = 0; + } + + ret = njs_string_new(vm, &retval, resolved_path, size, length); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + +done: + + if (ret == NJS_OK) { + return njs_fs_result(vm, &retval, calltype, callback, 2); + } + + return NJS_ERROR; +} + + +static njs_int_t njs_fs_fd_read(njs_vm_t *vm, int fd, njs_str_t *data) { u_char *p, *end, *start; @@ -880,6 +1075,30 @@ static const njs_object_prop_t njs_fs_p .writable = 1, .configurable = 1, }, + + { + .type = NJS_PROPERTY, + .name = njs_string("symlink"), + .value = njs_native_function2(njs_fs_symlink, 0, NJS_FS_PROMISE), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("unlink"), + .value = njs_native_function2(njs_fs_unlink, 0, NJS_FS_PROMISE), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("realpath"), + .value = njs_native_function2(njs_fs_realpath, 0, NJS_FS_PROMISE), + .writable = 1, + .configurable = 1, + }, }; @@ -1039,6 +1258,53 @@ static const njs_object_prop_t njs_fs_o .configurable = 1, }, + { + .type = NJS_PROPERTY, + .name = njs_string("symlink"), + .value = njs_native_function2(njs_fs_symlink, 0, NJS_FS_CALLBACK), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("symlinkSync"), + .value = njs_native_function2(njs_fs_symlink, 0, NJS_FS_DIRECT), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("unlink"), + .value = njs_native_function2(njs_fs_unlink, 0, NJS_FS_CALLBACK), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("unlinkSync"), + .value = njs_native_function2(njs_fs_unlink, 0, NJS_FS_DIRECT), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("realpath"), + .value = njs_native_function2(njs_fs_realpath, 0, NJS_FS_CALLBACK), + .writable = 1, + .configurable = 1, + }, + + { + .type = NJS_PROPERTY, + .name = njs_string("realpathSync"), + .value = njs_native_function2(njs_fs_realpath, 0, NJS_FS_DIRECT), + .writable = 1, + .configurable = 1, + }, }; diff -r c60911765952 -r f5adc2ea2d53 src/test/njs_interactive_test.c --- a/src/test/njs_interactive_test.c Tue Feb 18 18:55:50 2020 +0300 +++ b/src/test/njs_interactive_test.c Tue Feb 11 01:48:24 2020 +0300 @@ -241,7 +241,14 @@ static njs_interactive_test_t njs_test[ " 'writeFile'," " 'writeFileSync'," " 'appendFile'," - " 'appendFileSync']" + " 'appendFileSync'," + " 'symlink'," + " 'symlinkSync'," + " 'unlink'," + " 'unlinkSync'," + " 'realpath'," + " 'realpathSync'," + "]" ".every(v=>{ try {fs[v]();} catch (e) { return e.stack.search(`fs.${v} `) >= 0}})" ENTER), njs_str("true") }, diff -r c60911765952 -r f5adc2ea2d53 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Tue Feb 18 18:55:50 2020 +0300 +++ b/src/test/njs_unit_test.c Tue Feb 11 01:48:24 2020 +0300 @@ -16159,6 +16159,12 @@ static njs_unit_test_t njs_test[] = "'writeFileSync'," "'appendFile'," "'appendFileSync'," + "'symlink'," + "'symlinkSync'," + "'unlink'," + "'unlinkSync'," + "'realpath'," + "'realpathSync'," "]," "test = (fname) =>" "[undefined, null, false, NaN, Symbol(), {}, Object('/njs_unknown_path')]" @@ -16181,6 +16187,9 @@ static njs_unit_test_t njs_test[] = "'readFile'," "'writeFile'," "'appendFile'," + "'symlink'," + "'unlink'," + "'realpath'," "];" "func.every((x) => typeof fs[x] == 'function')"), njs_str("true")}, diff -r c60911765952 -r f5adc2ea2d53 test/js/fs_promises_002.js --- a/test/js/fs_promises_002.js Tue Feb 18 18:55:50 2020 +0300 +++ b/test/js/fs_promises_002.js Tue Feb 11 01:48:24 2020 +0300 @@ -1,6 +1,6 @@ var fs = require('fs'); var fsp = fs.promises; -var fname = '/tmp/njs_fs_promises_002'; +var fname = './build/test/fs_promises_002'; var testSync = new Promise((resolve, reject) => { var failed = false; diff -r c60911765952 -r f5adc2ea2d53 test/js/fs_promises_003.js --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test/js/fs_promises_003.js Tue Feb 11 01:48:24 2020 +0300 @@ -0,0 +1,108 @@ +var fs = require('fs'); +var fsp = fs.promises; +var fname = './build/test/fs_promises_003'; + + +var testSync = () => new Promise((resolve, reject) => { + try { + try { + fs.unlinkSync(fname); + } catch (e) { + void e; + } + + try { + fs.unlinkSync(fname); + throw new Error('unlinkSync error 1'); + } catch (e) { + if (e.syscall != 'unlink') { + throw e; + } + } + + fs.writeFileSync(fname, fname); + fs.unlinkSync(fname); + try { + fs.accessSync(fname); + reject(new Error('unlinkSync error 2')); + return; + } catch (e) { + void e; + } + + resolve(); + } catch (e) { + reject(e); + } +}); + + +var testCallback = () => new Promise((resolve, reject) => { + fs.unlink(fname, () => { + fs.unlink(fname, (err) => { + if (!err) { + reject(new Error('fs.unlink error 1')); + return; + } + if (err.syscall != 'unlink') { + reject(err); + return; + } + + fs.writeFileSync(fname, fname); + fs.unlink(fname, (err) => { + if (err) { + reject(err); + return; + } + try { + fs.accessSync(fname); + reject(new Error('fs.unlink error 2')); + return; + } catch (e) { + void e; + } + resolve(); + }); + }); + }); +}); + + +Promise.resolve() +.then(testSync) +.then(() => { + console.log('test fs.unlinkSync'); +}) +.catch((e) => { + console.log('test fs.unlinkSync failed', e); +}) + +.then(testCallback) +.then(() => { + console.log('test fs.unlink'); +}) +.catch((e) => { + console.log('test fs.unlink failed', e); +}) + +.then(() => fsp.unlink(fname) + .catch(() => {})) +.then(() => fsp.unlink(fname)) + .then(() => { throw new Error('fsp.unlink error 1'); }) +.catch((e) => { if (e.syscall != 'unlink') { throw e; } }) + +.then(() => { + fs.writeFileSync(fname, fname); + return fsp.unlink(fname); +}) +.then(() => fsp.access(fname)) + .then(() => { throw new Error('fsp.unlink error 2'); }) +.catch((e) => { if (e.syscall != 'access') { throw e; } }) + +.then(() => { + console.log('test fsp.unlink'); +}) +.catch((e) => { + console.log('test fsp.unlink failed', e); +}); diff -r c60911765952 -r f5adc2ea2d53 test/js/fs_promises_004.js --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test/js/fs_promises_004.js Tue Feb 11 01:48:24 2020 +0300 @@ -0,0 +1,200 @@ +var fs = require('fs'); +var fsp = fs.promises; +var dname = './build/test/'; +var fname = dname + 'fs_promises_004'; +var fname_utf8 = dname + 'fs_promises_???_004'; +var lname = dname + 'fs_promises_004_lnk'; + + +var testSync = () => new Promise((resolve, reject) => { + try { + try { + fs.unlinkSync(fname); + } catch (e) { + void e; + } + try { + fs.unlinkSync(lname); + } catch (e) { + void e; + } + + try { + fs.realpathSync(fname); + throw new Error('fs.realpathSync error 1'); + } catch (e) { + if (e.syscall != 'realpath') { // e.code + throw e; + } + } + + fs.writeFileSync(fname, fname); + fs.writeFileSync(fname_utf8, fname_utf8); + + var rname = fs.realpathSync(fname); + + fs.symlinkSync(rname, lname); + + if (fs.realpathSync(lname) != rname) { + throw new Error('fs.symlinkSync error 2'); + } + + if (fs.readFileSync(lname) != fname) { + throw new Error('fs.symlinkSync error 3'); + } + + var rname_utf8 = fs.realpathSync(fname_utf8); + if (rname_utf8.slice(-7,-4) != '???') { + throw new Error('fs.realpathSync error 2'); + } + + fs.unlinkSync(lname); + fs.accessSync(fname); + fs.unlinkSync(fname); + fs.unlinkSync(fname_utf8); + + resolve(); + + } catch (e) { + reject(e); + } +}); + + +var testCallback = () => new Promise((resolve, reject) => { + try { + try { + fs.unlinkSync(fname); + } catch (e) { + void e; + } + try { + fs.unlinkSync(lname); + } catch (e) { + void e; + } + + fs.realpath(fname, (err) => { + if (!err) { + reject(new Error('fs.realpath error 1')); + return; + } + if (err.syscall != 'realpath') { + reject(err); + return; + } + + try { + fs.writeFileSync(fname, fname); + } catch (e) { + reject(e); + return; + } + + fs.realpath(fname, (err, rname) => { + if (err) { + reject(err); + return; + } + + fs.symlink(rname, lname, (err) => { + if (err) { + reject(err); + return; + } + + fs.realpath(lname, undefined, (err, xname) => { + if (err) { + reject(err); + return; + } + + if (rname != xname) { + reject(new Error('fs.symlink error 1')); + return; + } + + try { + if (fs.readFileSync(lname) != fname) { + reject(new Error('fs.symlink error 2')); + return; + } + + fs.unlinkSync(lname); + fs.accessSync(fname); + fs.unlinkSync(fname); + + } catch (e) { + reject(e); + return; + } + + resolve(); + }); + }); + }); + }); + + } catch (e) { + reject(e); + } +}); + + +Promise.resolve() +.then(testSync) +.then(() => { + console.log('test fs.symlinkSync'); +}) +.catch((e) => { + console.log('test fs.symlinkSync failed', e); +}) + +.then(testCallback) +.then(() => { + console.log('test fs.symlink'); +}) +.catch((e) => { + console.log('test fs.symlink failed', e); +}) + +.then(() => fsp.unlink(fname) + .catch(() => {})) +.then(() => fsp.unlink(lname) + .catch(() => {})) +.then(() => fsp.realpath(fname) + .then(() => { throw new Error('fsp.realpath error 1') })) +.catch((e) => { + if (e.syscall != 'realpath') { + throw e; + } +}) +.then(() => { + fs.writeFileSync(fname, fname); + + return fsp.realpath(fname); +}) +.then((rname) => fsp.symlink(rname, lname) + .then(() => rname)) +.then((rname) => fsp.realpath(lname) + .then((xname) => { + if (rname != xname) { + throw new Error('fsp.symlink error 2'); + } + })) +.then(() => { + if (fs.readFileSync(lname) != fname) { + throw new Error('fsp.symlink error 3'); + } + + fs.unlinkSync(lname); + fs.accessSync(fname); + fs.unlinkSync(fname); +}) + +.then(() => { + console.log('test fsp.symlink'); +}) +.catch((e) => { + console.log('test fsp.symlink failed', e); +}); diff -r c60911765952 -r f5adc2ea2d53 test/njs_expect_test.exp --- a/test/njs_expect_test.exp Tue Feb 18 18:55:50 2020 +0300 +++ b/test/njs_expect_test.exp Tue Feb 11 01:48:24 2020 +0300 @@ -1096,3 +1096,13 @@ njs_run {"./test/js/fs_promises_002.js"} "testSync ok true testCallback ok true testPromise ok true" + +njs_run {"./test/js/fs_promises_003.js"} \ +"test fs.unlinkSync +test fs.unlink +test fsp.unlink" + +njs_run {"./test/js/fs_promises_004.js"} \ +"test fs.symlinkSync +test fs.symlink +test fsp.symlink" From xeioex at nginx.com Wed Feb 19 14:23:28 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 19 Feb 2020 14:23:28 +0000 Subject: [njs] Fixed Number.prototype.toPrecision(). Message-ID: details: https://hg.nginx.org/njs/rev/0173143d7b15 branches: changeset: 1334:0173143d7b15 user: Dmitry Volyntsev date: Wed Feb 19 17:21:32 2020 +0300 description: Fixed Number.prototype.toPrecision(). This closes #290 issue on Github. diffstat: src/njs_dtoa.c | 2 +- src/test/njs_unit_test.c | 4 ++++ 2 files changed, 5 insertions(+), 1 deletions(-) diffs (26 lines): diff -r f5adc2ea2d53 -r 0173143d7b15 src/njs_dtoa.c --- a/src/njs_dtoa.c Tue Feb 11 01:48:24 2020 +0300 +++ b/src/njs_dtoa.c Wed Feb 19 17:21:32 2020 +0300 @@ -532,7 +532,7 @@ njs_dtoa_prec_format(char *start, size_t if (point < (int) prec) { start[point] = '.'; - njs_memset(&start[point + 1], '0', prec - len); + njs_memset(&start[point + 1], '0', prec - point); } } else if (point < (int) prec) { diff -r f5adc2ea2d53 -r 0173143d7b15 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Tue Feb 11 01:48:24 2020 +0300 +++ b/src/test/njs_unit_test.c Wed Feb 19 17:21:32 2020 +0300 @@ -609,6 +609,10 @@ static njs_unit_test_t njs_test[] = { njs_str("(-(2**10000)).toPrecision()"), njs_str("-Infinity") }, + { njs_str("var v = parseFloat('9'.repeat(98));" + "[98,100].map(p=>v.toPrecision(p).length)"), + njs_str("98,101") }, + { njs_str("(-0).toPrecision(2)"), njs_str("0.0") }, From mdounin at mdounin.ru Thu Feb 20 15:55:03 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 20 Feb 2020 15:55:03 +0000 Subject: [nginx] Disabled multiple Transfer-Encoding headers. Message-ID: details: https://hg.nginx.org/nginx/rev/aca005d232ff branches: changeset: 7625:aca005d232ff user: Maxim Dounin date: Thu Feb 20 16:19:29 2020 +0300 description: Disabled multiple Transfer-Encoding headers. We anyway do not support more than one transfer encoding, so accepting requests with multiple Transfer-Encoding headers doesn't make sense. Further, we do not handle multiple headers, and ignore anything but the first header. Reported by Filippo Valsorda. diffstat: src/http/ngx_http_request.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -131,7 +131,7 @@ ngx_http_header_t ngx_http_headers_in[] { ngx_string("Transfer-Encoding"), offsetof(ngx_http_headers_in_t, transfer_encoding), - ngx_http_process_header_line }, + ngx_http_process_unique_header_line }, { ngx_string("TE"), offsetof(ngx_http_headers_in_t, te), From mdounin at mdounin.ru Thu Feb 20 15:55:06 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 20 Feb 2020 15:55:06 +0000 Subject: [nginx] Removed "Transfer-Encoding: identity" support. Message-ID: details: https://hg.nginx.org/nginx/rev/fe5976aae0e3 branches: changeset: 7626:fe5976aae0e3 user: Maxim Dounin date: Thu Feb 20 16:19:34 2020 +0300 description: Removed "Transfer-Encoding: identity" support. The "identity" transfer coding has been removed in RFC 7230. It is believed that it is not used in real life, and at the same time it provides a potential attack vector. diffstat: src/http/ngx_http_request.c | 5 +---- 1 files changed, 1 insertions(+), 4 deletions(-) diffs (15 lines): diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -1952,10 +1952,7 @@ ngx_http_process_request_header(ngx_http r->headers_in.content_length_n = -1; r->headers_in.chunked = 1; - } else if (r->headers_in.transfer_encoding->value.len != 8 - || ngx_strncasecmp(r->headers_in.transfer_encoding->value.data, - (u_char *) "identity", 8) != 0) - { + } else { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent unknown \"Transfer-Encoding\": \"%V\"", &r->headers_in.transfer_encoding->value); From mdounin at mdounin.ru Thu Feb 20 15:55:09 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 20 Feb 2020 15:55:09 +0000 Subject: [nginx] Disabled duplicate "Host" headers (ticket #1724). Message-ID: details: https://hg.nginx.org/nginx/rev/4f18393a1d51 branches: changeset: 7627:4f18393a1d51 user: Maxim Dounin date: Thu Feb 20 16:51:07 2020 +0300 description: Disabled duplicate "Host" headers (ticket #1724). Duplicate "Host" headers were allowed in nginx 0.7.0 (revision b9de93d804ea) as a workaround for some broken Motorola phones which used to generate requests with two "Host" headers[1]. It is believed that this workaround is no longer relevant. [1] http://mailman.nginx.org/pipermail/nginx-ru/2008-May/017845.html diffstat: src/http/ngx_http_request.c | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-) diffs (24 lines): diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -1755,10 +1755,18 @@ ngx_http_process_host(ngx_http_request_t ngx_int_t rc; ngx_str_t host; - if (r->headers_in.host == NULL) { - r->headers_in.host = h; + if (r->headers_in.host) { + ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + "client sent duplicate host header: \"%V: %V\", " + "previous value: \"%V: %V\"", + &h->key, &h->value, &r->headers_in.host->key, + &r->headers_in.host->value); + ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); + return NGX_ERROR; } + r->headers_in.host = h; + host = h->value; rc = ngx_http_validate_host(&host, r->pool, 0); From yar at nginx.com Thu Feb 20 16:05:15 2020 From: yar at nginx.com (Yaroslav Zhuravlev) Date: Thu, 20 Feb 2020 16:05:15 +0000 Subject: limit_req_zone Documentation Wrong In-Reply-To: References: Message-ID: Hello Aidan, > On 24 Jan 2020, at 22:17, Aidan Carson wrote: > > Hello, > > I believe the documentation for the limit_req_zone directive on this page is wrong: > > http://nginx.org/en/docs/http/ngx_http_limit_req_module.html > > It says that a rate parameter is not optional, but it is. The directive requires at least three parameters, but > > limit_req_zone $binary_remote_addr zone=limit:64k sync; > > or > > limit_req_zone $binary_remote_addr zone=limit:64k zone=limit:64k; > > are valid, omitting the rate. I see in the code that the default is 1r/s. Perhaps updating the documentation to list the default would be good, or changing the code to have the rate be required. > > Thank you, > > Aidan Carson Thank you for your feedback on the docs. The ?rate? parameter is assumed to be obligatory, though the syntax (http://nginx.org/r/limit_req_zone) may be constructed in a way to make it optional. For a common use case, the current behaviour is considered correct here, so the documentation would also be correct. I wouldn?t expect much changes here but let?s leave the latter to developers. Best regards, yar [...] From matwey.kornilov at gmail.com Mon Feb 24 11:22:11 2020 From: matwey.kornilov at gmail.com (Matwey V. Kornilov) Date: Mon, 24 Feb 2020 14:22:11 +0300 Subject: [PATCH 1 of 2] SSI: implemented "fsize" SSI command In-Reply-To: References: Message-ID: <6333f7316e47928c5737.1582543331@oak.local> # HG changeset patch # User Matwey V. Kornilov # Date 1582478281 -10800 # Sun Feb 23 20:18:01 2020 +0300 # Branch fsize_v2 # Node ID 6333f7316e47928c57379c72d24595df46f77445 # Parent 4f18393a1d51bce6103ea2f1b2587900f349ba3d SSI: implemented "fsize" SSI command. diff -r 4f18393a1d51 -r 6333f7316e47 src/http/modules/ngx_http_ssi_filter_module.c --- a/src/http/modules/ngx_http_ssi_filter_module.c Thu Feb 20 16:51:07 2020 +0300 +++ b/src/http/modules/ngx_http_ssi_filter_module.c Sun Feb 23 20:18:01 2020 +0300 @@ -70,6 +70,11 @@ } ngx_http_ssi_state_e; +static ngx_http_ssi_ctx_conf_t *ngx_http_ssi_ctx_conf_ro_snapshot( + ngx_http_ssi_ctx_t *ctx); +static ngx_http_ssi_ctx_conf_t *ngx_http_ssi_ctx_conf_rw_snapshot( + ngx_pool_t *pool, ngx_http_ssi_ctx_t *ctx); + static ngx_int_t ngx_http_ssi_output(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx); static void ngx_http_ssi_buffered(ngx_http_request_t *r, @@ -89,6 +94,10 @@ ngx_int_t rc); static ngx_int_t ngx_http_ssi_set_variable(ngx_http_request_t *r, void *data, ngx_int_t rc); +static ngx_int_t ngx_http_ssi_fsize(ngx_http_request_t *r, + ngx_http_ssi_ctx_t *ctx, ngx_str_t **params); +static ngx_int_t ngx_http_ssi_fsize_output(ngx_http_request_t *r, void *data, + ngx_int_t rc); static ngx_int_t ngx_http_ssi_echo(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, ngx_str_t **params); static ngx_int_t ngx_http_ssi_config(ngx_http_request_t *r, @@ -223,12 +232,16 @@ #define NGX_HTTP_SSI_INCLUDE_SET 3 #define NGX_HTTP_SSI_INCLUDE_STUB 4 +#define NGX_HTTP_SSI_FSIZE_VIRTUAL 0 +#define NGX_HTTP_SSI_FSIZE_FILE 1 + #define NGX_HTTP_SSI_ECHO_VAR 0 #define NGX_HTTP_SSI_ECHO_DEFAULT 1 #define NGX_HTTP_SSI_ECHO_ENCODING 2 #define NGX_HTTP_SSI_CONFIG_ERRMSG 0 #define NGX_HTTP_SSI_CONFIG_TIMEFMT 1 +#define NGX_HTTP_SSI_CONFIG_SIZEFMT 2 #define NGX_HTTP_SSI_SET_VAR 0 #define NGX_HTTP_SSI_SET_VALUE 1 @@ -248,6 +261,13 @@ }; +static ngx_http_ssi_param_t ngx_http_ssi_fsize_params[] = { + { ngx_string("virtual"), NGX_HTTP_SSI_FSIZE_VIRTUAL, 0, 0 }, + { ngx_string("file"), NGX_HTTP_SSI_FSIZE_FILE, 0, 0 }, + { ngx_null_string, 0, 0, 0 } +}; + + static ngx_http_ssi_param_t ngx_http_ssi_echo_params[] = { { ngx_string("var"), NGX_HTTP_SSI_ECHO_VAR, 1, 0 }, { ngx_string("default"), NGX_HTTP_SSI_ECHO_DEFAULT, 0, 0 }, @@ -259,6 +279,7 @@ static ngx_http_ssi_param_t ngx_http_ssi_config_params[] = { { ngx_string("errmsg"), NGX_HTTP_SSI_CONFIG_ERRMSG, 0, 0 }, { ngx_string("timefmt"), NGX_HTTP_SSI_CONFIG_TIMEFMT, 0, 0 }, + { ngx_string("sizefmt"), NGX_HTTP_SSI_CONFIG_SIZEFMT, 0, 0 }, { ngx_null_string, 0, 0, 0 } }; @@ -290,6 +311,8 @@ static ngx_http_ssi_command_t ngx_http_ssi_commands[] = { { ngx_string("include"), ngx_http_ssi_include, ngx_http_ssi_include_params, 0, 0, 1 }, + { ngx_string("fsize"), ngx_http_ssi_fsize, + ngx_http_ssi_fsize_params, 0, 0, 1 }, { ngx_string("echo"), ngx_http_ssi_echo, ngx_http_ssi_echo_params, 0, 0, 0 }, { ngx_string("config"), ngx_http_ssi_config, @@ -325,6 +348,36 @@ }; +static ngx_http_ssi_ctx_conf_t * +ngx_http_ssi_ctx_conf_ro_snapshot(ngx_http_ssi_ctx_t *ctx) +{ + ctx->conf->in_use = 1; + + return ctx->conf; +} + + +static ngx_http_ssi_ctx_conf_t * +ngx_http_ssi_ctx_conf_rw_snapshot(ngx_pool_t *pool, ngx_http_ssi_ctx_t *ctx) +{ + ngx_http_ssi_ctx_conf_t *conf; + + if (ctx->conf->in_use) + { + conf = ngx_pnalloc(pool, sizeof(ngx_http_ssi_ctx_conf_t)); + if (conf == NULL) { + return NULL; + } + + ngx_memcpy(conf, ctx->conf, sizeof(ngx_http_ssi_ctx_conf_t)); + conf->in_use = 0; + + ctx->conf = conf; + } + + return ctx->conf; +} + static ngx_int_t ngx_http_ssi_header_filter(ngx_http_request_t *r) @@ -360,9 +413,12 @@ ctx->params.nalloc = NGX_HTTP_SSI_PARAMS_N; ctx->params.pool = r->pool; - ctx->timefmt = ngx_http_ssi_timefmt; - ngx_str_set(&ctx->errmsg, + ctx->conf = &ctx->conf_snapshot; + + ctx->conf->timefmt = ngx_http_ssi_timefmt; + ngx_str_set(&ctx->conf->errmsg, "[an error occurred while processing the directive]"); + ctx->conf->in_use = 0; r->filter_need_in_memory = 1; @@ -851,8 +907,8 @@ } b->memory = 1; - b->pos = ctx->errmsg.data; - b->last = ctx->errmsg.data + ctx->errmsg.len; + b->pos = ctx->conf->errmsg.data; + b->last = ctx->conf->errmsg.data + ctx->conf->errmsg.len; cl->next = NULL; *ctx->last_out = cl; @@ -2243,6 +2299,155 @@ static ngx_int_t +ngx_http_ssi_fsize(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, + ngx_str_t **params) +{ + ngx_int_t rc; + ngx_str_t *uri, *file, args; + ngx_uint_t flags; + ngx_http_request_t *sr; + ngx_http_post_subrequest_t *psr; + + uri = params[NGX_HTTP_SSI_FSIZE_VIRTUAL]; + file = params[NGX_HTTP_SSI_FSIZE_FILE]; + + if (uri && file) { + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "fsize may be either virtual=\"%V\" or file=\"%V\"", + uri, file); + return NGX_HTTP_SSI_ERROR; + } + + if (uri == NULL && file == NULL) { + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "no parameter in \"fsize\" SSI command"); + return NGX_HTTP_SSI_ERROR; + } + + if (uri == NULL) { + uri = file; + } + + rc = ngx_http_ssi_evaluate_string(r, ctx, uri, NGX_HTTP_SSI_ADD_PREFIX); + + if (rc != NGX_OK) { + return rc; + } + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "ssi fsize: \"%V\"", uri); + + ngx_str_null(&args); + flags = NGX_HTTP_LOG_UNSAFE; + + if (ngx_http_parse_unsafe_uri(r, uri, &args, &flags) != NGX_OK) { + return NGX_HTTP_SSI_ERROR; + } + + psr = ngx_palloc(r->pool, sizeof(ngx_http_post_subrequest_t)); + if (psr == NULL) { + return NGX_ERROR; + } + + psr->handler = ngx_http_ssi_fsize_output; + psr->data = ngx_http_ssi_ctx_conf_ro_snapshot(ctx); + + if (ngx_http_subrequest(r, uri, &args, &sr, psr, flags) != NGX_OK) { + return NGX_HTTP_SSI_ERROR; + } + + sr->header_only = 1; + + return NGX_OK; +} + + +static ngx_int_t +ngx_http_ssi_fsize_output(ngx_http_request_t *r, void *data, ngx_int_t rc) +{ + off_t length; + u_char scale; + ngx_buf_t *b; + ngx_int_t size; + ngx_uint_t exact_size; + ngx_chain_t *out; + ngx_http_ssi_ctx_conf_t *conf; + + conf = data; + exact_size = conf->exact_size; + + if (rc == NGX_ERROR || r->connection->error || r->request_output) { + return rc; + } + + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "ssi fsize output: \"%V?%V\"", &r->uri, &r->args); + + b = ngx_create_temp_buf(r->pool, NGX_OFF_T_LEN + 2); + if (b == NULL) { + return NGX_ERROR; + } + + if (r->headers_out.status != NGX_HTTP_OK + || r->headers_out.content_length_n < 0) + { + b->pos = conf->errmsg.data; + b->last = conf->errmsg.data + conf->errmsg.len; + + } else if (exact_size) { + b->last = ngx_sprintf(b->last, "%O", r->headers_out.content_length_n); + + } else { + length = r->headers_out.content_length_n; + + if (length > 1024 * 1024 * 1024 - 1) { + size = (ngx_int_t) (length / (1024 * 1024 * 1024)); + if ((length % (1024 * 1024 * 1024)) > (1024 * 1024 * 1024 / 2 - 1)) + { + size++; + } + scale = 'G'; + + } else if (length > 1024 * 1024 - 1) { + size = (ngx_int_t) (length / (1024 * 1024)); + if ((length % (1024 * 1024)) > (1024 * 1024 / 2 - 1)) { + size++; + } + scale = 'M'; + + } else if (length > 9999) { + size = (ngx_int_t) (length / 1024); + if (length % 1024 > 511) { + size++; + } + scale = 'K'; + + } else { + size = (ngx_int_t) length; + scale = '\0'; + } + + if (scale) { + b->last = ngx_sprintf(b->last, "%i%c", size, scale); + + } else { + b->last = ngx_sprintf(b->last, "%i", size); + } + } + + out = ngx_alloc_chain_link(r->pool); + if (out == NULL) { + return NGX_ERROR; + } + + out->buf = b; + out->next = NULL; + + return ngx_http_output_filter(r, out); +} + + +static ngx_int_t ngx_http_ssi_echo(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, ngx_str_t **params) { @@ -2382,24 +2587,51 @@ ngx_http_ssi_config(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, ngx_str_t **params) { - ngx_str_t *value; + ngx_str_t *value; + ngx_http_ssi_ctx_conf_t *conf; + + conf = ngx_http_ssi_ctx_conf_rw_snapshot(r->pool, ctx); + if (conf == NULL) { + return NGX_ERROR; + } value = params[NGX_HTTP_SSI_CONFIG_TIMEFMT]; if (value) { - ctx->timefmt.len = value->len; - ctx->timefmt.data = ngx_pnalloc(r->pool, value->len + 1); - if (ctx->timefmt.data == NULL) { + conf->timefmt.len = value->len; + conf->timefmt.data = ngx_pnalloc(r->pool, value->len + 1); + if (conf->timefmt.data == NULL) { return NGX_ERROR; } - ngx_cpystrn(ctx->timefmt.data, value->data, value->len + 1); + ngx_cpystrn(conf->timefmt.data, value->data, value->len + 1); } value = params[NGX_HTTP_SSI_CONFIG_ERRMSG]; if (value) { - ctx->errmsg = *value; + conf->errmsg = *value; + } + + value = params[NGX_HTTP_SSI_CONFIG_SIZEFMT]; + + if (value) { + if (value->len == 5 + && ngx_strncasecmp(value->data, (u_char *) "bytes", 5) == 0) + { + conf->exact_size = 1; + + } else if (value->len == 6 + && ngx_strncasecmp(value->data, (u_char *) "abbrev", 6) == 0) + { + conf->exact_size = 0; + + } else { + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "unknown size format \"%V\" " + "in \"config\" SSI command", value); + return NGX_HTTP_SSI_ERROR; + } } return NGX_OK; @@ -2739,7 +2971,7 @@ ctx = ngx_http_get_module_ctx(r, ngx_http_ssi_filter_module); - timefmt = ctx ? &ctx->timefmt : &ngx_http_ssi_timefmt; + timefmt = ctx ? &ctx->conf->timefmt : &ngx_http_ssi_timefmt; if (timefmt->len == sizeof("%s") - 1 && timefmt->data[0] == '%' && timefmt->data[1] == 's') diff -r 4f18393a1d51 -r 6333f7316e47 src/http/modules/ngx_http_ssi_filter_module.h --- a/src/http/modules/ngx_http_ssi_filter_module.h Thu Feb 20 16:51:07 2020 +0300 +++ b/src/http/modules/ngx_http_ssi_filter_module.h Sun Feb 23 20:18:01 2020 +0300 @@ -37,6 +37,14 @@ typedef struct { + ngx_str_t timefmt; + ngx_str_t errmsg; + unsigned in_use:1; + unsigned exact_size:1; +} ngx_http_ssi_ctx_conf_t; + + +typedef struct { ngx_buf_t *buf; u_char *pos; @@ -79,8 +87,8 @@ ngx_http_request_t *wait; void *value_buf; - ngx_str_t timefmt; - ngx_str_t errmsg; + ngx_http_ssi_ctx_conf_t *conf; + ngx_http_ssi_ctx_conf_t conf_snapshot; } ngx_http_ssi_ctx_t; From matwey.kornilov at gmail.com Mon Feb 24 11:22:12 2020 From: matwey.kornilov at gmail.com (Matwey V. Kornilov) Date: Mon, 24 Feb 2020 14:22:12 +0300 Subject: [PATCH 2 of 2] SSI: implemented "flastmod" SSI command In-Reply-To: References: Message-ID: # HG changeset patch # User Matwey V. Kornilov # Date 1582541268 -10800 # Mon Feb 24 13:47:48 2020 +0300 # Branch fsize_v2 # Node ID f0353fa599b8e20dcb9dd44917cfd78f4edcb82c # Parent 6333f7316e47928c57379c72d24595df46f77445 SSI: implemented "flastmod" SSI command. diff -r 6333f7316e47 -r f0353fa599b8 src/http/modules/ngx_http_ssi_filter_module.c --- a/src/http/modules/ngx_http_ssi_filter_module.c Sun Feb 23 20:18:01 2020 +0300 +++ b/src/http/modules/ngx_http_ssi_filter_module.c Mon Feb 24 13:47:48 2020 +0300 @@ -98,6 +98,8 @@ ngx_http_ssi_ctx_t *ctx, ngx_str_t **params); static ngx_int_t ngx_http_ssi_fsize_output(ngx_http_request_t *r, void *data, ngx_int_t rc); +static ngx_int_t ngx_http_ssi_flastmod_output(ngx_http_request_t *r, + void *data, ngx_int_t rc); static ngx_int_t ngx_http_ssi_echo(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, ngx_str_t **params); static ngx_int_t ngx_http_ssi_config(ngx_http_request_t *r, @@ -313,6 +315,8 @@ ngx_http_ssi_include_params, 0, 0, 1 }, { ngx_string("fsize"), ngx_http_ssi_fsize, ngx_http_ssi_fsize_params, 0, 0, 1 }, + { ngx_string("flastmod"), ngx_http_ssi_fsize, + ngx_http_ssi_fsize_params, 0, 0, 1 }, { ngx_string("echo"), ngx_http_ssi_echo, ngx_http_ssi_echo_params, 0, 0, 0 }, { ngx_string("config"), ngx_http_ssi_config, @@ -2313,14 +2317,14 @@ if (uri && file) { ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, - "fsize may be either virtual=\"%V\" or file=\"%V\"", - uri, file); + "%V may be either virtual=\"%V\" or file=\"%V\"", + ctx->command, uri, file); return NGX_HTTP_SSI_ERROR; } if (uri == NULL && file == NULL) { ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, - "no parameter in \"fsize\" SSI command"); + "no parameter in \"%V\" SSI command", ctx->command); return NGX_HTTP_SSI_ERROR; } @@ -2352,6 +2356,13 @@ psr->handler = ngx_http_ssi_fsize_output; psr->data = ngx_http_ssi_ctx_conf_ro_snapshot(ctx); + if (ctx->command.len == sizeof("fsize") - 1) { + psr->handler = ngx_http_ssi_fsize_output; + + } else if (ctx->command.len == sizeof("flastmod") - 1) { + psr->handler = ngx_http_ssi_flastmod_output; + } + if (ngx_http_subrequest(r, uri, &args, &sr, psr, flags) != NGX_OK) { return NGX_HTTP_SSI_ERROR; } @@ -2448,6 +2459,77 @@ static ngx_int_t +ngx_http_ssi_flastmod_output(ngx_http_request_t *r, void *data, ngx_int_t rc) +{ + size_t len; + ngx_buf_t *b; + ngx_str_t *timefmt; + struct tm tm; + ngx_chain_t *out; + ngx_http_ssi_ctx_conf_t *conf; + + conf = data; + timefmt = &conf->timefmt; + + if (rc == NGX_ERROR || r->connection->error || r->request_output) { + return rc; + } + + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "ssi flastmod output: \"%V?%V\"", &r->uri, &r->args); + + if (r->headers_out.status != NGX_HTTP_OK || r->headers_out.last_modified_time == -1) { + b = ngx_calloc_buf(r->pool); + if (b == NULL) { + return NGX_ERROR; + } + + b->memory = 1; + b->pos = conf->errmsg.data; + b->last = conf->errmsg.data + conf->errmsg.len; + + } else if (timefmt->len == sizeof("%s") - 1 + && timefmt->data[0] == '%' && timefmt->data[1] == 's') + { + b = ngx_create_temp_buf(r->pool, NGX_TIME_T_LEN); + if (b == NULL) { + return NGX_ERROR; + } + + b->last = ngx_sprintf(b->last, "%T", r->headers_out.last_modified_time); + + } else { + b = ngx_create_temp_buf(r->pool, NGX_HTTP_SSI_DATE_LEN); + if (b == NULL) { + return NGX_ERROR; + } + + ngx_libc_localtime(r->headers_out.last_modified_time, &tm); + + len = strftime((char *)b->last, NGX_HTTP_SSI_DATE_LEN, + (char *) timefmt->data, &tm); + if (len == 0) { + b->pos = conf->errmsg.data; + b->last = conf->errmsg.data + conf->errmsg.len; + + } else { + b->last = b->last + len; + } + } + + out = ngx_alloc_chain_link(r->pool); + if (out == NULL) { + return NGX_ERROR; + } + + out->buf = b; + out->next = NULL; + + return ngx_http_output_filter(r, out); +} + + +static ngx_int_t ngx_http_ssi_echo(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, ngx_str_t **params) { From matwey.kornilov at gmail.com Mon Feb 24 11:22:10 2020 From: matwey.kornilov at gmail.com (Matwey V. Kornilov) Date: Mon, 24 Feb 2020 14:22:10 +0300 Subject: [PATCH 0 of 2] Implement #fsize and #flastmod SSI commands Message-ID: Hello, This series implements SSI fsize ans flastmod commands. Updates from the previous version http://mailman.nginx.org/pipermail/nginx-devel/2017-May/009914.html are the following: 1) fixed misc style issues using Maxim's proposed patch 2) fixed issue when multiple #config commands are happens on the page From yshpakov at hotmail.com Wed Feb 26 00:59:22 2020 From: yshpakov at hotmail.com (Yury Shpakov) Date: Wed, 26 Feb 2020 00:59:22 +0000 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> , <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> , , Message-ID: Hi Sergey, You mentioned that you can set up some delays in responses. How can I do it? Adding this module during compilation? https://github.com/openresty/echo-nginx-module I tried but it didn't want to compile. I got many compilation errors. Maybe I can set up delays somehow else? Thank you, Yury ________________________________ From: nginx-devel on behalf of Yury Shpakov Sent: Friday, February 14, 2020 6:08 PM To: Sergey Brester Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 So what is the meaning of Auth-Server and Auth-Port headers? So it's relevant only when nginx works as SMTP Proxy (not SMTP Server)? And these are host/port where to redirect SMTP requests? Yeah, I was all the time surprised -- how come, it's set as Proxy but there is no setting where it redirects SMTP communication to. A little bit unexpected place for those setting. Well, let me try... I ran Fake SMTP Server on port 25.(I found on Internet some fake SMTP Server). I configured my test SMTP client to localhost:25 (later to 127.0.0.1:25). They send/receive successfully. So both SMTP Client and (fake) SMTP Server work fine. 127.0.0.1 works fine too. I re-configured my test SMTP client to localhost:8025 (tried 127.0.0.1:8025 too). As well, I changed this section of config as follows: http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } The same error: 2020/02/14 17:37:18 [error] 15260#3328: *5 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Update: Detailed logging with debug information helped a lot. This is what I noticed in there: 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp auth state 2020/02/14 17:40:28 [debug] 3940#22096: *1 WSARecv: fd:584 rc:0 24 of 4096 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp rcpt to:"RCPT TO:" 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer del: 584: 1172123084 2020/02/14 17:40:28 [debug] 3940#22096: *1 malloc: 02F8C260:2048 2020/02/14 17:40:28 [debug] 3940#22096: *1 stream socket 588 2020/02/14 17:40:28 [debug] 3940#22096: *1 connect to [::1]:9000, fd:588 #2 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:768 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:16 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 So it's trying to use IP6 rather than IP4. And below: 2020/02/14 17:40:29 [debug] 3940#22096: *1 delete posted event 03171170 2020/02/14 17:40:29 [debug] 3940#22096: *1 mail auth http write handler 2020/02/14 17:40:29 [debug] 3940#22096: *1 WSASend: fd:588, -1, 0 of 306 2020/02/14 17:40:29 [error] 3940#22096: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 So, I replaced localhost with 127.0.0.1 like this: auth_http 127.0.0.1:9000/cgi-bin/nginxauth.cgi; And it worked. Since I forced it to use IP4. Any idea how to use host name instead of IP address and still have it working? Update 2: I figured it out. Googled a little bit and ended up with the following change to my config: http { server { listen 9000; listen [::]:9000 ipv6only=on; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } Now it works. But why just "listen 9000" doesn't listen on both IP4 and IP6? Is it a bug? Yury ________________________________ From: Sergey Brester Sent: Friday, February 14, 2020 5:59 AM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I don't know what is wrong with your config... I guess your smtp server does not answer properly. Is 127.0.0.2:143 really your SMTP-server? Because port 143 is mostly an IMAP port - but you've specified protocol smtp in the server section. Anyway I tested your config with my settings (replaced name and smtp-server, here 192.0.2.222:25) and enabled debug: + error_log logs/error-mail.log debug; mail { - server_name localhost; + server_name example.com; ... http { ... - add_header Auth-Server 127.0.0.2; - add_header Auth-Port 143; + add_header Auth-Server 192.0.2.222; + add_header Auth-Port 25; it works well - I see the test incoming mail (I send to myself via 8025 port) and following output in the log (a lot of irrelevant messages are removed): >>>>>>>>> 2020/02/14 11:24:04 [debug] 121280#128244: *1 smtp mail from:"mail FROM:" ... 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request line: "GET /cgi-bin/nginxauth.cgi HTTP/1.0" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http uri: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http args: "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http exten: "cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http process request header line 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Host: localhost" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Method: none" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-User: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Pass: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Protocol: smtp" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Login-Attempt: 1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-IP: 127.0.0.1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-Host: [UNAVAILABLE]" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-Helo: myhost.example.com" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-From: mail FROM:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-To: rcpt TO:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header done 2020/02/14 11:24:04 [debug] 121280#128244: *3 event timer del: 512: 1127939767 2020/02/14 11:24:04 [debug] 121280#128244: *3 generic phase: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 1 2020/02/14 11:24:04 [debug] 121280#128244: *3 search through nested static locations of "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 test location: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 using configuration "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http cl:-1 max:1048576 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 3 2020/02/14 11:24:04 [debug] 121280#128244: *3 http set discard body 2020/02/14 11:24:04 [debug] 121280#128244: *3 HTTP/1.1 204 No Content Server: nginx/1.17.4 Date: Fri, 14 Feb 2020 10:24:04 GMT Connection: close Auth-Status: OK Auth-Server: 192.0.2.222 Auth-Port: 25 2020/02/14 11:24:04 [debug] 121280#128244: *3 write new buf t:1 f:0 008AD6A0, pos 008AD6A0, size: 164 file: 0, size: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter: l:1 f:0 s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter limit 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 WSASend: fd:512, s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter 00000000 2020/02/14 11:24:04 [debug] 121280#128244: *3 http finalize request: 0, "/cgi-bin/nginxauth.cgi?" a:1, c:1 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request count:1 blk:0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http close request 2020/02/14 11:24:04 [debug] 121280#128244: *3 http log handler 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008ACC50, unused: 1161 2020/02/14 11:24:04 [debug] 121280#128244: *3 close http connection: 512 2020/02/14 11:24:04 [debug] 121280#128244: *3 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008AC848 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 0039FDE0, unused: 28 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http read handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 164 of 1024 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process status line 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process headers 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Server: nginx/1.17.4" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Date: Fri, 14 Feb 2020 10:24:04 GMT" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Connection: close" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Status: OK" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Server: 192.0.2.222" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Port: 25" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header done 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer del: 496: 1127939764 2020/02/14 11:24:04 [debug] 121280#128244: *1 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *1 free: 008AC040, unused: 196 2020/02/14 11:24:04 [debug] 121280#128244: *1 stream socket 496 2020/02/14 11:24:04 [debug] 121280#128244: *1 connect to 192.0.2.222:25, fd:496 #4 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer add: 496: 60000:1127939769 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 008AC040:4096 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 post event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 delete posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy dummy handler 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 22 of 4096 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy send ehlo 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 0039FDE0:256 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSASend: fd:496, 0, 25 of 25 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 196 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send mail from 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 47 of 47 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 60 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send rcpt to 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 45 of 45 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 63 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer add: 492: 86400000:1214280441 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer del: 496: 1127939769 2020/02/14 11:24:05 [info] 121280#128244: *1 client logged in, client: 127.0.0.1, server: 0.0.0.0:8025 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 1, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 63 of 63 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 6 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 6 of 6 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 50 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 50 of 50 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 170 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 170 of 170 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280535 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 56 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 56 of 56 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle ... <<<<<<<<< Regards, Sergey 13.02.2020 22:45, Yury Shpakov wrote: Hi Sergey, I reconfigured the config file as follows: === === === #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.2; # backend ip add_header Auth-Port 143; # backend port return 204; } } } === === === And now it's responding on port 9000 as expected: === === === C:\WINDOWS\system32>curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi HTTP/1.1 204 No Content Server: nginx/1.17.9 Date: Thu, 13 Feb 2020 21:30:54 GMT Connection: keep-alive Auth-Status: OK Auth-Server: 127.0.0.2 Auth-Port: 143 === === === However I'm still experiencing the same issue (in log file): === === === 2020/02/13 16:29:24 [notice] 35048#26192: signal process started 2020/02/13 16:29:34 [error] 31732#22720: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 === === === Tried under both admin and regular user. Any further ideas how to get it fixed please? Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 1:51 PM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I answered inline... 12.02.2020 18:59, Yury Shpakov wrote: Hi Sergey, Thank you for you response. I tried netstat /nabo and I don't see any reference to port 9000 at all. So a problem is to make nginx to listen on port 9000 (as server)? Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ for more examples. Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy - you can use some nginx location (and internal URL to same nginx instance) to specify that. Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link) But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 7:38 AM To: nginx-devel at nginx.org Cc: Yury Shpakov Subject: Re: nginx for Windows - WSASend() socket error 10057 It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... try netstat (in cmd as admin): netstat /nabo netstat /nabo | grep -A 1 ":9000\b" and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). additionally try to telnet or curl it: curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html). But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH - nginx.zip, where it works well). Regards, Sergey 12.02.2020 03:01, Yury Shpakov wrote: Hi there, Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: http://nginx.org/en/docs/howto_build_on_win32.html But excluded (don't care about SSL at this point so don't want to install/configure Perl now): --with-openssl=objs/lib/openssl-master \ --with-openssl-opt=no-asm \ --with-http_ssl_module \ And added: --with-mail nmake was successful and nginx.exe was created. However nginx.exe keeps failing with the error: WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Windows API says the following about this error: WSAENOTCONN 10057 Socket is not connected. A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using sendto) no address was supplied. Any other type of operation might also return this error?for example, setsockopt setting SO_KEEPALIVE if the connection has been reset. https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. docs.microsoft.com Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. Any recommendation for me to make it work? Tried to play with config (commenting/uncommenting): #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. Thank you, Yury _______________________________________________ nginx-devel mailing list nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.borisov at nginx.com Wed Feb 26 09:42:52 2020 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Wed, 26 Feb 2020 09:42:52 +0000 Subject: [njs] Fixed handling of space argument in JSON.stringify(). Message-ID: details: https://hg.nginx.org/njs/rev/079d4d4556f0 branches: changeset: 1335:079d4d4556f0 user: Alexander Borisov date: Wed Feb 26 12:41:51 2020 +0300 description: Fixed handling of space argument in JSON.stringify(). This closes #294 issue on GitHub. diffstat: src/njs_json.c | 22 ++++++++++++++++++++-- src/test/njs_unit_test.c | 12 ++++++++++++ 2 files changed, 32 insertions(+), 2 deletions(-) diffs (65 lines): diff -r 0173143d7b15 -r 079d4d4556f0 src/njs_json.c --- a/src/njs_json.c Wed Feb 19 17:21:32 2020 +0300 +++ b/src/njs_json.c Wed Feb 26 12:41:51 2020 +0300 @@ -187,10 +187,13 @@ static njs_int_t njs_json_stringify(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs, njs_index_t unused) { + size_t length; double num; njs_int_t i; njs_int_t ret; njs_value_t *replacer, *space; + const u_char *p; + njs_string_prop_t prop; njs_json_stringify_t *stringify, json_stringify; stringify = &json_stringify; @@ -220,8 +223,23 @@ njs_json_stringify(njs_vm_t *vm, njs_val if (njs_is_string(space) || njs_is_number(space)) { if (njs_is_string(space)) { - njs_string_get(space, &stringify->space); - stringify->space.length = njs_min(stringify->space.length, 10); + length = njs_string_prop(&prop, space); + + if (njs_is_byte_string(&prop)) { + njs_internal_error(vm, "space argument cannot be" + " a byte string"); + return NJS_ERROR; + } + + if (length > 10) { + p = njs_string_offset(prop.start, prop.start + prop.size, 10); + + } else { + p = prop.start + prop.size; + } + + stringify->space.start = prop.start; + stringify->space.length = p - prop.start; } else { num = njs_number(space); diff -r 0173143d7b15 -r 079d4d4556f0 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Wed Feb 19 17:21:32 2020 +0300 +++ b/src/test/njs_unit_test.c Wed Feb 26 12:41:51 2020 +0300 @@ -15674,6 +15674,18 @@ static njs_unit_test_t njs_test[] = { njs_str("JSON.stringify([{a:1,b:{c:2}},1], undefined, new Date())"), njs_str("[{\"a\":1,\"b\":{\"c\":2}},1]") }, + { njs_str("JSON.stringify([], null, '!?????').length"), + njs_str("10") }, + + { njs_str("JSON.stringify([], null, '!!?????????????????').length"), + njs_str("14") }, + + { njs_str("JSON.stringify([], null, '!?????????????????').length"), + njs_str("14") }, + + { njs_str("JSON.stringify([], null, String.bytesFrom([0x9d])).length"), + njs_str("InternalError: space argument cannot be a byte string") }, + { njs_str("var o = Object.defineProperty({}, 'a', { get() { return ()=> 1}, enumerable: true });" "JSON.stringify(o)"), njs_str("{}") }, From serg.brester at sebres.de Wed Feb 26 09:50:43 2020 From: serg.brester at sebres.de (Sergey Brester) Date: Wed, 26 Feb 2020 10:50:43 +0100 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> , <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> , , Message-ID: There are several possibilities to introduce a latency in nginx: - limit_req - https://www.nginx.com/blog/rate-limiting-nginx/#Two-Stage-Rate-Limiting [14] - Maxim's ngx_http_delay (I used it more for development purposes, like test or simulation of load etc); - some "slow" upstream backend that doing nothing, just waiting (preferably asynchronous). You seems to have some upstream (php?) serving auth_http requests, so you could for example implement some delay in case of failed attempt within php (or whatever you use there as backend). Note that it is always good if the latency will be implemented asynchronously (without a real "sleep") in order to avoid possible overload under DDoS similar circumstances. Regards, Sergey. Am 26.02.2020 01:59, schrieb Yury Shpakov: > Hi Sergey, > > You mentioned that you can set up some delays in responses. > How can I do it? > Adding this module during compilation? > https://github.com/openresty/echo-nginx-module [2] > > I tried but it didn't want to compile. I got many compilation errors. > > Maybe I can set up delays somehow else? > > Thank you, > Yury > > ------------------------- > > FROM: nginx-devel on behalf of Yury Shpakov > SENT: Friday, February 14, 2020 6:08 PM > TO: Sergey Brester > CC: nginx-devel at nginx.org > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > So what is the meaning of Auth-Server and Auth-Port headers? So it's relevant only when nginx works as SMTP Proxy (not SMTP Server)? And these are host/port where to redirect SMTP requests? > Yeah, I was all the time surprised -- how come, it's set as Proxy but there is no setting where it redirects SMTP communication to. A little bit unexpected place for those setting. > > Well, let me try... > > I ran Fake SMTP Server on port 25.(I found on Internet some fake SMTP Server). I configured my test SMTP client to localhost:25 (later to 127.0.0.1:25). They send/receive successfully. So both SMTP Client and (fake) SMTP Server work fine. > 127.0.0.1 works fine too. > > I re-configured my test SMTP client to localhost:8025 (tried 127.0.0.1:8025 too). As well, I changed this section of config as follows: > http { > > server { > listen 9000; > > location /cgi-bin/nginxauth.cgi { > add_header Auth-Status OK; > add_header Auth-Server 127.0.0.1; # backend ip > add_header Auth-Port 25; # backend port > return 204; > } > } > } > The same error: > 2020/02/14 17:37:18 [error] 15260#3328: *5 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 > > UPDATE: > Detailed logging with debug information helped a lot. > This is what I noticed in there: > > 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp auth state > > 2020/02/14 17:40:28 [debug] 3940#22096: *1 WSARecv: fd:584 rc:0 24 of 4096 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp rcpt to:"RCPT TO:" > 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer del: 584: 1172123084 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 malloc: 02F8C260:2048 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 stream socket 588 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 connect to [::1]:9000, fd:588 #2 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:768 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:16 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 > So it's trying to use IP6 rather than IP4. > And below: > > 2020/02/14 17:40:29 [debug] 3940#22096: *1 delete posted event 03171170 > > 2020/02/14 17:40:29 [debug] 3940#22096: *1 mail auth http write handler > 2020/02/14 17:40:29 [debug] 3940#22096: *1 WSASend: fd:588, -1, 0 of 306 > 2020/02/14 17:40:29 [error] 3940#22096: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 > 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 > 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 > So, I replaced localhost with 127.0.0.1 like this: > auth_http 127.0.0.1:9000/cgi-bin/nginxauth.cgi; > > And it worked. Since I forced it to use IP4. > Any idea how to use host name instead of IP address and still have it working? > > UPDATE 2: > I figured it out. Googled a little bit and ended up with the following change to my config: > > http { > > server { > listen 9000; > listen [::]:9000 ipv6only=on; > > location /cgi-bin/nginxauth.cgi { > add_header Auth-Status OK; > add_header Auth-Server 127.0.0.1; # backend ip > add_header Auth-Port 25; # backend port > return 204; > } > } > } > Now it works. > But why just "listen 9000" doesn't listen on both IP4 and IP6? > Is it a bug? > > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Friday, February 14, 2020 5:59 AM > TO: Yury Shpakov > CC: nginx-devel at nginx.org > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > I don't know what is wrong with your config... I guess your smtp server does not answer properly. > > Is 127.0.0.2:143 really your SMTP-server? > Because port 143 is mostly an IMAP port - but you've specified PROTOCOL SMTP in the server section. > > Anyway I tested your config with my settings (replaced name and smtp-server, here 192.0.2.222:25) and enabled debug: > > + error_log logs/error-mail.log DEBUG; > mail { > - server_name localhost; > + server_name EXAMPLE.COM; > ... > http { > ... > - add_header Auth-Server 127.0.0.2; > - add_header Auth-Port 143; > + add_header Auth-Server 192.0.2.222; > + add_header Auth-Port 25; > > it works well - I see the test incoming mail (I send to myself via 8025 port) and following output in the log (a lot of irrelevant messages are removed): > >>>>>>>>>> > > 2020/02/14 11:24:04 [debug] 121280#128244: *1 smtp mail from:"mail FROM:" > ... > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request line: "GET /cgi-bin/nginxauth.cgi HTTP/1.0" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http uri: "/cgi-bin/nginxauth.cgi" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http args: "" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http exten: "cgi" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http process request header line > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Host: localhost" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Method: none" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-User: " > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Pass: " > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Protocol: smtp" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Login-Attempt: 1" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-IP: 127.0.0.1" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-Host: [UNAVAILABLE]" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-Helo: myhost.example.com" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-From: mail FROM:" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-To: rcpt TO:" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header done > 2020/02/14 11:24:04 [debug] 121280#128244: *3 event timer del: 512: 1127939767 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 generic phase: 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 1 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 search through nested static locations of "" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 test location: "/cgi-bin/nginxauth.cgi" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 using configuration "/cgi-bin/nginxauth.cgi" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http cl:-1 max:1048576 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 3 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http set discard body > 2020/02/14 11:24:04 [debug] 121280#128244: *3 HTTP/1.1 204 No Content > Server: nginx/1.17.4 > Date: Fri, 14 Feb 2020 10:24:04 GMT > Connection: close > Auth-Status: OK > Auth-Server: 192.0.2.222 > Auth-Port: 25 > > 2020/02/14 11:24:04 [debug] 121280#128244: *3 write new buf t:1 f:0 008AD6A0, pos 008AD6A0, size: 164 file: 0, size: 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter: l:1 f:0 s:164 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter limit 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 WSASend: fd:512, s:164 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter 00000000 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http finalize request: 0, "/cgi-bin/nginxauth.cgi?" a:1, c:1 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request count:1 blk:0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http close request > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http log handler > 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008ACC50, unused: 1161 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 close http connection: 512 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 reusable connection: 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008AC848 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 0039FDE0, unused: 28 > 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http read handler > 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 164 of 1024 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process status line > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process headers > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Server: nginx/1.17.4" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Date: Fri, 14 Feb 2020 10:24:04 GMT" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Connection: close" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Status: OK" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Server: 192.0.2.222" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Port: 25" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header done > 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer del: 496: 1127939764 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 reusable connection: 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 free: 008AC040, unused: 196 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 stream socket 496 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 connect to 192.0.2.222:25, fd:496 #4 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer add: 496: 60000:1127939769 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 008AC040:4096 > 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:04 [debug] 121280#128244: *1 post event 00897120 > 2020/02/14 11:24:04 [debug] 121280#128244: posted event 00897120 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 delete posted event 00897120 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy dummy handler > 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy smtp auth handler > 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 22 of 4096 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy send ehlo > 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 0039FDE0:256 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSASend: fd:496, 0, 25 of 25 > 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 196 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send mail from > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 47 of 47 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 60 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send rcpt to > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 45 of 45 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 63 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer add: 492: 86400000:1214280441 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer del: 496: 1127939769 > 2020/02/14 11:24:05 [info] 121280#128244: *1 client logged in, client: 127.0.0.1, server: 0.0.0.0:8025 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 1, #496 > #492 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 63 of 63 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 6 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 6 of 6 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 50 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 50 of 50 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 170 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 170 of 170 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280535 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 56 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 56 of 56 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > ... > > <<<<<<<<< > > Regards, > Sergey > > 13.02.2020 22:45, Yury Shpakov wrote: > Hi Sergey, > > I reconfigured the config file as follows: > > === === === > > #user nobody; > worker_processes 1; > > #error_log logs/error.log; > #error_log logs/error.log notice; > #error_log logs/error.log info; > > #pid logs/nginx.pid; > > events { > worker_connections 1024; > } > > mail { > server_name localhost; > auth_http localhost:9000/cgi-bin/nginxauth.cgi; > # auth_http none; > > smtp_auth none; > # smtp_auth login plain cram-md5; > # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; > xclient off; > > server { > listen 8025; > protocol smtp; > proxy on; > proxy_pass_error_message on; > } > } > > http { > server { > listen 9000; > > location /cgi-bin/nginxauth.cgi { > add_header Auth-Status OK; > add_header Auth-Server 127.0.0.2; # backend ip > add_header Auth-Port 143; # backend port > return 204; > } > } > } > === === === > > And now it's responding on port 9000 as expected: > > === === === > C:WINDOWSsystem32>curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi > > HTTP/1.1 204 No Content > Server: nginx/1.17.9 > Date: Thu, 13 Feb 2020 21:30:54 GMT > Connection: keep-alive > Auth-Status: OK > Auth-Server: 127.0.0.2 Auth-Port: 143 > === === === > > However I'm still experiencing the same issue (in log file): > > === === === > 2020/02/13 16:29:24 [notice] 35048#26192: signal process started > > 2020/02/13 16:29:34 [error] 31732#22720: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 === === === > > Tried under both admin and regular user. > > Any further ideas how to get it fixed please? > > Thank you, > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Wednesday, February 12, 2020 1:51 PM > TO: Yury Shpakov > CC: nginx-devel at nginx.org > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > I answered inline... > > 12.02.2020 18:59, Yury Shpakov wrote: > Hi Sergey, > > Thank you for you response. > > I tried netstat /nabo and I don't see any reference to port 9000 at all. > So a problem is to make nginx to listen on port 9000 (as server)? > Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? > With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). > Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). > > See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ [3] for more examples. > Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. > > I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. > > Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy [4] - you can use some nginx location (and internal URL to same nginx instance) to specify that. > > Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. > At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. > Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. > And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. > > And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). > Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link [5]) > But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). > > Thank you, > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Wednesday, February 12, 2020 7:38 AM > TO: nginx-devel at nginx.org > CC: Yury Shpakov > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... > > try netstat (in cmd as admin): > > netstat /nabo > netstat /nabo | grep -A 1 ":9000b" > > and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). > > additionally try to telnet or curl it: > > curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi > > if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). > > If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [6]). > > But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. > It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). > > Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [7] > > Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH [5] - nginx.zip [8], where it works well). > > Regards, > Sergey > > 12.02.2020 03:01, Yury Shpakov wrote: > Hi there, > > Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: > http://nginx.org/en/docs/howto_build_on_win32.html [9] > But excluded (don't care about SSL at this point so don't want to install/configure Perl now): > --with-openssl=objs/lib/openssl-master > > --with-openssl-opt=no-asm > --with-http_ssl_module > And added: > --with-mail > > nmake was successful and nginx.exe was created. > However nginx.exe keeps failing with the error: > WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 > Windows API says the following about this error: > > WSAENOTCONN10057 > Socket is not connected.A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using SENDTO [10]) no address was supplied. Any other type of operation might also return this error--for example, SETSOCKOPT [11] setting SO_KEEPALIVE [12] if the connection has been reset. > > https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 [13] > > Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs [13] > Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. > docs.microsoft.com > > Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. > Any recommendation for me to make it work? > Tried to play with config (commenting/uncommenting): > > #user nobody; > worker_processes 1; > #error_log logs/error.log; > #error_log logs/error.log notice; > #error_log logs/error.log info; > #pid logs/nginx.pid; > events { > worker_connections 1024; > } > mail { > server_name localhost; > auth_http localhost:9000/cgi-bin/nginxauth.cgi; > # auth_http none; > smtp_auth none; > # smtp_auth login plain cram-md5; > # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; > xclient off; > server { > listen 8025; > protocol smtp; > proxy on; > proxy_pass_error_message on; > } > } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. > Thank you, > Yury > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel [1] Links: ------ [1] http://mailman.nginx.org/mailman/listinfo/nginx-devel [2] https://github.com/openresty/echo-nginx-module [3] https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ [4] https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy [5] https://github.com/sebres/nginx/releases/tag/release-1.13.0 [6] http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [7] https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [8] https://github.com/sebres/nginx/files/2246440/nginx.zip [9] http://nginx.org/en/docs/howto_build_on_win32.html [10] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-sendto [11] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-setsockopt [12] https://docs.microsoft.com/en-us/windows/desktop/winsock/so-keepalive [13] https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 [14] https://www.nginx.com/blog/rate-limiting-nginx/#Two-Stage-Rate-Limiting -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.borisov at nginx.com Wed Feb 26 13:22:48 2020 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Wed, 26 Feb 2020 13:22:48 +0000 Subject: [njs] Lexer refactoring. Message-ID: details: https://hg.nginx.org/njs/rev/87d05fb35ff9 branches: changeset: 1336:87d05fb35ff9 user: Alexander Borisov date: Wed Feb 26 16:22:10 2020 +0300 description: Lexer refactoring. diffstat: src/njs_builtin.c | 54 +-- src/njs_function.c | 2 +- src/njs_generator.c | 64 ++- src/njs_lexer.c | 632 ++++++++++++++++++++++++++------------------- src/njs_lexer.h | 80 ++++- src/njs_lexer_keyword.c | 180 +++--------- src/njs_lexer_tables.h | 146 ++++++++++ src/njs_module.c | 2 +- src/njs_parser.c | 76 +++-- src/njs_parser.h | 33 +- src/njs_parser_terminal.c | 40 +- src/njs_shell.c | 43 +- src/njs_variable.c | 358 ++++++++++--------------- src/njs_variable.h | 52 ++- src/njs_vm.c | 2 +- src/njs_vm.h | 2 +- utils/lexer_keyword.py | 245 +++++++++++++++++ 17 files changed, 1226 insertions(+), 785 deletions(-) diffs (truncated from 3036 to 1000 lines): diff -r 079d4d4556f0 -r 87d05fb35ff9 src/njs_builtin.c --- a/src/njs_builtin.c Wed Feb 26 12:41:51 2020 +0300 +++ b/src/njs_builtin.c Wed Feb 26 16:22:10 2020 +0300 @@ -134,12 +134,6 @@ njs_builtin_objects_create(njs_vm_t *vm) } njs_lvlhsh_init(&shared->keywords_hash); - - ret = njs_lexer_keywords_init(vm->mem_pool, &shared->keywords_hash); - if (njs_slow_path(ret != NJS_OK)) { - return NJS_ERROR; - } - njs_lvlhsh_init(&shared->values_hash); pattern = njs_regexp_pattern_create(vm, (u_char *) "(?:)", @@ -495,7 +489,6 @@ njs_builtin_completions(njs_vm_t *vm) njs_arr_t *array; njs_str_t *completion; njs_int_t ret; - njs_keyword_t *keyword; njs_lvlhsh_each_t lhe; njs_builtin_traverse_t ctx; const njs_object_prop_t *prop; @@ -505,23 +498,9 @@ njs_builtin_completions(njs_vm_t *vm) return NULL; } - /* Keywords completions. */ - - njs_lvlhsh_each_init(&lhe, &njs_keyword_hash_proto); - - for ( ;; ) { - keyword = njs_lvlhsh_each(&vm->shared->keywords_hash, &lhe); - - if (keyword == NULL) { - break; - } - - completion = njs_arr_add(array); - if (njs_slow_path(completion == NULL)) { - return NULL; - } - - *completion = keyword->name; + ret = njs_lexer_keywords(array); + if (njs_slow_path(ret != NJS_OK)) { + return NULL; } /* Global object completions. */ @@ -570,12 +549,14 @@ njs_vm_completions(njs_vm_t *vm, njs_str static njs_arr_t * njs_vm_expression_completions(njs_vm_t *vm, njs_str_t *expression) { - u_char *p, *end; - njs_int_t ret; - njs_value_t *value; - njs_variable_t *var; - njs_object_prop_t *prop; - njs_lvlhsh_query_t lhq; + u_char *p, *end; + njs_int_t ret; + njs_value_t *value; + njs_variable_t *var; + njs_rbtree_node_t *node; + njs_object_prop_t *prop; + njs_lvlhsh_query_t lhq; + njs_variable_node_t var_node; if (njs_slow_path(vm->parser == NULL)) { return NULL; @@ -588,16 +569,23 @@ njs_vm_expression_completions(njs_vm_t * while (p < end && *p != '.') { p++; } - lhq.proto = &njs_variables_hash_proto; + lhq.proto = &njs_lexer_hash_proto; lhq.key.length = p - lhq.key.start; lhq.key_hash = njs_djb_hash(lhq.key.start, lhq.key.length); - ret = njs_lvlhsh_find(&vm->parser->scope->variables, &lhq); + ret = njs_lvlhsh_find(&vm->shared->keywords_hash, &lhq); if (njs_slow_path(ret != NJS_OK)) { return NULL; } - var = lhq.value; + var_node.key = (uintptr_t) lhq.value; + + node = njs_rbtree_find(&vm->parser->scope->variables, &var_node.node); + if (njs_slow_path(node == NULL)) { + return NULL; + } + + var = ((njs_variable_node_t *) node)->variable; value = njs_vmcode_operand(vm, var->index); if (!njs_is_object(value)) { diff -r 079d4d4556f0 -r 87d05fb35ff9 src/njs_function.c --- a/src/njs_function.c Wed Feb 26 12:41:51 2020 +0300 +++ b/src/njs_function.c Wed Feb 26 16:22:10 2020 +0300 @@ -932,7 +932,7 @@ njs_function_constructor(njs_vm_t *vm, n scope = parser->scope; - ret = njs_variables_copy(vm, &scope->variables, &vm->variables_hash); + ret = njs_variables_copy(vm, &scope->variables, vm->variables_hash); if (njs_slow_path(ret != NJS_OK)) { return ret; } diff -r 079d4d4556f0 -r 87d05fb35ff9 src/njs_generator.c --- a/src/njs_generator.c Wed Feb 26 12:41:51 2020 +0300 +++ b/src/njs_generator.c Wed Feb 26 16:22:10 2020 +0300 @@ -384,7 +384,8 @@ njs_generate(njs_vm_t *vm, njs_generator return njs_generate_inc_dec_operation(vm, generator, node, 1); case NJS_TOKEN_NULL: - case NJS_TOKEN_BOOLEAN: + case NJS_TOKEN_TRUE: + case NJS_TOKEN_FALSE: case NJS_TOKEN_NUMBER: case NJS_TOKEN_STRING: node->index = njs_value_index(vm, &node->u.value, generator->runtime); @@ -2321,9 +2322,10 @@ static njs_int_t njs_generate_function_declaration(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node) { - njs_int_t ret; - njs_variable_t *var; - njs_function_lambda_t *lambda; + njs_int_t ret; + njs_variable_t *var; + njs_function_lambda_t *lambda; + const njs_lexer_entry_t *lex_entry; var = njs_variable_resolve(vm, node); if (njs_slow_path(var == NULL)) { @@ -2337,14 +2339,18 @@ njs_generate_function_declaration(njs_vm lambda = njs_function_lambda(&var->value); - ret = njs_generate_function_scope(vm, lambda, node, - &node->u.reference.name); + lex_entry = njs_lexer_entry(node->u.reference.unique_id); + if (njs_slow_path(lex_entry == NULL)) { + return NJS_ERROR; + } + + ret = njs_generate_function_scope(vm, lambda, node, &lex_entry->name); if (njs_slow_path(ret != NJS_OK)) { return ret; } if (vm->debug != NULL) { - ret = njs_generate_function_debug(vm, &var->name, lambda, node); + ret = njs_generate_function_debug(vm, &lex_entry->name, lambda, node); } return ret; @@ -2473,15 +2479,17 @@ njs_generate_lambda_variables(njs_vm_t * { njs_index_t index; njs_variable_t *var; + njs_rbtree_node_t *rb_node; njs_vmcode_move_t *move; - njs_lvlhsh_each_t lhe; njs_vmcode_this_t *this; + njs_variable_node_t *var_node; njs_vmcode_arguments_t *arguments; - njs_lvlhsh_each_init(&lhe, &njs_variables_hash_proto); - - for ( ;; ) { - var = njs_lvlhsh_each(&node->scope->variables, &lhe); + rb_node = njs_rbtree_min(&node->scope->variables); + + while (njs_rbtree_is_there_successor(&node->scope->variables, rb_node)) { + var_node = (njs_variable_node_t *) rb_node; + var = var_node->variable; if (var == NULL) { break; @@ -2504,6 +2512,8 @@ njs_generate_lambda_variables(njs_vm_t * NJS_VMCODE_ARGUMENTS, 1); arguments->dst = var->index; } + + rb_node = njs_rbtree_node_successor(&node->scope->variables, rb_node); } return NJS_OK; @@ -3297,11 +3307,11 @@ static njs_int_t njs_generate_global_reference(njs_vm_t *vm, njs_generator_t *generator, njs_parser_node_t *node, njs_bool_t exception) { - njs_str_t *name; - njs_int_t ret; - njs_index_t index; - njs_value_t property; - njs_vmcode_prop_get_t *prop_get; + njs_int_t ret; + njs_index_t index; + njs_value_t property; + njs_vmcode_prop_get_t *prop_get; + const njs_lexer_entry_t *lex_entry; index = njs_generate_dest_index(vm, generator, node); if (njs_slow_path(index == NJS_INDEX_ERROR)) { @@ -3314,11 +3324,13 @@ njs_generate_global_reference(njs_vm_t * prop_get->value = index; prop_get->object = NJS_INDEX_GLOBAL_OBJECT; - /* FIXME: cache keys in a hash. */ - - name = &node->u.reference.name; - - ret = njs_string_set(vm, &property, name->start, name->length); + lex_entry = njs_lexer_entry(node->u.reference.unique_id); + if (njs_slow_path(lex_entry == NULL)) { + return NJS_ERROR; + } + + ret = njs_string_set(vm, &property, lex_entry->name.start, + lex_entry->name.length); if (njs_slow_path(ret != NJS_OK)) { return NJS_ERROR; } @@ -3343,6 +3355,7 @@ njs_generate_reference_error(njs_vm_t *v njs_parser_node_t *node) { njs_jump_off_t ret; + const njs_lexer_entry_t *lex_entry; njs_vmcode_reference_error_t *ref_err; if (njs_slow_path(!node->u.reference.not_defined)) { @@ -3365,7 +3378,12 @@ njs_generate_reference_error(njs_vm_t *v } } - return njs_name_copy(vm, &ref_err->name, &node->u.reference.name); + lex_entry = njs_lexer_entry(node->u.reference.unique_id); + if (njs_slow_path(lex_entry == NULL)) { + return NJS_ERROR; + } + + return njs_name_copy(vm, &ref_err->name, &lex_entry->name); } diff -r 079d4d4556f0 -r 87d05fb35ff9 src/njs_lexer.c --- a/src/njs_lexer.c Wed Feb 26 12:41:51 2020 +0300 +++ b/src/njs_lexer.c Wed Feb 26 16:22:10 2020 +0300 @@ -18,23 +18,28 @@ struct njs_lexer_multi_s { }; +static njs_int_t njs_lexer_hash_test(njs_lvlhsh_query_t *lhq, void *data); +static njs_int_t njs_lexer_word(njs_lexer_t *lexer, njs_lexer_token_t *token); +static void njs_lexer_string(njs_lexer_t *lexer, njs_lexer_token_t *token, + u_char quote); +static void njs_lexer_number(njs_lexer_t *lexer, njs_lexer_token_t *token); +static void njs_lexer_multi(njs_lexer_t *lexer, njs_lexer_token_t *token, + const njs_lexer_multi_t *multi, size_t length); +static void njs_lexer_division(njs_lexer_t *lexer, njs_lexer_token_t *token); + static njs_lexer_token_t *njs_lexer_token_push(njs_vm_t *vm, njs_lexer_t *lexer); static njs_lexer_token_t *njs_lexer_token_pop(njs_lexer_t *lexer); -static njs_token_t njs_lexer_token_name_resolve(njs_lexer_t *lexer, - njs_lexer_token_t *lt); -static njs_token_t njs_lexer_next_token(njs_lexer_t *lexer, - njs_lexer_token_t *lt); -static njs_token_t njs_lexer_word(njs_lexer_t *lexer, njs_lexer_token_t *lt, - u_char c); -static njs_token_t njs_lexer_string(njs_lexer_t *lexer, njs_lexer_token_t *lt, - u_char quote); -static njs_token_t njs_lexer_number(njs_lexer_t *lexer, njs_lexer_token_t *lt, - u_char c); -static njs_token_t njs_lexer_multi(njs_lexer_t *lexer, njs_lexer_token_t *lt, - njs_token_t token, njs_uint_t n, const njs_lexer_multi_t *multi); -static njs_token_t njs_lexer_division(njs_lexer_t *lexer, - njs_token_t token); + + +const njs_lvlhsh_proto_t njs_lexer_hash_proto + njs_aligned(64) = +{ + NJS_LVLHSH_DEFAULT, + njs_lexer_hash_test, + njs_lvlhsh_alloc, + njs_lvlhsh_free, +}; static const uint8_t njs_tokens[256] njs_aligned(64) = { @@ -297,7 +302,8 @@ njs_lexer_init(njs_vm_t *vm, njs_lexer_t lexer->start = start; lexer->end = end; lexer->line = 1; - lexer->keywords_hash = vm->shared->keywords_hash; + lexer->keywords_hash = &vm->shared->keywords_hash; + lexer->mem_pool = vm->mem_pool; njs_queue_init(&lexer->preread); @@ -312,9 +318,9 @@ njs_lexer_token(njs_vm_t *vm, njs_lexer_ lexer->prev_start = lexer->start; - if (lexer->lexer_token != NULL) { - lexer->prev_token = lexer->lexer_token->token; - njs_mp_free(vm->mem_pool, lexer->lexer_token); + if (lexer->token != NULL) { + lexer->prev_token = lexer->token->type; + njs_mp_free(vm->mem_pool, lexer->token); } if (njs_queue_is_empty(&lexer->preread)) { @@ -324,9 +330,9 @@ njs_lexer_token(njs_vm_t *vm, njs_lexer_ } } - lexer->lexer_token = njs_lexer_token_pop(lexer); + lexer->token = njs_lexer_token_pop(lexer); - return njs_lexer_token_name_resolve(lexer, lexer->lexer_token); + return lexer->token->type; } @@ -350,9 +356,7 @@ njs_lexer_peek_token(njs_vm_t *vm, njs_l /* NJS_TOKEN_DIVISION stands for regexp literal. */ - if (lt->token == NJS_TOKEN_DIVISION - || lt->token == NJS_TOKEN_END) - { + if (lt->type == NJS_TOKEN_DIVISION || lt->type == NJS_TOKEN_END) { break; } @@ -368,25 +372,50 @@ njs_lexer_peek_token(njs_vm_t *vm, njs_l } } - return njs_lexer_token_name_resolve(lexer, lt); + return lt->type; +} + + +njs_int_t +njs_lexer_rollback(njs_vm_t *vm, njs_lexer_t *lexer) +{ + njs_lexer_token_t *lt; + + lt = njs_mp_zalloc(vm->mem_pool, sizeof(njs_lexer_token_t)); + if (njs_slow_path(lt == NULL)) { + return NJS_ERROR; + } + + *lt = *lexer->token; + + njs_queue_insert_head(&lexer->preread, <->link); + + return NJS_OK; } static njs_lexer_token_t * njs_lexer_token_push(njs_vm_t *vm, njs_lexer_t *lexer) { - njs_lexer_token_t *lt; + njs_int_t ret; + njs_lexer_token_t *token; - lt = njs_mp_zalloc(vm->mem_pool, sizeof(njs_lexer_token_t)); - if (njs_slow_path(lt == NULL)) { + token = njs_mp_zalloc(vm->mem_pool, sizeof(njs_lexer_token_t)); + if (njs_slow_path(token == NULL)) { return NULL; } - lt->token = njs_lexer_next_token(lexer, lt); + do { + ret = njs_lexer_next_token(lexer, token); + if (njs_slow_path(ret != NJS_OK)) { + return NULL; + } - njs_queue_insert_tail(&lexer->preread, <->link); + } while (token->type == NJS_TOKEN_COMMENT); - return lt; + njs_queue_insert_tail(&lexer->preread, &token->link); + + return token; } @@ -403,197 +432,223 @@ njs_lexer_token_pop(njs_lexer_t *lexer) njs_int_t -njs_lexer_rollback(njs_vm_t *vm, njs_lexer_t *lexer) +njs_lexer_next_token(njs_lexer_t *lexer, njs_lexer_token_t *token) { - njs_lexer_token_t *lt; + u_char c, *p; + + c = ' '; - lt = njs_mp_zalloc(vm->mem_pool, sizeof(njs_lexer_token_t)); - if (njs_slow_path(lt == NULL)) { - return NJS_ERROR; + while (lexer->start < lexer->end) { + c = *lexer->start++; + + if (njs_tokens[c] != NJS_TOKEN_SPACE) { + break; + } } - *lt = *lexer->lexer_token; + lexer->keyword = 0; + token->type = njs_tokens[c]; + + switch (token->type) { + + case NJS_TOKEN_LETTER: + return njs_lexer_word(lexer, token); + + case NJS_TOKEN_DOUBLE_QUOTE: + case NJS_TOKEN_SINGLE_QUOTE: + njs_lexer_string(lexer, token, c); + break; + + case NJS_TOKEN_DOT: + p = lexer->start; + + if (p + 1 < lexer->end + && njs_tokens[p[0]] == NJS_TOKEN_DOT + && njs_tokens[p[1]] == NJS_TOKEN_DOT) + { + token->text.start = lexer->start - 1; + token->text.length = (p - token->text.start) + 2; + + token->type = NJS_TOKEN_ELLIPSIS; + + lexer->start += 2; + + return NJS_OK; + } + + if (p == lexer->end || njs_tokens[*p] != NJS_TOKEN_DIGIT) { + token->text.start = lexer->start - 1; + token->text.length = p - token->text.start; + + token->type = NJS_TOKEN_DOT; + + return NJS_OK; + } + + /* Fall through. */ + + case NJS_TOKEN_DIGIT: + njs_lexer_number(lexer, token); + break; + + case NJS_TOKEN_DIVISION: + njs_lexer_division(lexer, token); + break; + + case NJS_TOKEN_ASSIGNMENT: + njs_lexer_multi(lexer, token, njs_assignment_token, + njs_nitems(njs_assignment_token)); + break; + + case NJS_TOKEN_ADDITION: + njs_lexer_multi(lexer, token, njs_addition_token, + njs_nitems(njs_addition_token)); + break; - njs_queue_insert_head(&lexer->preread, <->link); + case NJS_TOKEN_SUBSTRACTION: + njs_lexer_multi(lexer, token, njs_substraction_token, + njs_nitems(njs_substraction_token)); + break; + + case NJS_TOKEN_MULTIPLICATION: + njs_lexer_multi(lexer, token, njs_multiplication_token, + njs_nitems(njs_multiplication_token)); + break; + + case NJS_TOKEN_REMAINDER: + njs_lexer_multi(lexer, token, njs_remainder_token, + njs_nitems(njs_remainder_token)); + break; + + case NJS_TOKEN_BITWISE_AND: + njs_lexer_multi(lexer, token, njs_bitwise_and_token, + njs_nitems(njs_bitwise_and_token)); + break; + + case NJS_TOKEN_BITWISE_XOR: + njs_lexer_multi(lexer, token, njs_bitwise_xor_token, + njs_nitems(njs_bitwise_xor_token)); + break; + + case NJS_TOKEN_BITWISE_OR: + njs_lexer_multi(lexer, token, njs_bitwise_or_token, + njs_nitems(njs_bitwise_or_token)); + break; + + case NJS_TOKEN_LOGICAL_NOT: + njs_lexer_multi(lexer, token, njs_logical_not_token, + njs_nitems(njs_logical_not_token)); + break; + + case NJS_TOKEN_LESS: + njs_lexer_multi(lexer, token, njs_less_token, + njs_nitems(njs_less_token)); + break; + + case NJS_TOKEN_GREATER: + njs_lexer_multi(lexer, token, njs_greater_token, + njs_nitems(njs_greater_token)); + break; + + case NJS_TOKEN_CONDITIONAL: + njs_lexer_multi(lexer, token, njs_conditional_token, + njs_nitems(njs_conditional_token)); + break; + + case NJS_TOKEN_SPACE: + token->type = NJS_TOKEN_END; + return NJS_OK; + + case NJS_TOKEN_LINE_END: + lexer->line++; + + /* Fall through. */ + + default: + token->text.start = lexer->start - 1; + token->text.length = lexer->start - token->text.start; + + break; + } return NJS_OK; } -static njs_token_t -njs_lexer_token_name_resolve(njs_lexer_t *lexer, njs_lexer_token_t *lt) +static njs_int_t +njs_lexer_hash_test(njs_lvlhsh_query_t *lhq, void *data) { - if (lt->token == NJS_TOKEN_NAME) { - njs_lexer_keyword(lexer, lt); + njs_lexer_entry_t *entry; + + entry = data; + + if (entry->name.length == lhq->key.length + && memcmp(entry->name.start, lhq->key.start, lhq->key.length) == 0) + { + return NJS_OK; } - return lt->token; + return NJS_DECLINED; } -static njs_token_t -njs_lexer_next_token(njs_lexer_t *lexer, njs_lexer_token_t *lt) +static njs_lexer_entry_t * +njs_lexer_keyword_find(njs_lexer_t *lexer, u_char *key, size_t length, + uint32_t hash) { - u_char c, *p; - njs_uint_t n; - njs_token_t token; - const njs_lexer_multi_t *multi; - - lt->text.start = lexer->start; - - while (lexer->start < lexer->end) { - c = *lexer->start++; - - token = njs_tokens[c]; - - switch (token) { - - case NJS_TOKEN_SPACE: - lt->text.start = lexer->start; - continue; - - case NJS_TOKEN_LETTER: - return njs_lexer_word(lexer, lt, c); - - case NJS_TOKEN_DOUBLE_QUOTE: - case NJS_TOKEN_SINGLE_QUOTE: - return njs_lexer_string(lexer, lt, c); - - case NJS_TOKEN_DOT: - p = lexer->start; - - if (p + 1 < lexer->end - && njs_tokens[p[0]] == NJS_TOKEN_DOT - && njs_tokens[p[1]] == NJS_TOKEN_DOT) - { - lt->text.length = (p - lt->text.start) + 2; - lexer->start += 2; - return NJS_TOKEN_ELLIPSIS; - } - - if (p == lexer->end || njs_tokens[*p] != NJS_TOKEN_DIGIT) { - lt->text.length = p - lt->text.start; - return NJS_TOKEN_DOT; - } - - /* Fall through. */ - - case NJS_TOKEN_DIGIT: - return njs_lexer_number(lexer, lt, c); - - case NJS_TOKEN_ASSIGNMENT: - n = njs_nitems(njs_assignment_token), - multi = njs_assignment_token; - - goto multi; - - case NJS_TOKEN_ADDITION: - n = njs_nitems(njs_addition_token), - multi = njs_addition_token; - - goto multi; - - case NJS_TOKEN_SUBSTRACTION: - n = njs_nitems(njs_substraction_token), - multi = njs_substraction_token; - - goto multi; - - case NJS_TOKEN_MULTIPLICATION: - n = njs_nitems(njs_multiplication_token), - multi = njs_multiplication_token; + njs_int_t ret; + njs_lexer_entry_t *entry; + njs_lvlhsh_query_t lhq; - goto multi; - - case NJS_TOKEN_DIVISION: - token = njs_lexer_division(lexer, token); - - if (token != NJS_TOKEN_AGAIN) { - goto done; - } - - continue; - - case NJS_TOKEN_REMAINDER: - n = njs_nitems(njs_remainder_token), - multi = njs_remainder_token; - - goto multi; - - case NJS_TOKEN_BITWISE_AND: - n = njs_nitems(njs_bitwise_and_token), - multi = njs_bitwise_and_token; - - goto multi; - - case NJS_TOKEN_BITWISE_XOR: - n = njs_nitems(njs_bitwise_xor_token), - multi = njs_bitwise_xor_token; - - goto multi; - - case NJS_TOKEN_BITWISE_OR: - n = njs_nitems(njs_bitwise_or_token), - multi = njs_bitwise_or_token; - - goto multi; + lhq.key.start = key; + lhq.key.length = length; - case NJS_TOKEN_LOGICAL_NOT: - n = njs_nitems(njs_logical_not_token), - multi = njs_logical_not_token; - - goto multi; - - case NJS_TOKEN_LESS: - n = njs_nitems(njs_less_token), - multi = njs_less_token; - - goto multi; - - case NJS_TOKEN_GREATER: - n = njs_nitems(njs_greater_token), - multi = njs_greater_token; - - goto multi; + lhq.key_hash = hash; + lhq.proto = &njs_lexer_hash_proto; - case NJS_TOKEN_CONDITIONAL: - n = njs_nitems(njs_conditional_token), - multi = njs_conditional_token; - - goto multi; - - case NJS_TOKEN_LINE_END: - lexer->line++; - - /* Fall through. */ - - default: - goto done; - } - - multi: - - return njs_lexer_multi(lexer, lt, token, n, multi); + ret = njs_lvlhsh_find(lexer->keywords_hash, &lhq); + if (ret == NJS_OK) { + return lhq.value; } - token = NJS_TOKEN_END; + entry = njs_mp_alloc(lexer->mem_pool, sizeof(njs_lexer_entry_t)); + if (njs_slow_path(entry == NULL)) { + return NULL; + } -done: + entry->name.start = njs_mp_alloc(lexer->mem_pool, length + 1); + if (njs_slow_path(entry->name.start == NULL)) { + return NULL; + } + + memcpy(entry->name.start, key, length); - lt->text.length = lexer->start - lt->text.start; + entry->name.start[length] = '\0'; + entry->name.length = length; + + lhq.value = entry; + lhq.pool = lexer->mem_pool; - return token; + ret = njs_lvlhsh_insert(lexer->keywords_hash, &lhq); + if (njs_slow_path(ret != NJS_OK)) { + return NULL; + } + + return entry; } -static njs_token_t -njs_lexer_word(njs_lexer_t *lexer, njs_lexer_token_t *lt, u_char c) +static njs_int_t +njs_lexer_word(njs_lexer_t *lexer, njs_lexer_token_t *token) { - u_char *p; + u_char *p, c; + uint32_t hash_id; + const njs_lexer_entry_t *entry; + const njs_lexer_keyword_entry_t *key_entry; /* TODO: UTF-8 */ - static const uint8_t letter_digit[32] njs_aligned(32) = { + static const uint8_t letter_digit[32] njs_aligned(32) = { 0x00, 0x00, 0x00, 0x00, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ /* '&%$ #"! /.-, |*)( 7654 3210 ?>=< ;:98 */ @@ -611,9 +666,10 @@ njs_lexer_word(njs_lexer_t *lexer, njs_l 0x00, 0x00, 0x00, 0x00, /* 0000 0000 0000 0000 0000 0000 0000 0000 */ }; - lt->token_line = lexer->line; - lt->key_hash = njs_djb_hash_add(NJS_DJB_HASH_INIT, c); - lt->text.start = lexer->start - 1; + token->line = lexer->line; + token->text.start = lexer->start - 1; + + hash_id = njs_djb_hash_add(NJS_DJB_HASH_INIT, *token->text.start); for (p = lexer->start; p < lexer->end; p++) { c = *p; @@ -622,25 +678,46 @@ njs_lexer_word(njs_lexer_t *lexer, njs_l break; } - lt->key_hash = njs_djb_hash_add(lt->key_hash, c); + hash_id = njs_djb_hash_add(hash_id, c); } + token->text.length = p - token->text.start; lexer->start = p; - lt->text.length = p - lt->text.start; + + key_entry = njs_lexer_keyword(token->text.start, token->text.length); + + if (key_entry == NULL) { + entry = njs_lexer_keyword_find(lexer, token->text.start, + token->text.length, hash_id); + if (njs_slow_path(entry == NULL)) { + return NJS_ERROR; + } - return NJS_TOKEN_NAME; + token->type = NJS_TOKEN_NAME; + + } else { + entry = &key_entry->value->entry; + token->type = key_entry->value->type; + + lexer->keyword = 1; + } + + token->unique_id = (uintptr_t) entry; + + return NJS_OK; } -static njs_token_t -njs_lexer_string(njs_lexer_t *lexer, njs_lexer_token_t *lt, u_char quote) +static void +njs_lexer_string(njs_lexer_t *lexer, njs_lexer_token_t *token, u_char quote) { u_char *p, c; njs_bool_t escape; escape = 0; - lt->text.start = lexer->start; + p = lexer->start; + token->text.start = p; while (p < lexer->end) { @@ -670,31 +747,31 @@ njs_lexer_string(njs_lexer_t *lexer, njs if (c == quote) { lexer->start = p; - lt->text.length = (p - 1) - lt->text.start; + token->text.length = (p - 1) - token->text.start; - if (escape == 0) { - return NJS_TOKEN_STRING; - } - - return NJS_TOKEN_ESCAPE_STRING; + token->type = (escape == 0) ? NJS_TOKEN_STRING + : NJS_TOKEN_ESCAPE_STRING; + return; } } - lt->text.start--; - lt->text.length = p - lt->text.start; + token->text.start--; + token->text.length = p - token->text.start; - return NJS_TOKEN_UNTERMINATED_STRING; + token->type = NJS_TOKEN_UNTERMINATED_STRING; } -static njs_token_t -njs_lexer_number(njs_lexer_t *lexer, njs_lexer_token_t *lt, u_char c) +static void +njs_lexer_number(njs_lexer_t *lexer, njs_lexer_token_t *token) { + u_char c; const u_char *p; - lt->text.start = lexer->start - 1; + c = lexer->start[-1]; + p = lexer->start; - p = lexer->start; + token->text.start = lexer->start - 1; if (c == '0' && p != lexer->end) { @@ -707,7 +784,7 @@ njs_lexer_number(njs_lexer_t *lexer, njs goto illegal_token; } - lt->number = njs_number_hex_parse(&p, lexer->end); + token->number = njs_number_hex_parse(&p, lexer->end); goto done; } @@ -721,7 +798,7 @@ njs_lexer_number(njs_lexer_t *lexer, njs goto illegal_token; } - lt->number = njs_number_oct_parse(&p, lexer->end); + token->number = njs_number_oct_parse(&p, lexer->end); if (p < lexer->end && (*p == '8' || *p == '9')) { goto illegal_trailer; @@ -739,7 +816,7 @@ njs_lexer_number(njs_lexer_t *lexer, njs goto illegal_token; } - lt->number = njs_number_bin_parse(&p, lexer->end); + token->number = njs_number_bin_parse(&p, lexer->end); if (p < lexer->end && (*p >= '2' && *p <= '9')) { goto illegal_trailer; @@ -756,14 +833,16 @@ njs_lexer_number(njs_lexer_t *lexer, njs } p--; - lt->number = njs_number_dec_parse(&p, lexer->end); + token->number = njs_number_dec_parse(&p, lexer->end); done: lexer->start = (u_char *) p; - lt->text.length = p - lt->text.start; + token->text.length = p - token->text.start; - return NJS_TOKEN_NUMBER; + token->type = NJS_TOKEN_NUMBER; + + return; illegal_trailer: @@ -771,92 +850,105 @@ illegal_trailer: illegal_token: - lt->text.length = p - lt->text.start; + token->text.length = p - token->text.start; - return NJS_TOKEN_ILLEGAL; + token->type = NJS_TOKEN_ILLEGAL; } -static njs_token_t -njs_lexer_multi(njs_lexer_t *lexer, njs_lexer_token_t *lt, njs_token_t token, - njs_uint_t n, const njs_lexer_multi_t *multi) +static void +njs_lexer_multi(njs_lexer_t *lexer, njs_lexer_token_t *token, + const njs_lexer_multi_t *multi, size_t length) { u_char c; - if (lexer->start < lexer->end) { + token->text.start = lexer->start - 1; + + while (length != 0 && multi != NULL) { c = lexer->start[0]; - do { - if (c == multi->symbol) { - lexer->start++; + if (c == multi->symbol) { + lexer->start++; - if (multi->count == 0) { - token = multi->token; - break; - } + token->type = multi->token; - return njs_lexer_multi(lexer, lt, multi->token, multi->count, - multi->next); + if (multi->count == 0) { + break; } + length = multi->count; + multi = multi->next; + + } else { + length--; multi++; - n--; - - } while (n != 0); + } } - lt->text.length = lexer->start - lt->text.start; From alexander.borisov at nginx.com Wed Feb 26 13:22:50 2020 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Wed, 26 Feb 2020 13:22:50 +0000 Subject: [njs] Improved fields and typedefs naming in parser. Message-ID: details: https://hg.nginx.org/njs/rev/112c7f54f402 branches: changeset: 1337:112c7f54f402 user: Alexander Borisov date: Wed Feb 26 16:22:12 2020 +0300 description: Improved fields and typedefs naming in parser. njs_token_t -> njs_token_type_t node->token -> node->token_type lexer->prev_token -> lexer->prev_type diffstat: src/njs_generator.c | 30 +- src/njs_lexer.c | 6 +- src/njs_lexer.h | 12 +- src/njs_module.c | 10 +- src/njs_parser.c | 994 ++++++++++++++++++++++--------------------- src/njs_parser.h | 84 +- src/njs_parser_expression.c | 475 ++++++++++---------- src/njs_parser_terminal.c | 195 ++++---- src/njs_regexp.c | 2 +- src/njs_regexp.h | 2 +- 10 files changed, 909 insertions(+), 901 deletions(-) diffs (truncated from 3972 to 1000 lines): diff -r 87d05fb35ff9 -r 112c7f54f402 src/njs_generator.c --- a/src/njs_generator.c Wed Feb 26 16:22:10 2020 +0300 +++ b/src/njs_generator.c Wed Feb 26 16:22:12 2020 +0300 @@ -264,7 +264,7 @@ njs_generate(njs_vm_t *vm, njs_generator return NJS_OK; } - switch (node->token) { + switch (node->token_type) { case NJS_TOKEN_VAR: return njs_generate_var_statement(vm, generator, node); @@ -664,7 +664,7 @@ njs_generate_if_statement(njs_vm_t *vm, jump_offset = njs_code_offset(generator, cond_jump); label_offset = jump_offset + offsetof(njs_vmcode_cond_jump_t, offset); - if (node->right != NULL && node->right->token == NJS_TOKEN_BRANCHING) { + if (node->right != NULL && node->right->token_type == NJS_TOKEN_BRANCHING) { /* The "then" branch in a case of "if/then/else" statement. */ @@ -836,7 +836,7 @@ njs_generate_switch_statement(njs_vm_t * for (branch = swtch->right; branch != NULL; branch = branch->left) { - if (branch->token != NJS_TOKEN_DEFAULT) { + if (branch->token_type != NJS_TOKEN_DEFAULT) { /* The "case" expression. */ @@ -887,7 +887,7 @@ njs_generate_switch_statement(njs_vm_t * for (branch = swtch->right; branch != NULL; branch = branch->left) { - if (branch->token == NJS_TOKEN_DEFAULT) { + if (branch->token_type == NJS_TOKEN_DEFAULT) { njs_code_set_jump_offset(generator, njs_vmcode_jump_t, jump_offset); jump = NULL; node = branch; @@ -1580,7 +1580,7 @@ njs_generate_stop_statement(njs_vm_t *vm index = NJS_INDEX_NONE; node = node->right; - if (node != NULL && node->token != NJS_TOKEN_FUNCTION) { + if (node != NULL && node->token_type != NJS_TOKEN_FUNCTION) { index = node->index; } @@ -1626,7 +1626,7 @@ njs_generate_assignment(njs_vm_t *vm, nj expr = node->right; expr->dest = NULL; - if (lvalue->token == NJS_TOKEN_NAME) { + if (lvalue->token_type == NJS_TOKEN_NAME) { ret = njs_generate_variable(vm, generator, lvalue, NJS_DECLARATION); if (njs_slow_path(ret != NJS_OK)) { @@ -1679,7 +1679,7 @@ njs_generate_assignment(njs_vm_t *vm, nj * Preserve object and property values stored in variables in a case * if the variables can be changed by side effects in expression. */ - if (object->token == NJS_TOKEN_NAME) { + if (object->token_type == NJS_TOKEN_NAME) { src = object->index; index = njs_generate_node_temp_index_get(vm, generator, object); @@ -1690,7 +1690,7 @@ njs_generate_assignment(njs_vm_t *vm, nj njs_generate_code_move(generator, move, index, src); } - if (property->token == NJS_TOKEN_NAME) { + if (property->token_type == NJS_TOKEN_NAME) { src = property->index; index = njs_generate_node_temp_index_get(vm, generator, property); @@ -1707,7 +1707,7 @@ njs_generate_assignment(njs_vm_t *vm, nj return ret; } - switch (lvalue->token) { + switch (lvalue->token_type) { case NJS_TOKEN_PROPERTY_INIT: njs_generate_code(generator, njs_vmcode_prop_set_t, prop_set, NJS_VMCODE_PROPERTY_INIT, 3); @@ -1749,7 +1749,7 @@ njs_generate_operation_assignment(njs_vm lvalue = node->left; - if (lvalue->token == NJS_TOKEN_NAME) { + if (lvalue->token_type == NJS_TOKEN_NAME) { ret = njs_generate_variable(vm, generator, lvalue, NJS_DECLARATION); if (njs_slow_path(ret != NJS_OK)) { @@ -1912,7 +1912,7 @@ njs_generate_property_accessor(njs_vm_t accessor->value = function->index; accessor->object = object->index; accessor->property = property->index; - accessor->type = (node->token == NJS_TOKEN_PROPERTY_GETTER) + accessor->type = (node->token_type == NJS_TOKEN_PROPERTY_GETTER) ? NJS_OBJECT_PROP_GETTER : NJS_OBJECT_PROP_SETTER; return NJS_OK; @@ -2094,7 +2094,7 @@ njs_generate_3addr_operation(njs_vm_t *v right = node->right; - if (left->token == NJS_TOKEN_NAME) { + if (left->token_type == NJS_TOKEN_NAME) { if (njs_slow_path(njs_parser_has_side_effect(right))) { njs_generate_code(generator, njs_vmcode_move_t, move, @@ -2184,7 +2184,7 @@ njs_generate_typeof_operation(njs_vm_t * expr = node->left; - if (expr->token == NJS_TOKEN_NAME) { + if (expr->token_type == NJS_TOKEN_NAME) { ret = njs_generate_variable(vm, generator, expr, NJS_TYPEOF); if (njs_slow_path(ret != NJS_OK)) { return NJS_ERROR; @@ -2227,7 +2227,7 @@ njs_generate_inc_dec_operation(njs_vm_t lvalue = node->left; - if (lvalue->token == NJS_TOKEN_NAME) { + if (lvalue->token_type == NJS_TOKEN_NAME) { ret = njs_generate_variable(vm, generator, lvalue, NJS_DECLARATION); if (njs_slow_path(ret != NJS_OK)) { @@ -2860,7 +2860,7 @@ njs_generate_try_statement(njs_vm_t *vm, catch_exit_label = undef_label; catch_cont_label = undef_label; - if (node->token == NJS_TOKEN_CATCH) { + if (node->token_type == NJS_TOKEN_CATCH) { /* A "try/catch" case. */ catch_index = njs_variable_index(vm, node->left); diff -r 87d05fb35ff9 -r 112c7f54f402 src/njs_lexer.c --- a/src/njs_lexer.c Wed Feb 26 16:22:10 2020 +0300 +++ b/src/njs_lexer.c Wed Feb 26 16:22:12 2020 +0300 @@ -311,7 +311,7 @@ njs_lexer_init(njs_vm_t *vm, njs_lexer_t } -njs_token_t +njs_token_type_t njs_lexer_token(njs_vm_t *vm, njs_lexer_t *lexer) { njs_lexer_token_t *lt; @@ -319,7 +319,7 @@ njs_lexer_token(njs_vm_t *vm, njs_lexer_ lexer->prev_start = lexer->start; if (lexer->token != NULL) { - lexer->prev_token = lexer->token->type; + lexer->prev_type = lexer->token->type; njs_mp_free(vm->mem_pool, lexer->token); } @@ -336,7 +336,7 @@ njs_lexer_token(njs_vm_t *vm, njs_lexer_ } -njs_token_t +njs_token_type_t njs_lexer_peek_token(njs_vm_t *vm, njs_lexer_t *lexer, size_t offset) { size_t i; diff -r 87d05fb35ff9 -r 112c7f54f402 src/njs_lexer.h --- a/src/njs_lexer.h Wed Feb 26 16:22:10 2020 +0300 +++ b/src/njs_lexer.h Wed Feb 26 16:22:12 2020 +0300 @@ -195,7 +195,7 @@ typedef enum { NJS_TOKEN_SUPER, NJS_TOKEN_RESERVED, -} njs_token_t; +} njs_token_type_t; typedef struct { @@ -205,7 +205,7 @@ typedef struct { typedef struct { njs_lexer_entry_t entry; - njs_token_t type; + njs_token_type_t type; } njs_keyword_t; @@ -219,7 +219,7 @@ typedef struct { typedef struct { - njs_token_t type:16; + njs_token_type_t type:16; uint32_t line; uintptr_t unique_id; njs_str_t text; @@ -234,7 +234,7 @@ typedef struct { uint8_t keyword; u_char *prev_start; - njs_token_t prev_token:16; + njs_token_type_t prev_type:16; uint32_t line; njs_str_t file; @@ -251,8 +251,8 @@ typedef struct { njs_int_t njs_lexer_init(njs_vm_t *vm, njs_lexer_t *lexer, njs_str_t *file, u_char *start, u_char *end); -njs_token_t njs_lexer_token(njs_vm_t *vm, njs_lexer_t *lexer); -njs_token_t njs_lexer_peek_token(njs_vm_t *vm, njs_lexer_t *lexer, +njs_token_type_t njs_lexer_token(njs_vm_t *vm, njs_lexer_t *lexer); +njs_token_type_t njs_lexer_peek_token(njs_vm_t *vm, njs_lexer_t *lexer, size_t offset); njs_int_t njs_lexer_rollback(njs_vm_t *vm, njs_lexer_t *lexer); diff -r 87d05fb35ff9 -r 112c7f54f402 src/njs_module.c --- a/src/njs_module.c Wed Feb 26 16:22:10 2020 +0300 +++ b/src/njs_module.c Wed Feb 26 16:22:12 2020 +0300 @@ -104,7 +104,7 @@ njs_parser_module(njs_vm_t *vm, njs_pars njs_str_t name, text; njs_lexer_t *prev, lexer; njs_module_t *module; - njs_token_t token; + njs_token_type_t type; njs_parser_node_t *node; njs_module_info_t info; @@ -161,13 +161,13 @@ njs_parser_module(njs_vm_t *vm, njs_pars parser->lexer = &lexer; - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { goto fail; } - token = njs_parser_module_lambda(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { + type = njs_parser_module_lambda(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { goto fail; } diff -r 87d05fb35ff9 -r 112c7f54f402 src/njs_parser.c --- a/src/njs_parser.c Wed Feb 26 16:22:10 2020 +0300 +++ b/src/njs_parser.c Wed Feb 26 16:22:12 2020 +0300 @@ -11,55 +11,59 @@ static njs_int_t njs_parser_scope_begin(njs_vm_t *vm, njs_parser_t *parser, njs_scope_t type); static void njs_parser_scope_end(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_statement_chain(njs_vm_t *vm, - njs_parser_t *parser, njs_token_t token, njs_bool_t top); -static njs_token_t njs_parser_statement(njs_vm_t *vm, njs_parser_t *parser, - njs_token_t token); -static njs_token_t njs_parser_block_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_statement_chain(njs_vm_t *vm, + njs_parser_t *parser, njs_token_type_t type, njs_bool_t top); +static njs_token_type_t njs_parser_statement(njs_vm_t *vm, njs_parser_t *parser, + njs_token_type_t type); +static njs_token_type_t njs_parser_block_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_block(njs_vm_t *vm, - njs_parser_t *parser, njs_token_t token); -static njs_token_t njs_parser_labelled_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_block(njs_vm_t *vm, + njs_parser_t *parser, njs_token_type_t type); +static njs_token_type_t njs_parser_labelled_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_function_declaration(njs_vm_t *vm, +static njs_token_type_t njs_parser_function_declaration(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_lambda_arguments(njs_vm_t *vm, +static njs_token_type_t njs_parser_lambda_arguments(njs_vm_t *vm, njs_parser_t *parser, njs_function_lambda_t *lambda, njs_index_t index, - njs_token_t token); -static njs_token_t njs_parser_lambda_argument(njs_vm_t *vm, + njs_token_type_t type); +static njs_token_type_t njs_parser_lambda_argument(njs_vm_t *vm, njs_parser_t *parser, njs_index_t index); -static njs_token_t njs_parser_lambda_body(njs_vm_t *vm, njs_parser_t *parser, - njs_token_t token); +static njs_token_type_t njs_parser_lambda_body(njs_vm_t *vm, + njs_parser_t *parser, njs_token_type_t type); static njs_parser_node_t *njs_parser_return_set(njs_vm_t *vm, njs_parser_t *parser, njs_parser_node_t *expr); -static njs_token_t njs_parser_return_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_return_statement(njs_vm_t *vm, + njs_parser_t *parser); +static njs_token_type_t njs_parser_var_statement(njs_vm_t *vm, + njs_parser_t *parser, njs_token_type_t parent, njs_bool_t var_in); +static njs_token_type_t njs_parser_if_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_var_statement(njs_vm_t *vm, njs_parser_t *parser, - njs_token_t parent, njs_bool_t var_in); -static njs_token_t njs_parser_if_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_switch_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_switch_statement(njs_vm_t *vm, + njs_parser_t *parser); +static njs_token_type_t njs_parser_while_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_while_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_do_while_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_do_while_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_for_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_for_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_var_in_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_var_in_statement(njs_vm_t *vm, njs_parser_t *parser, njs_parser_node_t *name); -static njs_token_t njs_parser_for_in_statement(njs_vm_t *vm, - njs_parser_t *parser, njs_str_t *name, njs_token_t token); -static njs_token_t njs_parser_brk_statement(njs_vm_t *vm, - njs_parser_t *parser, njs_token_t token); -static njs_token_t njs_parser_try_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_try_block(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_throw_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_for_in_statement(njs_vm_t *vm, + njs_parser_t *parser, njs_str_t *name, njs_token_type_t type); +static njs_token_type_t njs_parser_brk_statement(njs_vm_t *vm, + njs_parser_t *parser, njs_token_type_t type); +static njs_token_type_t njs_parser_try_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_import_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_try_block(njs_vm_t *vm, + njs_parser_t *parser); +static njs_token_type_t njs_parser_throw_statement(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_export_statement(njs_vm_t *vm, +static njs_token_type_t njs_parser_import_statement(njs_vm_t *vm, + njs_parser_t *parser); +static njs_token_type_t njs_parser_export_statement(njs_vm_t *vm, njs_parser_t *parser); static njs_int_t njs_parser_export_sink(njs_vm_t *vm, njs_parser_t *parser); -static njs_token_t njs_parser_grouping_expression(njs_vm_t *vm, +static njs_token_type_t njs_parser_grouping_expression(njs_vm_t *vm, njs_parser_t *parser); @@ -77,7 +81,7 @@ njs_int_t njs_parser(njs_vm_t *vm, njs_parser_t *parser, njs_parser_t *prev) { njs_int_t ret; - njs_token_t token; + njs_token_type_t type; njs_parser_node_t *node; ret = njs_parser_scope_begin(vm, parser, NJS_SCOPE_GLOBAL); @@ -97,16 +101,16 @@ njs_parser(njs_vm_t *vm, njs_parser_t *p } } - token = njs_parser_token(vm, parser); - - while (token != NJS_TOKEN_END) { - - token = njs_parser_statement_chain(vm, parser, token, 1); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { + type = njs_parser_token(vm, parser); + + while (type != NJS_TOKEN_END) { + + type = njs_parser_statement_chain(vm, parser, type, 1); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { return NJS_ERROR; } - if (token == NJS_TOKEN_CLOSE_BRACE && vm->options.trailer) { + if (type == NJS_TOKEN_CLOSE_BRACE && vm->options.trailer) { parser->lexer->start--; break; } @@ -125,7 +129,7 @@ njs_parser(njs_vm_t *vm, njs_parser_t *p njs_parser_chain_top_set(parser, node); } - node->token = NJS_TOKEN_END; + node->token_type = NJS_TOKEN_END; if (njs_slow_path(parser->count != 0)) { njs_internal_error(vm, "parser->count != 0"); @@ -264,9 +268,9 @@ njs_parser_scope_rbtree_compare(njs_rbtr } -static njs_token_t +static njs_token_type_t njs_parser_statement_chain(njs_vm_t *vm, njs_parser_t *parser, - njs_token_t token, njs_bool_t top) + njs_token_type_t type, njs_bool_t top) { njs_parser_node_t *stmt, *last, *node, *new_node, **child; @@ -277,17 +281,17 @@ njs_parser_statement_chain(njs_vm_t *vm, njs_parser_enter(vm, parser); - token = njs_parser_statement(vm, parser, token); + type = njs_parser_statement(vm, parser, type); njs_parser_leave(parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return njs_parser_unexpected_token(vm, parser, token); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return njs_parser_unexpected_token(vm, parser, type); } if (parser->node == NULL) { /* The statement is empty block or just semicolon. */ - return token; + return type; } new_node = parser->node; @@ -319,26 +323,26 @@ njs_parser_statement_chain(njs_vm_t *vm, *child = stmt; - while (token == NJS_TOKEN_SEMICOLON) { - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { + while (type == NJS_TOKEN_SEMICOLON) { + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { break; } } - return token; + return type; } -static njs_token_t +static njs_token_type_t njs_parser_statement(njs_vm_t *vm, njs_parser_t *parser, - njs_token_t token) + njs_token_type_t type) { size_t offset; parser->node = NULL; - switch (token) { + switch (type) { case NJS_TOKEN_FUNCTION: return njs_parser_function_declaration(vm, parser); @@ -371,37 +375,37 @@ njs_parser_statement(njs_vm_t *vm, njs_p if (vm->options.trailer) { parser->node = NULL; njs_thread_log_debug("BLOCK END"); - return token; + return type; } /* Fall through. */ default: - switch (token) { + switch (type) { case NJS_TOKEN_VAR: - token = njs_parser_var_statement(vm, parser, token, 0); + type = njs_parser_var_statement(vm, parser, type, 0); break; case NJS_TOKEN_RETURN: - token = njs_parser_return_statement(vm, parser); + type = njs_parser_return_statement(vm, parser); break; case NJS_TOKEN_THROW: - token = njs_parser_throw_statement(vm, parser); + type = njs_parser_throw_statement(vm, parser); break; case NJS_TOKEN_CONTINUE: case NJS_TOKEN_BREAK: - token = njs_parser_brk_statement(vm, parser, token); + type = njs_parser_brk_statement(vm, parser, type); break; case NJS_TOKEN_IMPORT: - token = njs_parser_import_statement(vm, parser); + type = njs_parser_import_statement(vm, parser); break; case NJS_TOKEN_EXPORT: - token = njs_parser_export_statement(vm, parser); + type = njs_parser_export_statement(vm, parser); break; case NJS_TOKEN_NAME: @@ -413,30 +417,30 @@ njs_parser_statement(njs_vm_t *vm, njs_p /* Fall through. */ default: - token = njs_parser_expression(vm, parser, token); + type = njs_parser_expression(vm, parser, type); break; } - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } /* * An expression must be terminated by semicolon, * or by a close curly brace or by the end of line. */ - switch (token) { + switch (type) { case NJS_TOKEN_SEMICOLON: return njs_parser_token(vm, parser); case NJS_TOKEN_CLOSE_BRACE: case NJS_TOKEN_END: - return token; + return type; default: - if (parser->lexer->prev_token == NJS_TOKEN_LINE_END) { - return token; + if (parser->lexer->prev_type == NJS_TOKEN_LINE_END) { + return type; } return NJS_TOKEN_ILLEGAL; @@ -445,16 +449,16 @@ njs_parser_statement(njs_vm_t *vm, njs_p } -static njs_token_t +static njs_token_type_t njs_parser_block_statement(njs_vm_t *vm, njs_parser_t *parser) { njs_int_t ret; - njs_token_t token; + njs_token_type_t type; njs_parser_node_t *node; - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } ret = njs_parser_scope_begin(vm, parser, NJS_SCOPE_BLOCK); @@ -464,10 +468,10 @@ njs_parser_block_statement(njs_vm_t *vm, parser->node = NULL; - while (token != NJS_TOKEN_CLOSE_BRACE) { - token = njs_parser_statement_chain(vm, parser, token, 0); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + while (type != NJS_TOKEN_CLOSE_BRACE) { + type = njs_parser_statement_chain(vm, parser, type, 0); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } } @@ -486,16 +490,16 @@ njs_parser_block_statement(njs_vm_t *vm, } -static njs_token_t -njs_parser_block(njs_vm_t *vm, njs_parser_t *parser, njs_token_t token) +static njs_token_type_t +njs_parser_block(njs_vm_t *vm, njs_parser_t *parser, njs_token_type_t type) { - if (token == NJS_TOKEN_FUNCTION) { + if (type == NJS_TOKEN_FUNCTION) { njs_parser_syntax_error(vm, parser, "Functions can only be declared at top level or inside a block"); return NJS_TOKEN_ILLEGAL; } - return njs_parser_statement(vm, parser, token); + return njs_parser_statement(vm, parser, type); } @@ -540,14 +544,14 @@ njs_parser_variable_node(njs_vm_t *vm, n } -static njs_token_t +static njs_token_type_t njs_parser_labelled_statement(njs_vm_t *vm, njs_parser_t *parser) { - uintptr_t unique_id; - njs_int_t ret; - njs_str_t name; - njs_token_t token; - njs_variable_t *label; + uintptr_t unique_id; + njs_int_t ret; + njs_str_t name; + njs_variable_t *label; + njs_token_type_t type; name = *njs_parser_text(parser); unique_id = njs_parser_key_hash(parser); @@ -564,19 +568,19 @@ njs_parser_labelled_statement(njs_vm_t * return NJS_TOKEN_ERROR; } - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } - token = njs_parser_match(vm, parser, token, NJS_TOKEN_COLON); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_match(vm, parser, type, NJS_TOKEN_COLON); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } - token = njs_parser_statement(vm, parser, token); - - if (njs_fast_path(token > NJS_TOKEN_ILLEGAL)) { + type = njs_parser_statement(vm, parser, type); + + if (njs_fast_path(type > NJS_TOKEN_ILLEGAL)) { if (parser->node != NULL) { /* The statement is not empty block or just semicolon. */ @@ -593,7 +597,7 @@ njs_parser_labelled_statement(njs_vm_t * } } - return token; + return type; } @@ -635,13 +639,13 @@ njs_parser_function_alloc(njs_vm_t *vm, } -static njs_token_t +static njs_token_type_t njs_parser_function_declaration(njs_vm_t *vm, njs_parser_t *parser) { njs_int_t ret; - njs_token_t token; njs_variable_t *var; njs_function_t *function; + njs_token_type_t type; njs_parser_node_t *node; node = njs_parser_node_new(vm, parser, NJS_TOKEN_FUNCTION); @@ -651,13 +655,13 @@ njs_parser_function_declaration(njs_vm_t node->token_line = njs_parser_token_line(parser); - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } - if (token != NJS_TOKEN_NAME) { - if (njs_parser_restricted_identifier(token)) { + if (type != NJS_TOKEN_NAME) { + if (njs_parser_restricted_identifier(type)) { njs_parser_syntax_error(vm, parser, "Identifier \"%V\" " "is forbidden in function declaration", njs_parser_text(parser)); @@ -676,9 +680,9 @@ njs_parser_function_declaration(njs_vm_t return NJS_TOKEN_ERROR; } - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } parser->node = node; @@ -688,22 +692,22 @@ njs_parser_function_declaration(njs_vm_t return NJS_TOKEN_ERROR; } - token = njs_parser_function_lambda(vm, parser, function->u.lambda, token); + type = njs_parser_function_lambda(vm, parser, function->u.lambda, type); function->args_count = function->u.lambda->nargs - function->u.lambda->rest_parameters; - return token; + return type; } -njs_token_t +njs_token_type_t njs_parser_function_expression(njs_vm_t *vm, njs_parser_t *parser) { njs_int_t ret; - njs_token_t token; njs_variable_t *var; njs_function_t *function; + njs_token_type_t type; njs_parser_node_t *node; njs_function_lambda_t *lambda; @@ -715,9 +719,9 @@ njs_parser_function_expression(njs_vm_t node->token_line = njs_parser_token_line(parser); parser->node = node; - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } /* @@ -729,15 +733,15 @@ njs_parser_function_expression(njs_vm_t return NJS_TOKEN_ERROR; } - if (token == NJS_TOKEN_NAME) { + if (type == NJS_TOKEN_NAME) { var = njs_parser_variable_add(vm, parser, NJS_VARIABLE_SHIM); if (njs_slow_path(var == NULL)) { return NJS_TOKEN_ERROR; } - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } function = njs_parser_function_alloc(vm, parser, var); @@ -757,17 +761,17 @@ njs_parser_function_expression(njs_vm_t node->u.value.data.u.lambda = lambda; - token = njs_parser_function_lambda(vm, parser, lambda, token); + type = njs_parser_function_lambda(vm, parser, lambda, type); njs_parser_scope_end(vm, parser); - return token; + return type; } -njs_token_t +njs_token_type_t njs_parser_function_lambda(njs_vm_t *vm, njs_parser_t *parser, - njs_function_lambda_t *lambda, njs_token_t token) + njs_function_lambda_t *lambda, njs_token_type_t type) { njs_int_t ret; njs_index_t index; @@ -782,61 +786,61 @@ njs_parser_function_lambda(njs_vm_t *vm, /* A "this" reservation. */ index += sizeof(njs_value_t); - token = njs_parser_lambda_arguments(vm, parser, lambda, index, token); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_lambda_arguments(vm, parser, lambda, index, type); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } - token = njs_parser_lambda_body(vm, parser, token); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_lambda_body(vm, parser, type); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } njs_parser_scope_end(vm, parser); - return token; + return type; } -static njs_token_t +static njs_token_type_t njs_parser_lambda_arguments(njs_vm_t *vm, njs_parser_t *parser, - njs_function_lambda_t *lambda, njs_index_t index, njs_token_t token) + njs_function_lambda_t *lambda, njs_index_t index, njs_token_type_t type) { - token = njs_parser_match(vm, parser, token, NJS_TOKEN_OPEN_PARENTHESIS); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_match(vm, parser, type, NJS_TOKEN_OPEN_PARENTHESIS); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } lambda->nargs = 0; - while (token != NJS_TOKEN_CLOSE_PARENTHESIS) { + while (type != NJS_TOKEN_CLOSE_PARENTHESIS) { if (njs_slow_path(lambda->rest_parameters)) { return NJS_TOKEN_ILLEGAL; } - if (njs_slow_path(token == NJS_TOKEN_ELLIPSIS)) { + if (njs_slow_path(type == NJS_TOKEN_ELLIPSIS)) { lambda->rest_parameters = 1; - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { return NJS_TOKEN_ILLEGAL; } } - if (njs_slow_path(token != NJS_TOKEN_NAME)) { + if (njs_slow_path(type != NJS_TOKEN_NAME)) { return NJS_TOKEN_ILLEGAL; } - token = njs_parser_lambda_argument(vm, parser, index); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_lambda_argument(vm, parser, index); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } - if (token == NJS_TOKEN_COMMA) { - token = njs_parser_token(vm, parser); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + if (type == NJS_TOKEN_COMMA) { + type = njs_parser_token(vm, parser); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } } @@ -848,7 +852,7 @@ njs_parser_lambda_arguments(njs_vm_t *vm } -static njs_token_t +static njs_token_type_t njs_parser_lambda_argument(njs_vm_t *vm, njs_parser_t *parser, njs_index_t index) { @@ -872,16 +876,17 @@ njs_parser_lambda_argument(njs_vm_t *vm, } -static njs_token_t -njs_parser_lambda_body(njs_vm_t *vm, njs_parser_t *parser, njs_token_t token) +static njs_token_type_t +njs_parser_lambda_body(njs_vm_t *vm, njs_parser_t *parser, + njs_token_type_t type) { njs_parser_node_t *body, *last, *parent; parent = parser->node; - token = njs_parser_lambda_statements(vm, parser, token); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_lambda_statements(vm, parser, type); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } last = NULL; @@ -900,7 +905,7 @@ njs_parser_lambda_body(njs_vm_t *vm, njs } } - if (last == NULL || last->token != NJS_TOKEN_RETURN) { + if (last == NULL || last->token_type != NJS_TOKEN_RETURN) { /* * There is no function body or the last function body * body statement is not "return" statement. @@ -915,7 +920,7 @@ njs_parser_lambda_body(njs_vm_t *vm, njs parser->node = parent; - return token; + return type; } @@ -946,21 +951,21 @@ njs_parser_return_set(njs_vm_t *vm, njs_ } -njs_token_t +njs_token_type_t njs_parser_lambda_statements(njs_vm_t *vm, njs_parser_t *parser, - njs_token_t token) + njs_token_type_t type) { - token = njs_parser_match(vm, parser, token, NJS_TOKEN_OPEN_BRACE); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_match(vm, parser, type, NJS_TOKEN_OPEN_BRACE); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } parser->node = NULL; - while (token != NJS_TOKEN_CLOSE_BRACE) { - token = njs_parser_statement_chain(vm, parser, token, 1); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + while (type != NJS_TOKEN_CLOSE_BRACE) { + type = njs_parser_statement_chain(vm, parser, type, 1); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } } @@ -968,10 +973,10 @@ njs_parser_lambda_statements(njs_vm_t *v } -static njs_token_t +static njs_token_type_t njs_parser_return_statement(njs_vm_t *vm, njs_parser_t *parser) { - njs_token_t token; + njs_token_type_t type; njs_parser_node_t *node; njs_parser_scope_t *scope; @@ -997,12 +1002,12 @@ njs_parser_return_statement(njs_vm_t *vm parser->node = node; - token = njs_lexer_token(vm, parser->lexer); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_lexer_token(vm, parser->lexer); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } - switch (token) { + switch (type) { case NJS_TOKEN_LINE_END: return njs_parser_token(vm, parser); @@ -1010,31 +1015,31 @@ njs_parser_return_statement(njs_vm_t *vm case NJS_TOKEN_SEMICOLON: case NJS_TOKEN_CLOSE_BRACE: case NJS_TOKEN_END: - return token; + return type; default: - token = njs_parser_expression(vm, parser, token); - if (njs_slow_path(token <= NJS_TOKEN_ILLEGAL)) { - return token; + type = njs_parser_expression(vm, parser, type); + if (njs_slow_path(type <= NJS_TOKEN_ILLEGAL)) { + return type; } - if (parser->node->token == NJS_TOKEN_FUNCTION) { + if (parser->node->token_type == NJS_TOKEN_FUNCTION) { From xeioex at nginx.com Wed Feb 26 14:49:01 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 26 Feb 2020 14:49:01 +0000 Subject: [njs] Fixed JSON.stringify() with Number() and String() objects. Message-ID: details: https://hg.nginx.org/njs/rev/061afad25256 branches: changeset: 1338:061afad25256 user: Alexander Borisov date: Tue Feb 25 16:03:20 2020 +0300 description: Fixed JSON.stringify() with Number() and String() objects. According to the specification. diffstat: src/njs_json.c | 136 ++++++++++++++++++++++++++++++---------------- src/test/njs_unit_test.c | 58 +++++++++++++++---- 2 files changed, 133 insertions(+), 61 deletions(-) diffs (318 lines): diff -r 112c7f54f402 -r 061afad25256 src/njs_json.c --- a/src/njs_json.c Wed Feb 26 16:22:12 2020 +0300 +++ b/src/njs_json.c Tue Feb 25 16:03:20 2020 +0300 @@ -94,7 +94,8 @@ static njs_int_t njs_json_stringify_repl static njs_int_t njs_json_stringify_array(njs_vm_t *vm, njs_json_stringify_t *stringify); -static void njs_json_append_value(njs_chb_t *chain, const njs_value_t *value); +static njs_int_t njs_json_append_value(njs_vm_t *vm, njs_chb_t *chain, + njs_value_t *value); static void njs_json_append_string(njs_chb_t *chain, const njs_value_t *value, char quote); static void njs_json_append_number(njs_chb_t *chain, const njs_value_t *value); @@ -188,7 +189,7 @@ njs_json_stringify(njs_vm_t *vm, njs_val njs_index_t unused) { size_t length; - double num; + int64_t i64; njs_int_t i; njs_int_t ret; njs_value_t *replacer, *space; @@ -217,45 +218,67 @@ njs_json_stringify(njs_vm_t *vm, njs_val njs_set_undefined(&stringify->replacer); } - stringify->space.length = 0; - space = njs_arg(args, nargs, 3); - if (njs_is_string(space) || njs_is_number(space)) { - if (njs_is_string(space)) { - length = njs_string_prop(&prop, space); - - if (njs_is_byte_string(&prop)) { - njs_internal_error(vm, "space argument cannot be" - " a byte string"); - return NJS_ERROR; - } - - if (length > 10) { - p = njs_string_offset(prop.start, prop.start + prop.size, 10); - - } else { - p = prop.start + prop.size; - } - - stringify->space.start = prop.start; - stringify->space.length = p - prop.start; + switch (space->type) { + case NJS_OBJECT_STRING: + ret = njs_value_to_string(vm, space, space); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + /* Fall through. */ + + case NJS_STRING: + length = njs_string_prop(&prop, space); + + if (njs_is_byte_string(&prop)) { + njs_internal_error(vm, "space argument cannot be" + " a byte string"); + return NJS_ERROR; + } + + if (length > 10) { + p = njs_string_offset(prop.start, prop.start + prop.size, 10); } else { - num = njs_number(space); - - if (!isnan(num) && !isinf(num) && num > 0) { - num = njs_min(num, 10); - - stringify->space.length = (size_t) num; - stringify->space.start = stringify->space_buf; - - for (i = 0; i < (int) num; i++) { - stringify->space.start[i] = ' '; - } + p = prop.start + prop.size; + } + + stringify->space.start = prop.start; + stringify->space.length = p - prop.start; + + break; + + case NJS_OBJECT_NUMBER: + ret = njs_value_to_numeric(vm, space, space); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + + /* Fall through. */ + + case NJS_NUMBER: + i64 = njs_min(njs_number_to_integer(njs_number(space)), 10); + + if (i64 > 0) { + stringify->space.length = i64; + stringify->space.start = stringify->space_buf; + + for (i = 0; i < i64; i++) { + stringify->space.start[i] = ' '; } + + break; } - } + + /* Fall through. */ + + default: + stringify->space.length = 0; + + break; + } return njs_json_stringify_iterator(vm, stringify, njs_arg(args, nargs, 1)); @@ -1238,7 +1261,10 @@ njs_json_stringify_iterator(njs_vm_t *vm break; } - njs_json_append_value(&chain, value); + ret = njs_json_append_value(vm, &chain, value); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } break; @@ -1288,7 +1314,10 @@ njs_json_stringify_iterator(njs_vm_t *vm } state->written = 1; - njs_json_append_value(&chain, value); + ret = njs_json_append_value(vm, &chain, value); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } break; } @@ -1471,10 +1500,6 @@ njs_json_stringify_array(njs_vm_t *vm, n } switch (value->type) { - case NJS_OBJECT_NUMBER: - value = njs_object_value(value); - /* Fall through. */ - case NJS_NUMBER: ret = njs_number_to_string(vm, &num_value, value); if (njs_slow_path(ret != NJS_OK)) { @@ -1484,9 +1509,14 @@ njs_json_stringify_array(njs_vm_t *vm, n value = &num_value; break; + case NJS_OBJECT_NUMBER: case NJS_OBJECT_STRING: - value = njs_object_value(value); - break; + ret = njs_value_to_string(vm, value, value); + if (njs_slow_path(ret != NJS_OK)) { + return NJS_ERROR; + } + + /* Fall through. */ case NJS_STRING: break; @@ -1513,12 +1543,18 @@ njs_json_stringify_array(njs_vm_t *vm, n } -static void -njs_json_append_value(njs_chb_t *chain, const njs_value_t *value) +static njs_int_t +njs_json_append_value(njs_vm_t *vm, njs_chb_t *chain, njs_value_t *value) { + njs_int_t ret; + switch (value->type) { case NJS_OBJECT_STRING: - value = njs_object_value(value); + ret = njs_value_to_string(vm, value, value); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + /* Fall through. */ case NJS_STRING: @@ -1526,7 +1562,11 @@ njs_json_append_value(njs_chb_t *chain, break; case NJS_OBJECT_NUMBER: - value = njs_object_value(value); + ret = njs_value_to_numeric(vm, value, value); + if (njs_slow_path(ret != NJS_OK)) { + return ret; + } + /* Fall through. */ case NJS_NUMBER: @@ -1555,6 +1595,8 @@ njs_json_append_value(njs_chb_t *chain, default: njs_chb_append_literal(chain, "null"); } + + return NJS_OK; } diff -r 112c7f54f402 -r 061afad25256 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Wed Feb 26 16:22:12 2020 +0300 +++ b/src/test/njs_unit_test.c Tue Feb 25 16:03:20 2020 +0300 @@ -15454,6 +15454,10 @@ static njs_unit_test_t njs_test[] = { njs_str("JSON.stringify(new String('abc'))"), njs_str("\"abc\"") }, + { njs_str("var s = new String('abc'); s.toString = () => 'xxx'; " + "JSON.stringify(s)"), + njs_str("\"xxx\"") }, + { njs_str("JSON.stringify(123)"), njs_str("123") }, @@ -15466,6 +15470,10 @@ static njs_unit_test_t njs_test[] = { njs_str("JSON.stringify(new Number(123))"), njs_str("123") }, + { njs_str("var n = new Number(8.5); n.valueOf = () => 42;" + "JSON.stringify(n)"), + njs_str("42") }, + { njs_str("JSON.stringify(true)"), njs_str("true") }, @@ -15662,30 +15670,44 @@ static njs_unit_test_t njs_test[] = { njs_str("JSON.stringify([{a:1,b:{c:2}},1], undefined, '#')"), njs_str("[\n#{\n##\"a\": 1,\n##\"b\": {\n###\"c\": 2\n##}\n#},\n#1\n]") }, - { njs_str("JSON.stringify([1], undefined, 'AAAAABBBBBC')"), + { njs_str("JSON.stringify([1], null, 'AAAAABBBBBC')"), + njs_str("[\nAAAAABBBBB1\n]") }, + + { njs_str("var s = new String('A'); s.toString = () => 'AAAAABBBBBC';" + "JSON.stringify([1], null, s)"), njs_str("[\nAAAAABBBBB1\n]") }, - { njs_str("JSON.stringify([1], undefined, 11)"), + { njs_str("JSON.stringify([1], null, '!?????').length"), + njs_str("11") }, + + { njs_str("JSON.stringify([1], null, 'ABC') === JSON.stringify([1], null, new String('ABC'))"), + njs_str("true") }, + + { njs_str("JSON.stringify([1], null, '!!?????????????????').length"), + njs_str("15") }, + + { njs_str("JSON.stringify([1], null, String.bytesFrom([0x9d])).length"), + njs_str("InternalError: space argument cannot be a byte string") }, + + { njs_str("JSON.stringify([1], null, 11)"), njs_str("[\n 1\n]") }, + { njs_str("JSON.stringify([1], null, 5) === JSON.stringify([1], null, 5.9)"), + njs_str("true") }, + + { njs_str("JSON.stringify([1], null, 5) === JSON.stringify([1], null, new Number(5))"), + njs_str("true") }, + + { njs_str("var s = new Number(23); s.valueOf = () => 5;" + "JSON.stringify([1], null, s)"), + njs_str("[\n 1\n]") }, + { njs_str("JSON.stringify([{a:1,b:{c:2}},1], undefined, -1)"), njs_str("[{\"a\":1,\"b\":{\"c\":2}},1]") }, { njs_str("JSON.stringify([{a:1,b:{c:2}},1], undefined, new Date())"), njs_str("[{\"a\":1,\"b\":{\"c\":2}},1]") }, - { njs_str("JSON.stringify([], null, '!?????').length"), - njs_str("10") }, - - { njs_str("JSON.stringify([], null, '!!?????????????????').length"), - njs_str("14") }, - - { njs_str("JSON.stringify([], null, '!?????????????????').length"), - njs_str("14") }, - - { njs_str("JSON.stringify([], null, String.bytesFrom([0x9d])).length"), - njs_str("InternalError: space argument cannot be a byte string") }, - { njs_str("var o = Object.defineProperty({}, 'a', { get() { return ()=> 1}, enumerable: true });" "JSON.stringify(o)"), njs_str("{}") }, @@ -15806,6 +15828,14 @@ static njs_unit_test_t njs_test[] = { njs_str("JSON.stringify({'1':1,'2':2,'3':3}, [1, new Number(2)])"), njs_str("{\"1\":1,\"2\":2}") }, + { njs_str("var s = new String('str'); s.toString = () => 'xxx';" + "JSON.stringify({str:1,xxx:2}, [s])"), + njs_str("{\"xxx\":2}") }, + + { njs_str("var n = new String(123); n.toString = () => '42';" + "JSON.stringify({123:1,42:2}, [n])"), + njs_str("{\"42\":2}") }, + { njs_str("var objs = []; var o = JSON.stringify({a:1}," " function(k, v) {objs.push(this); return v});" "JSON.stringify(objs)"), From xeioex at nginx.com Wed Feb 26 14:49:03 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 26 Feb 2020 14:49:03 +0000 Subject: [njs] Fixed Unicode Escaping in JSON.stringify() according to spec. Message-ID: details: https://hg.nginx.org/njs/rev/6444ae6ab276 branches: changeset: 1339:6444ae6ab276 user: Dmitry Volyntsev date: Wed Feb 26 17:48:46 2020 +0300 description: Fixed Unicode Escaping in JSON.stringify() according to spec. Lowecase hexadecimal number are required. diffstat: src/njs_json.c | 4 ++-- src/test/njs_unit_test.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diffs (33 lines): diff -r 061afad25256 -r 6444ae6ab276 src/njs_json.c --- a/src/njs_json.c Tue Feb 25 16:03:20 2020 +0300 +++ b/src/njs_json.c Wed Feb 26 17:48:46 2020 +0300 @@ -1610,7 +1610,7 @@ njs_json_append_string(njs_chb_t *chain, njs_string_prop_t string; static char hex2char[16] = { '0', '1', '2', '3', '4', '5', '6', '7', - '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' }; + '8', '9', 'a', 'b', 'c', 'd', 'e', 'f' }; (void) njs_string_prop(&string, value); @@ -1619,7 +1619,7 @@ njs_json_append_string(njs_chb_t *chain, utf8 = (string.length != 0 && string.length != string.size); size = njs_max(string.size + 2, 7); - dst = njs_chb_reserve(chain, string.size + 2); + dst = njs_chb_reserve(chain, size); if (njs_slow_path(dst == NULL)) { return; } diff -r 061afad25256 -r 6444ae6ab276 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Tue Feb 25 16:03:20 2020 +0300 +++ b/src/test/njs_unit_test.c Wed Feb 26 17:48:46 2020 +0300 @@ -15612,7 +15612,7 @@ static njs_unit_test_t njs_test[] = njs_str("\"\\n\\t\\r\\\"\\f\\b\"") }, { njs_str("JSON.stringify('\x00\x01\x02\x1f')"), - njs_str("\"\\u0000\\u0001\\u0002\\u001F\"") }, + njs_str("\"\\u0000\\u0001\\u0002\\u001f\"") }, { njs_str("JSON.stringify('abc\x00')"), njs_str("\"abc\\u0000\"") }, From yshpakov at hotmail.com Wed Feb 26 20:36:40 2020 From: yshpakov at hotmail.com (Yury Shpakov) Date: Wed, 26 Feb 2020 20:36:40 +0000 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> , <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> , , , Message-ID: Hi Sergey, I added couple lines in my config and re-ran nginx: === === === worker_processes 1; events { worker_connections 1024; } #error_log logs/error-mail.log debug; mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; smtp_auth none; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } http { limit_req_zone $binary_remote_addr zone=ip:10m rate=5r/s; server { listen 9000; listen [::]:9000 ipv6only=on; location /cgi-bin/nginxauth.cgi { limit_req zone=ip burst=12 delay=8; add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } === === === My C# test code is very simple: using (var smtpClient = new SmtpClient("localhost", 8025) {Timeout = 60 * 60 * 1000}) { for (int i = 0; i < 1000; i++) { smtpClient.Send("noreply at wmata.com", "ys at wmata.com", "Email subject", "Email body"); } } And with no delays or failures upstream (Fake) SMTP Server receives all 1000 emails (actually within 3 seconds): [cid:5540ee21-0775-49e2-91b6-d9aa695bcbe6] Received from 03:22:16 to 03:22:19 all 1000. So latency is not working. Maybe I missed some step? Maybe I needed to add module ngx_http_limit_req_module and recompile nginx? But if there is no such module in nginx, I would see configuration errors (like with this typo in config): [cid:1bbedf2c-c4cd-4b45-b895-cbbed2dff658] Regards, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 26, 2020 4:50 AM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 There are several possibilities to introduce a latency in nginx: - limit_req - https://www.nginx.com/blog/rate-limiting-nginx/#Two-Stage-Rate-Limiting - Maxim's ngx_http_delay (I used it more for development purposes, like test or simulation of load etc); - some "slow" upstream backend that doing nothing, just waiting (preferably asynchronous). You seems to have some upstream (php?) serving auth_http requests, so you could for example implement some delay in case of failed attempt within php (or whatever you use there as backend). Note that it is always good if the latency will be implemented asynchronously (without a real "sleep") in order to avoid possible overload under DDoS similar circumstances. Regards, Sergey. Am 26.02.2020 01:59, schrieb Yury Shpakov: Hi Sergey, You mentioned that you can set up some delays in responses. How can I do it? Adding this module during compilation? https://github.com/openresty/echo-nginx-module I tried but it didn't want to compile. I got many compilation errors. Maybe I can set up delays somehow else? Thank you, Yury ________________________________ From: nginx-devel on behalf of Yury Shpakov Sent: Friday, February 14, 2020 6:08 PM To: Sergey Brester Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 So what is the meaning of Auth-Server and Auth-Port headers? So it's relevant only when nginx works as SMTP Proxy (not SMTP Server)? And these are host/port where to redirect SMTP requests? Yeah, I was all the time surprised -- how come, it's set as Proxy but there is no setting where it redirects SMTP communication to. A little bit unexpected place for those setting. Well, let me try... I ran Fake SMTP Server on port 25.(I found on Internet some fake SMTP Server). I configured my test SMTP client to localhost:25 (later to 127.0.0.1:25). They send/receive successfully. So both SMTP Client and (fake) SMTP Server work fine. 127.0.0.1 works fine too. I re-configured my test SMTP client to localhost:8025 (tried 127.0.0.1:8025 too). As well, I changed this section of config as follows: http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } The same error: 2020/02/14 17:37:18 [error] 15260#3328: *5 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Update: Detailed logging with debug information helped a lot. This is what I noticed in there: 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp auth state 2020/02/14 17:40:28 [debug] 3940#22096: *1 WSARecv: fd:584 rc:0 24 of 4096 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp rcpt to:"RCPT TO:" 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer del: 584: 1172123084 2020/02/14 17:40:28 [debug] 3940#22096: *1 malloc: 02F8C260:2048 2020/02/14 17:40:28 [debug] 3940#22096: *1 stream socket 588 2020/02/14 17:40:28 [debug] 3940#22096: *1 connect to [::1]:9000, fd:588 #2 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:768 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:16 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 So it's trying to use IP6 rather than IP4. And below: 2020/02/14 17:40:29 [debug] 3940#22096: *1 delete posted event 03171170 2020/02/14 17:40:29 [debug] 3940#22096: *1 mail auth http write handler 2020/02/14 17:40:29 [debug] 3940#22096: *1 WSASend: fd:588, -1, 0 of 306 2020/02/14 17:40:29 [error] 3940#22096: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 So, I replaced localhost with 127.0.0.1 like this: auth_http 127.0.0.1:9000/cgi-bin/nginxauth.cgi; And it worked. Since I forced it to use IP4. Any idea how to use host name instead of IP address and still have it working? Update 2: I figured it out. Googled a little bit and ended up with the following change to my config: http { server { listen 9000; listen [::]:9000 ipv6only=on; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } Now it works. But why just "listen 9000" doesn't listen on both IP4 and IP6? Is it a bug? Yury ________________________________ From: Sergey Brester Sent: Friday, February 14, 2020 5:59 AM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I don't know what is wrong with your config... I guess your smtp server does not answer properly. Is 127.0.0.2:143 really your SMTP-server? Because port 143 is mostly an IMAP port - but you've specified protocol smtp in the server section. Anyway I tested your config with my settings (replaced name and smtp-server, here 192.0.2.222:25) and enabled debug: + error_log logs/error-mail.log debug; mail { - server_name localhost; + server_name example.com; ... http { ... - add_header Auth-Server 127.0.0.2; - add_header Auth-Port 143; + add_header Auth-Server 192.0.2.222; + add_header Auth-Port 25; it works well - I see the test incoming mail (I send to myself via 8025 port) and following output in the log (a lot of irrelevant messages are removed): >>>>>>>>> 2020/02/14 11:24:04 [debug] 121280#128244: *1 smtp mail from:"mail FROM:" ... 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request line: "GET /cgi-bin/nginxauth.cgi HTTP/1.0" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http uri: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http args: "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http exten: "cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http process request header line 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Host: localhost" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Method: none" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-User: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Pass: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Protocol: smtp" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Login-Attempt: 1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-IP: 127.0.0.1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-Host: [UNAVAILABLE]" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-Helo: myhost.example.com" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-From: mail FROM:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-To: rcpt TO:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header done 2020/02/14 11:24:04 [debug] 121280#128244: *3 event timer del: 512: 1127939767 2020/02/14 11:24:04 [debug] 121280#128244: *3 generic phase: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 1 2020/02/14 11:24:04 [debug] 121280#128244: *3 search through nested static locations of "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 test location: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 using configuration "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http cl:-1 max:1048576 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 3 2020/02/14 11:24:04 [debug] 121280#128244: *3 http set discard body 2020/02/14 11:24:04 [debug] 121280#128244: *3 HTTP/1.1 204 No Content Server: nginx/1.17.4 Date: Fri, 14 Feb 2020 10:24:04 GMT Connection: close Auth-Status: OK Auth-Server: 192.0.2.222 Auth-Port: 25 2020/02/14 11:24:04 [debug] 121280#128244: *3 write new buf t:1 f:0 008AD6A0, pos 008AD6A0, size: 164 file: 0, size: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter: l:1 f:0 s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter limit 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 WSASend: fd:512, s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter 00000000 2020/02/14 11:24:04 [debug] 121280#128244: *3 http finalize request: 0, "/cgi-bin/nginxauth.cgi?" a:1, c:1 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request count:1 blk:0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http close request 2020/02/14 11:24:04 [debug] 121280#128244: *3 http log handler 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008ACC50, unused: 1161 2020/02/14 11:24:04 [debug] 121280#128244: *3 close http connection: 512 2020/02/14 11:24:04 [debug] 121280#128244: *3 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008AC848 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 0039FDE0, unused: 28 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http read handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 164 of 1024 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process status line 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process headers 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Server: nginx/1.17.4" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Date: Fri, 14 Feb 2020 10:24:04 GMT" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Connection: close" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Status: OK" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Server: 192.0.2.222" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Port: 25" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header done 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer del: 496: 1127939764 2020/02/14 11:24:04 [debug] 121280#128244: *1 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *1 free: 008AC040, unused: 196 2020/02/14 11:24:04 [debug] 121280#128244: *1 stream socket 496 2020/02/14 11:24:04 [debug] 121280#128244: *1 connect to 192.0.2.222:25, fd:496 #4 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer add: 496: 60000:1127939769 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 008AC040:4096 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 post event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 delete posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy dummy handler 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 22 of 4096 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy send ehlo 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 0039FDE0:256 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSASend: fd:496, 0, 25 of 25 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 196 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send mail from 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 47 of 47 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 60 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send rcpt to 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 45 of 45 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 63 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer add: 492: 86400000:1214280441 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer del: 496: 1127939769 2020/02/14 11:24:05 [info] 121280#128244: *1 client logged in, client: 127.0.0.1, server: 0.0.0.0:8025 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 1, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 63 of 63 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 6 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 6 of 6 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 50 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 50 of 50 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 170 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 170 of 170 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280535 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 56 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 56 of 56 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle ... <<<<<<<<< Regards, Sergey 13.02.2020 22:45, Yury Shpakov wrote: Hi Sergey, I reconfigured the config file as follows: === === === #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.2; # backend ip add_header Auth-Port 143; # backend port return 204; } } } === === === And now it's responding on port 9000 as expected: === === === C:\WINDOWS\system32>curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi HTTP/1.1 204 No Content Server: nginx/1.17.9 Date: Thu, 13 Feb 2020 21:30:54 GMT Connection: keep-alive Auth-Status: OK Auth-Server: 127.0.0.2 Auth-Port: 143 === === === However I'm still experiencing the same issue (in log file): === === === 2020/02/13 16:29:24 [notice] 35048#26192: signal process started 2020/02/13 16:29:34 [error] 31732#22720: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 === === === Tried under both admin and regular user. Any further ideas how to get it fixed please? Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 1:51 PM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I answered inline... 12.02.2020 18:59, Yury Shpakov wrote: Hi Sergey, Thank you for you response. I tried netstat /nabo and I don't see any reference to port 9000 at all. So a problem is to make nginx to listen on port 9000 (as server)? Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ for more examples. Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy - you can use some nginx location (and internal URL to same nginx instance) to specify that. Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link) But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 7:38 AM To: nginx-devel at nginx.org Cc: Yury Shpakov Subject: Re: nginx for Windows - WSASend() socket error 10057 It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... try netstat (in cmd as admin): netstat /nabo netstat /nabo | grep -A 1 ":9000\b" and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). additionally try to telnet or curl it: curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html). But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH - nginx.zip, where it works well). Regards, Sergey 12.02.2020 03:01, Yury Shpakov wrote: Hi there, Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: http://nginx.org/en/docs/howto_build_on_win32.html But excluded (don't care about SSL at this point so don't want to install/configure Perl now): --with-openssl=objs/lib/openssl-master \ --with-openssl-opt=no-asm \ --with-http_ssl_module \ And added: --with-mail nmake was successful and nginx.exe was created. However nginx.exe keeps failing with the error: WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Windows API says the following about this error: WSAENOTCONN 10057 Socket is not connected. A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using sendto) no address was supplied. Any other type of operation might also return this error?for example, setsockopt setting SO_KEEPALIVE if the connection has been reset. https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. docs.microsoft.com Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. Any recommendation for me to make it work? Tried to play with config (commenting/uncommenting): #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. Thank you, Yury _______________________________________________ nginx-devel mailing list nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 42105 bytes Desc: image.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 9367 bytes Desc: image.png URL: From thibaultcha at fastmail.com Thu Feb 27 00:49:55 2020 From: thibaultcha at fastmail.com (Thibault Charbonnier) Date: Wed, 26 Feb 2020 16:49:55 -0800 Subject: [PATCH] Ensured SIGQUIT deletes listening UNIX socket files. Message-ID: <4b6bc48b-6c7a-7c76-51ae-1038421ed36c@fastmail.com> # HG changeset patch # User Thibault Charbonnier # Date 1582764433 28800 # Wed Feb 26 16:47:13 2020 -0800 # Node ID 55ea1a9197a6f28d4da00909e5ea8585f6a08239 # Parent 4f18393a1d51bce6103ea2f1b2587900f349ba3d Ensured SIGQUIT deletes listening UNIX socket files. Prior to this patch, the SIGQUIT signal handling (graceful shutdown) did not remove UNIX socket files since ngx_master_process_cycle reimplemented listening socket closings in lieu of using ngx_close_listening_sockets. Since ngx_master_process_exit will call the aforementioned ngx_close_listening_sockets, we can remove the custom implementation and now expect listening sockets to be closed properly by ngx_close_listening_sockets instead. This fixes the trac issue #753 (https://trac.nginx.org/nginx/ticket/753). diff -r 4f18393a1d51 -r 55ea1a9197a6 src/os/unix/ngx_process_cycle.c --- a/src/os/unix/ngx_process_cycle.c Thu Feb 20 16:51:07 2020 +0300 +++ b/src/os/unix/ngx_process_cycle.c Wed Feb 26 16:47:13 2020 -0800 @@ -77,12 +77,11 @@ u_char *p; size_t size; ngx_int_t i; - ngx_uint_t n, sigio; + ngx_uint_t sigio; sigset_t set; struct itimerval itv; ngx_uint_t live; ngx_msec_t delay; - ngx_listening_t *ls; ngx_core_conf_t *ccf; sigemptyset(&set); @@ -205,16 +204,6 @@ ngx_signal_worker_processes(cycle, ngx_signal_value(NGX_SHUTDOWN_SIGNAL)); - ls = cycle->listening.elts; - for (n = 0; n < cycle->listening.nelts; n++) { - if (ngx_close_socket(ls[n].fd) == -1) { - ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_socket_errno, - ngx_close_socket_n " %V failed", - &ls[n].addr_text); - } - } - cycle->listening.nelts = 0; - continue; } From serg.brester at sebres.de Thu Feb 27 10:20:41 2020 From: serg.brester at sebres.de (Sergey Brester) Date: Thu, 27 Feb 2020 11:20:41 +0100 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> , <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> , , , Message-ID: <20891f45d8b2d29e42b9660acba9ddba@sebres.de> Hmmm... I could imagine that this doesn't work as expected because mail::auth_http simply makes fewer requests as you may assume (due to keep-alive + some internal "cache" for established connections). I have no time to trace it right now (you can enable debug and look how often /cgi-bin/nginxauth.cgi will be called internally). Normally it would be more proper to place limit_req into mail/server sections, but I'm pretty sure you would get something like "limit_req directive is not allowed here". One should extend this module to allow that in mail/server sections. So it looks like this method is not suitable for you at the moment. @nginx-devel: I don't see a troubles to extend directives of "ngx_http_limit_req_module" to consider mail/server too. Are there some objections against that? Regards, Serg. Am 26.02.2020 21:36, schrieb Yury Shpakov: > Hi Sergey, > > I added couple lines in my config and re-ran nginx: > === === === > > worker_processes 1; > > events { > worker_connections 1024; > } > > #error_log logs/error-mail.log debug; > > mail { > server_name localhost; > auth_http localhost:9000/cgi-bin/nginxauth.cgi; > > smtp_auth none; > xclient off; > > server { > listen 8025; > protocol smtp; > proxy on; > proxy_pass_error_message on; > } > } > > http { > limit_req_zone $binary_remote_addr zone=ip:10m rate=5r/s; > server { > listen 9000; > listen [::]:9000 ipv6only=on; > > location /cgi-bin/nginxauth.cgi { > limit_req zone=ip burst=12 delay=8; > add_header Auth-Status OK; > add_header Auth-Server 127.0.0.1; # backend ip > add_header Auth-Port 25; # backend port > return 204; > } > } > } > === === === > > My C# test code is very simple: > > using (var smtpClient = new SmtpClient("localhost", 8025) {Timeout = 60 * 60 * 1000}) > { > for (int i = 0; i < 1000; i++) > { > smtpClient.Send("noreply at wmata.com", "ys at wmata.com", "Email subject", "Email body"); > } > } > > And with no delays or failures upstream (Fake) SMTP Server receives all 1000 emails (actually within 3 seconds): > > Received from 03:22:16 to 03:22:19 all 1000. > > So latency is not working. Maybe I missed some step? Maybe I needed to add module ngx_http_limit_req_module and recompile nginx? > But if there is no such module in nginx, I would see configuration errors (like with this typo in config): > > Regards, > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Wednesday, February 26, 2020 4:50 AM > TO: Yury Shpakov > CC: nginx-devel at nginx.org > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > There are several possibilities to introduce a latency in nginx: > > - limit_req - https://www.nginx.com/blog/rate-limiting-nginx/#Two-Stage-Rate-Limiting [2] > > - Maxim's ngx_http_delay (I used it more for development purposes, like test or simulation of load etc); > > - some "slow" upstream backend that doing nothing, just waiting (preferably asynchronous). > > You seems to have some upstream (php?) serving auth_http requests, so you could for example implement some delay in case of failed attempt within php (or whatever you use there as backend). > Note that it is always good if the latency will be implemented asynchronously (without a real "sleep") in order to avoid possible overload under DDoS similar circumstances. > > Regards, > Sergey. > > Am 26.02.2020 01:59, schrieb Yury Shpakov: > Hi Sergey, > > You mentioned that you can set up some delays in responses. > How can I do it? > Adding this module during compilation? > https://github.com/openresty/echo-nginx-module [3] > > I tried but it didn't want to compile. I got many compilation errors. > > Maybe I can set up delays somehow else? > > Thank you, > Yury > > ------------------------- > > FROM: nginx-devel on behalf of Yury Shpakov > SENT: Friday, February 14, 2020 6:08 PM > TO: Sergey Brester > CC: nginx-devel at nginx.org > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > So what is the meaning of Auth-Server and Auth-Port headers? So it's relevant only when nginx works as SMTP Proxy (not SMTP Server)? And these are host/port where to redirect SMTP requests? > Yeah, I was all the time surprised -- how come, it's set as Proxy but there is no setting where it redirects SMTP communication to. A little bit unexpected place for those setting. > > Well, let me try... > > I ran Fake SMTP Server on port 25.(I found on Internet some fake SMTP Server). I configured my test SMTP client to localhost:25 (later to 127.0.0.1:25). They send/receive successfully. So both SMTP Client and (fake) SMTP Server work fine. > 127.0.0.1 works fine too. > > I re-configured my test SMTP client to localhost:8025 (tried 127.0.0.1:8025 too). As well, I changed this section of config as follows: > http { > > server { > listen 9000; > > location /cgi-bin/nginxauth.cgi { > add_header Auth-Status OK; > add_header Auth-Server 127.0.0.1; # backend ip > add_header Auth-Port 25; # backend port > return 204; > } > } > } > The same error: > 2020/02/14 17:37:18 [error] 15260#3328: *5 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 > > UPDATE: > Detailed logging with debug information helped a lot. > This is what I noticed in there: > > 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp auth state > > 2020/02/14 17:40:28 [debug] 3940#22096: *1 WSARecv: fd:584 rc:0 24 of 4096 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp rcpt to:"RCPT TO:" > 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer del: 584: 1172123084 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 malloc: 02F8C260:2048 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 stream socket 588 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 connect to [::1]:9000, fd:588 #2 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:768 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:16 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 > 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 > So it's trying to use IP6 rather than IP4. > And below: > > 2020/02/14 17:40:29 [debug] 3940#22096: *1 delete posted event 03171170 > > 2020/02/14 17:40:29 [debug] 3940#22096: *1 mail auth http write handler > 2020/02/14 17:40:29 [debug] 3940#22096: *1 WSASend: fd:588, -1, 0 of 306 > 2020/02/14 17:40:29 [error] 3940#22096: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 > 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 > 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 > So, I replaced localhost with 127.0.0.1 like this: > auth_http 127.0.0.1:9000/cgi-bin/nginxauth.cgi; > > And it worked. Since I forced it to use IP4. > Any idea how to use host name instead of IP address and still have it working? > > UPDATE 2: > I figured it out. Googled a little bit and ended up with the following change to my config: > > http { > > server { > listen 9000; > listen [::]:9000 ipv6only=on; > > location /cgi-bin/nginxauth.cgi { > add_header Auth-Status OK; > add_header Auth-Server 127.0.0.1; # backend ip > add_header Auth-Port 25; # backend port > return 204; > } > } > } > Now it works. > But why just "listen 9000" doesn't listen on both IP4 and IP6? > Is it a bug? > > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Friday, February 14, 2020 5:59 AM > TO: Yury Shpakov > CC: nginx-devel at nginx.org > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > I don't know what is wrong with your config... I guess your smtp server does not answer properly. > > Is 127.0.0.2:143 really your SMTP-server? > Because port 143 is mostly an IMAP port - but you've specified PROTOCOL SMTP in the server section. > > Anyway I tested your config with my settings (replaced name and smtp-server, here 192.0.2.222:25) and enabled debug: > > + error_log logs/error-mail.log DEBUG; > mail { > - server_name localhost; > + server_name EXAMPLE.COM; > ... > http { > ... > - add_header Auth-Server 127.0.0.2; > - add_header Auth-Port 143; > + add_header Auth-Server 192.0.2.222; > + add_header Auth-Port 25; > > it works well - I see the test incoming mail (I send to myself via 8025 port) and following output in the log (a lot of irrelevant messages are removed): > >>>>>>>>>> > > 2020/02/14 11:24:04 [debug] 121280#128244: *1 smtp mail from:"mail FROM:" > ... > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request line: "GET /cgi-bin/nginxauth.cgi HTTP/1.0" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http uri: "/cgi-bin/nginxauth.cgi" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http args: "" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http exten: "cgi" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http process request header line > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Host: localhost" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Method: none" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-User: " > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Pass: " > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Protocol: smtp" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Login-Attempt: 1" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-IP: 127.0.0.1" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-Host: [UNAVAILABLE]" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-Helo: myhost.example.com" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-From: mail FROM:" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-To: rcpt TO:" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header done > 2020/02/14 11:24:04 [debug] 121280#128244: *3 event timer del: 512: 1127939767 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 generic phase: 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 1 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 search through nested static locations of "" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 test location: "/cgi-bin/nginxauth.cgi" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 using configuration "/cgi-bin/nginxauth.cgi" > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http cl:-1 max:1048576 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 3 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http set discard body > 2020/02/14 11:24:04 [debug] 121280#128244: *3 HTTP/1.1 204 No Content > Server: nginx/1.17.4 > Date: Fri, 14 Feb 2020 10:24:04 GMT > Connection: close > Auth-Status: OK > Auth-Server: 192.0.2.222 > Auth-Port: 25 > > 2020/02/14 11:24:04 [debug] 121280#128244: *3 write new buf t:1 f:0 008AD6A0, pos 008AD6A0, size: 164 file: 0, size: 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter: l:1 f:0 s:164 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter limit 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 WSASend: fd:512, s:164 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter 00000000 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http finalize request: 0, "/cgi-bin/nginxauth.cgi?" a:1, c:1 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request count:1 blk:0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http close request > 2020/02/14 11:24:04 [debug] 121280#128244: *3 http log handler > 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008ACC50, unused: 1161 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 close http connection: 512 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 reusable connection: 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008AC848 > 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 0039FDE0, unused: 28 > 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http read handler > 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 164 of 1024 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process status line > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process headers > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Server: nginx/1.17.4" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Date: Fri, 14 Feb 2020 10:24:04 GMT" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Connection: close" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Status: OK" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Server: 192.0.2.222" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Port: 25" > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header done > 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer del: 496: 1127939764 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 reusable connection: 0 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 free: 008AC040, unused: 196 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 stream socket 496 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 connect to 192.0.2.222:25, fd:496 #4 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer add: 496: 60000:1127939769 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 008AC040:4096 > 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:04 [debug] 121280#128244: *1 post event 00897120 > 2020/02/14 11:24:04 [debug] 121280#128244: posted event 00897120 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 delete posted event 00897120 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy dummy handler > 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy smtp auth handler > 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 22 of 4096 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy send ehlo > 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 0039FDE0:256 > 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSASend: fd:496, 0, 25 of 25 > 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 196 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send mail from > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 47 of 47 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 60 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send rcpt to > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 45 of 45 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 63 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer add: 492: 86400000:1214280441 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer del: 496: 1127939769 > 2020/02/14 11:24:05 [info] 121280#128244: *1 client logged in, client: 127.0.0.1, server: 0.0.0.0:8025 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 1, #496 > #492 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 63 of 63 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 6 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 6 of 6 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 50 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 50 of 50 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 170 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 170 of 170 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280535 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 56 of 4096 > 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 56 of 56 > 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle > ... > > <<<<<<<<< > > Regards, > Sergey > > 13.02.2020 22:45, Yury Shpakov wrote: > Hi Sergey, > > I reconfigured the config file as follows: > > === === === > > #user nobody; > worker_processes 1; > > #error_log logs/error.log; > #error_log logs/error.log notice; > #error_log logs/error.log info; > > #pid logs/nginx.pid; > > events { > worker_connections 1024; > } > > mail { > server_name localhost; > auth_http localhost:9000/cgi-bin/nginxauth.cgi; > # auth_http none; > > smtp_auth none; > # smtp_auth login plain cram-md5; > # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; > xclient off; > > server { > listen 8025; > protocol smtp; > proxy on; > proxy_pass_error_message on; > } > } > > http { > server { > listen 9000; > > location /cgi-bin/nginxauth.cgi { > add_header Auth-Status OK; > add_header Auth-Server 127.0.0.2; # backend ip > add_header Auth-Port 143; # backend port > return 204; > } > } > } > === === === > > And now it's responding on port 9000 as expected: > > === === === > C:WINDOWSsystem32>curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi > > HTTP/1.1 204 No Content > Server: nginx/1.17.9 > Date: Thu, 13 Feb 2020 21:30:54 GMT > Connection: keep-alive > Auth-Status: OK > Auth-Server: 127.0.0.2 Auth-Port: 143 > === === === > > However I'm still experiencing the same issue (in log file): > > === === === > 2020/02/13 16:29:24 [notice] 35048#26192: signal process started > > 2020/02/13 16:29:34 [error] 31732#22720: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 === === === > > Tried under both admin and regular user. > > Any further ideas how to get it fixed please? > > Thank you, > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Wednesday, February 12, 2020 1:51 PM > TO: Yury Shpakov > CC: nginx-devel at nginx.org > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > I answered inline... > > 12.02.2020 18:59, Yury Shpakov wrote: > Hi Sergey, > > Thank you for you response. > > I tried netstat /nabo and I don't see any reference to port 9000 at all. > So a problem is to make nginx to listen on port 9000 (as server)? > Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? > With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). > Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). > > See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ [4] for more examples. > Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. > > I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. > > Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy [5] - you can use some nginx location (and internal URL to same nginx instance) to specify that. > > Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. > At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. > Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. > And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. > > And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). > Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link [6]) > But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). > > Thank you, > Yury > > ------------------------- > > FROM: Sergey Brester > SENT: Wednesday, February 12, 2020 7:38 AM > TO: nginx-devel at nginx.org > CC: Yury Shpakov > SUBJECT: Re: nginx for Windows - WSASend() socket error 10057 > > It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... > > try netstat (in cmd as admin): > > netstat /nabo > netstat /nabo | grep -A 1 ":9000b" > > and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). > > additionally try to telnet or curl it: > > curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi > > if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). > > If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [7]). > > But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. > It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). > > Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [8] > > Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH [6] - nginx.zip [9], where it works well). > > Regards, > Sergey > > 12.02.2020 03:01, Yury Shpakov wrote: > Hi there, > > Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: > http://nginx.org/en/docs/howto_build_on_win32.html [10] > But excluded (don't care about SSL at this point so don't want to install/configure Perl now): > --with-openssl=objs/lib/openssl-master > > --with-openssl-opt=no-asm > --with-http_ssl_module > And added: > --with-mail > > nmake was successful and nginx.exe was created. > However nginx.exe keeps failing with the error: > WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 > Windows API says the following about this error: > > WSAENOTCONN10057 > Socket is not connected.A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using SENDTO [11]) no address was supplied. Any other type of operation might also return this error--for example, SETSOCKOPT [12] setting SO_KEEPALIVE [13] if the connection has been reset. > > https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 [14] > > Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs [14] > Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. > docs.microsoft.com > > Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. > Any recommendation for me to make it work? > Tried to play with config (commenting/uncommenting): > > #user nobody; > worker_processes 1; > #error_log logs/error.log; > #error_log logs/error.log notice; > #error_log logs/error.log info; > #pid logs/nginx.pid; > events { > worker_connections 1024; > } > mail { > server_name localhost; > auth_http localhost:9000/cgi-bin/nginxauth.cgi; > # auth_http none; > smtp_auth none; > # smtp_auth login plain cram-md5; > # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; > xclient off; > server { > listen 8025; > protocol smtp; > proxy on; > proxy_pass_error_message on; > } > } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. > Thank you, > Yury > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel [1] Links: ------ [1] http://mailman.nginx.org/mailman/listinfo/nginx-devel [2] https://www.nginx.com/blog/rate-limiting-nginx/#Two-Stage-Rate-Limiting [3] https://github.com/openresty/echo-nginx-module [4] https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ [5] https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy [6] https://github.com/sebres/nginx/releases/tag/release-1.13.0 [7] http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html [8] https://forum.nginx.org/read.php?2,257206,257207#msg-257207 [9] https://github.com/sebres/nginx/files/2246440/nginx.zip [10] http://nginx.org/en/docs/howto_build_on_win32.html [11] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-sendto [12] https://docs.microsoft.com/en-us/windows/desktop/api/winsock/nf-winsock-setsockopt [13] https://docs.microsoft.com/en-us/windows/desktop/winsock/so-keepalive [14] https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 42105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 9367 bytes Desc: not available URL: From alexander.borisov at nginx.com Thu Feb 27 11:30:46 2020 From: alexander.borisov at nginx.com (Alexander Borisov) Date: Thu, 27 Feb 2020 11:30:46 +0000 Subject: [njs] Fixed heap-buffer-overflow in lexer introduced in 87d05fb35ff9. Message-ID: details: https://hg.nginx.org/njs/rev/3313d0d593a0 branches: changeset: 1340:3313d0d593a0 user: Alexander Borisov date: Thu Feb 27 14:30:14 2020 +0300 description: Fixed heap-buffer-overflow in lexer introduced in 87d05fb35ff9. diffstat: src/njs_lexer.c | 2 +- src/test/njs_unit_test.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletions(-) diffs (37 lines): diff -r 6444ae6ab276 -r 3313d0d593a0 src/njs_lexer.c --- a/src/njs_lexer.c Wed Feb 26 17:48:46 2020 +0300 +++ b/src/njs_lexer.c Thu Feb 27 14:30:14 2020 +0300 @@ -864,7 +864,7 @@ njs_lexer_multi(njs_lexer_t *lexer, njs_ token->text.start = lexer->start - 1; - while (length != 0 && multi != NULL) { + while (length != 0 && multi != NULL && lexer->start < lexer->end) { c = lexer->start[0]; if (c == multi->symbol) { diff -r 6444ae6ab276 -r 3313d0d593a0 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Wed Feb 26 17:48:46 2020 +0300 +++ b/src/test/njs_unit_test.c Thu Feb 27 14:30:14 2020 +0300 @@ -33,6 +33,21 @@ static njs_unit_test_t njs_test[] = { njs_str("/***/1/*\n**/"), njs_str("1") }, + { njs_str(">"), + njs_str("SyntaxError: Unexpected token \">\" in 1") }, + + { njs_str(">>"), + njs_str("SyntaxError: Unexpected token \">>\" in 1") }, + + { njs_str(">>>"), + njs_str("SyntaxError: Unexpected token \">>>\" in 1") }, + + { njs_str("=="), + njs_str("SyntaxError: Unexpected token \"==\" in 1") }, + + { njs_str("?"), + njs_str("SyntaxError: Unexpected token \"?\" in 1") }, + /* Variable declarations. */ { njs_str("var x"), From mdounin at mdounin.ru Thu Feb 27 11:47:08 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 27 Feb 2020 14:47:08 +0300 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: <20891f45d8b2d29e42b9660acba9ddba@sebres.de> References: <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> <20891f45d8b2d29e42b9660acba9ddba@sebres.de> Message-ID: <20200227114708.GD12894@mdounin.ru> Hello! On Thu, Feb 27, 2020 at 11:20:41AM +0100, Sergey Brester wrote: > I could imagine that this doesn't work as expected because > mail::auth_http simply makes fewer requests as you may assume (due to > keep-alive + some internal "cache" for established connections). There is no keep-alive nor any cache for established connections in auth_http. The auth_http server is called for each authentication attempt. As long as authentication is successful, nginx establishes an opaque connection to the backend server as indicated by the auth server response. [...] > @nginx-devel: I don't see a troubles to extend directives of > "ngx_http_limit_req_module" to consider mail/server too. > Are there some objections against that? There are no requests in mail. Further, to delay authentication for clients auth_http already has the Auth-Wait header. Further, I don't see what you are trying to debug here. As I see from the messages in this thread, the issue was lack of IPv6 listener while using an IPv6 address in auth_http, and it is already resolved. -- Maxim Dounin http://mdounin.ru/ From serg.brester at sebres.de Thu Feb 27 13:29:18 2020 From: serg.brester at sebres.de (Dipl. Ing. Sergey Brester) Date: Thu, 27 Feb 2020 14:29:18 +0100 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: <20200227114708.GD12894@mdounin.ru> References: <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> <20891f45d8b2d29e42b9660acba9ddba@sebres.de> <20200227114708.GD12894@mdounin.ru> Message-ID: Am 27.02.2020 12:47, schrieb Maxim Dounin: > Further, I don't see what you are trying to debug here. As I see > from the messages in this thread, the issue was lack of IPv6 > listener while using an IPv6 address in auth_http, and it is > already resolved. Well, as for the initial issue - sure. :) But it "continues" with the new question "How I could delay" and one of my suggestions was "use limit_req, Luke" (which did not work for some reason against location of `auth_http`). I don't know why the people put several questions in one thread, but you should ask them not me. So he can indeed try "Auth-Wait" now (I totally forget about that, thx for reminder). Anyway the suggestion to debug has come after "strange" behavior of limit_req by "auth_http" requests, especially as you say that this location is called each time the client authenticate: > There is no keep-alive nor any cache for established connections in auth_http. > > The auth_http server is called for each authentication attempt. Regards, Sergey -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Thu Feb 27 15:24:29 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 27 Feb 2020 18:24:29 +0300 Subject: [PATCH] Ensured SIGQUIT deletes listening UNIX socket files. In-Reply-To: <4b6bc48b-6c7a-7c76-51ae-1038421ed36c@fastmail.com> References: <4b6bc48b-6c7a-7c76-51ae-1038421ed36c@fastmail.com> Message-ID: <20200227152429.GI12894@mdounin.ru> Hello! On Wed, Feb 26, 2020 at 04:49:55PM -0800, Thibault Charbonnier wrote: > # HG changeset patch > # User Thibault Charbonnier > # Date 1582764433 28800 > # Wed Feb 26 16:47:13 2020 -0800 > # Node ID 55ea1a9197a6f28d4da00909e5ea8585f6a08239 > # Parent 4f18393a1d51bce6103ea2f1b2587900f349ba3d > Ensured SIGQUIT deletes listening UNIX socket files. > > Prior to this patch, the SIGQUIT signal handling (graceful shutdown) did not > remove UNIX socket files since ngx_master_process_cycle reimplemented > listening > socket closings in lieu of using ngx_close_listening_sockets. > > Since ngx_master_process_exit will call the aforementioned > ngx_close_listening_sockets, we can remove the custom implementation and now > expect listening sockets to be closed properly by > ngx_close_listening_sockets > instead. > > This fixes the trac issue #753 (https://trac.nginx.org/nginx/ticket/753). > > diff -r 4f18393a1d51 -r 55ea1a9197a6 src/os/unix/ngx_process_cycle.c > --- a/src/os/unix/ngx_process_cycle.c Thu Feb 20 16:51:07 2020 +0300 > +++ b/src/os/unix/ngx_process_cycle.c Wed Feb 26 16:47:13 2020 -0800 > @@ -77,12 +77,11 @@ > u_char *p; > size_t size; > ngx_int_t i; > - ngx_uint_t n, sigio; > + ngx_uint_t sigio; > sigset_t set; > struct itimerval itv; > ngx_uint_t live; > ngx_msec_t delay; > - ngx_listening_t *ls; > ngx_core_conf_t *ccf; > > sigemptyset(&set); > @@ -205,16 +204,6 @@ > ngx_signal_worker_processes(cycle, > > ngx_signal_value(NGX_SHUTDOWN_SIGNAL)); > > - ls = cycle->listening.elts; > - for (n = 0; n < cycle->listening.nelts; n++) { > - if (ngx_close_socket(ls[n].fd) == -1) { > - ngx_log_error(NGX_LOG_EMERG, cycle->log, > ngx_socket_errno, > - ngx_close_socket_n " %V failed", > - &ls[n].addr_text); > - } > - } > - cycle->listening.nelts = 0; > - > continue; > } > Have you checked what happens during binary upgrade with your patch? Previous attempt to fix this was here: http://mailman.nginx.org/pipermail/nginx-devel/2016-December/009207.html http://mailman.nginx.org/pipermail/nginx-devel/2016-December/009208.html Yet it failed to address binary upgrade case properly, see here: http://mailman.nginx.org/pipermail/nginx-devel/2016-December/009239.html -- Maxim Dounin http://mdounin.ru/ From pluknet at nginx.com Thu Feb 27 16:05:14 2020 From: pluknet at nginx.com (Sergey Kandaurov) Date: Thu, 27 Feb 2020 16:05:14 +0000 Subject: [nginx] Disabled connection reuse while in SSL handshake. Message-ID: details: https://hg.nginx.org/nginx/rev/2e3bfd696ecb branches: changeset: 7628:2e3bfd696ecb user: Sergey Kandaurov date: Thu Feb 27 19:03:21 2020 +0300 description: Disabled connection reuse while in SSL handshake. During SSL handshake, the connection could be reused in the OCSP stapling callback, if configured, which subsequently leads to a segmentation fault. diffstat: src/http/ngx_http_request.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (21 lines): diff -r 4f18393a1d51 -r 2e3bfd696ecb src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c Thu Feb 20 16:51:07 2020 +0300 +++ b/src/http/ngx_http_request.c Thu Feb 27 19:03:21 2020 +0300 @@ -748,6 +748,8 @@ ngx_http_ssl_handshake(ngx_event_t *rev) return; } + ngx_reusable_connection(c, 0); + rc = ngx_ssl_handshake(c); if (rc == NGX_AGAIN) { @@ -756,8 +758,6 @@ ngx_http_ssl_handshake(ngx_event_t *rev) ngx_add_timer(rev, c->listening->post_accept_timeout); } - ngx_reusable_connection(c, 0); - c->ssl->handler = ngx_http_ssl_handshake_handler; return; } From yshpakov at hotmail.com Thu Feb 27 19:40:38 2020 From: yshpakov at hotmail.com (Yury Shpakov) Date: Thu, 27 Feb 2020 19:40:38 +0000 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: <20891f45d8b2d29e42b9660acba9ddba@sebres.de> References: , <766a2741f22fa9d6e5e8c2ec2817935a@sebres.de> , <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> , , , , <20891f45d8b2d29e42b9660acba9ddba@sebres.de> Message-ID: Hi Sergey and guys, I wrote my own auth_http server. And it was hit only one time for 1000 SMTP requests. So auth_http is requested once per session/connection. For this example it's hit once: using (var smtpClient = new SmtpClient("localhost", 8025) { Timeout = 60 * 60 * 1000 }) { for (int i = 0; i < 1000; i++) { smtpClient.Send("noreply at wmata.com", "ys at wmata.com", "Email subject", "Email body"); } } For this example it's hit 1000 times: for (int i = 0; i < 1000; i++) { using (var smtpClient = new SmtpClient("localhost", 8025) { Timeout = 60 * 60 * 1000 }) { smtpClient.Send("noreply at wmata.com", "ys at wmata.com", "Email subject", "Email body"); } } Evidently by the same reason this one didn't work either: smtp_greeting_delay 10s; This options is not gonna work either: - Maxim's ngx_http_delay (I used it more for development purposes, like test or simulation of load etc); If I do this I don't need nginx at all. But my boss said: "Use nginx. Period.": - some "slow" upstream backend that doing nothing, just waiting (preferably asynchronous). So the question is still open: how to make nginx wait on SMTP processing to model latency for my load tests? Thank you, Yury ________________________________ From: Sergey Brester Sent: Thursday, February 27, 2020 5:20 AM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 Hmmm... I could imagine that this doesn't work as expected because mail::auth_http simply makes fewer requests as you may assume (due to keep-alive + some internal "cache" for established connections). I have no time to trace it right now (you can enable debug and look how often /cgi-bin/nginxauth.cgi will be called internally). Normally it would be more proper to place limit_req into mail/server sections, but I'm pretty sure you would get something like "limit_req directive is not allowed here". One should extend this module to allow that in mail/server sections. So it looks like this method is not suitable for you at the moment. @nginx-devel: I don't see a troubles to extend directives of "ngx_http_limit_req_module" to consider mail/server too. Are there some objections against that? Regards, Serg. Am 26.02.2020 21:36, schrieb Yury Shpakov: Hi Sergey, I added couple lines in my config and re-ran nginx: === === === worker_processes 1; events { worker_connections 1024; } #error_log logs/error-mail.log debug; mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; smtp_auth none; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } http { limit_req_zone $binary_remote_addr zone=ip:10m rate=5r/s; server { listen 9000; listen [::]:9000 ipv6only=on; location /cgi-bin/nginxauth.cgi { limit_req zone=ip burst=12 delay=8; add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } === === === My C# test code is very simple: using (var smtpClient = new SmtpClient("localhost", 8025) {Timeout = 60 * 60 * 1000}) { for (int i = 0; i < 1000; i++) { smtpClient.Send("noreply at wmata.com", "ys at wmata.com", "Email subject", "Email body"); } } And with no delays or failures upstream (Fake) SMTP Server receives all 1000 emails (actually within 3 seconds): [cid:15827988415e5797f99115f690369219 at sebres.de] Received from 03:22:16 to 03:22:19 all 1000. So latency is not working. Maybe I missed some step? Maybe I needed to add module ngx_http_limit_req_module and recompile nginx? But if there is no such module in nginx, I would see configuration errors (like with this typo in config): [cid:15827988415e5797f991a91907815525 at sebres.de] Regards, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 26, 2020 4:50 AM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 There are several possibilities to introduce a latency in nginx: - limit_req - https://www.nginx.com/blog/rate-limiting-nginx/#Two-Stage-Rate-Limiting - Maxim's ngx_http_delay (I used it more for development purposes, like test or simulation of load etc); - some "slow" upstream backend that doing nothing, just waiting (preferably asynchronous). You seems to have some upstream (php?) serving auth_http requests, so you could for example implement some delay in case of failed attempt within php (or whatever you use there as backend). Note that it is always good if the latency will be implemented asynchronously (without a real "sleep") in order to avoid possible overload under DDoS similar circumstances. Regards, Sergey. Am 26.02.2020 01:59, schrieb Yury Shpakov: Hi Sergey, You mentioned that you can set up some delays in responses. How can I do it? Adding this module during compilation? https://github.com/openresty/echo-nginx-module I tried but it didn't want to compile. I got many compilation errors. Maybe I can set up delays somehow else? Thank you, Yury ________________________________ From: nginx-devel on behalf of Yury Shpakov Sent: Friday, February 14, 2020 6:08 PM To: Sergey Brester Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 So what is the meaning of Auth-Server and Auth-Port headers? So it's relevant only when nginx works as SMTP Proxy (not SMTP Server)? And these are host/port where to redirect SMTP requests? Yeah, I was all the time surprised -- how come, it's set as Proxy but there is no setting where it redirects SMTP communication to. A little bit unexpected place for those setting. Well, let me try... I ran Fake SMTP Server on port 25.(I found on Internet some fake SMTP Server). I configured my test SMTP client to localhost:25 (later to 127.0.0.1:25). They send/receive successfully. So both SMTP Client and (fake) SMTP Server work fine. 127.0.0.1 works fine too. I re-configured my test SMTP client to localhost:8025 (tried 127.0.0.1:8025 too). As well, I changed this section of config as follows: http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } The same error: 2020/02/14 17:37:18 [error] 15260#3328: *5 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Update: Detailed logging with debug information helped a lot. This is what I noticed in there: 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp auth state 2020/02/14 17:40:28 [debug] 3940#22096: *1 WSARecv: fd:584 rc:0 24 of 4096 2020/02/14 17:40:28 [debug] 3940#22096: *1 smtp rcpt to:"RCPT TO:" 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer del: 584: 1172123084 2020/02/14 17:40:28 [debug] 3940#22096: *1 malloc: 02F8C260:2048 2020/02/14 17:40:28 [debug] 3940#22096: *1 stream socket 588 2020/02/14 17:40:28 [debug] 3940#22096: *1 connect to [::1]:9000, fd:588 #2 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:768 2020/02/14 17:40:28 [debug] 3940#22096: *1 select add event fd:588 ev:16 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 2020/02/14 17:40:28 [debug] 3940#22096: *1 event timer add: 588: 60000:1172123084 So it's trying to use IP6 rather than IP4. And below: 2020/02/14 17:40:29 [debug] 3940#22096: *1 delete posted event 03171170 2020/02/14 17:40:29 [debug] 3940#22096: *1 mail auth http write handler 2020/02/14 17:40:29 [debug] 3940#22096: *1 WSASend: fd:588, -1, 0 of 306 2020/02/14 17:40:29 [error] 3940#22096: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 2020/02/14 17:40:29 [debug] 3940#22096: *1 event timer del: 588: 1172123084 So, I replaced localhost with 127.0.0.1 like this: auth_http 127.0.0.1:9000/cgi-bin/nginxauth.cgi; And it worked. Since I forced it to use IP4. Any idea how to use host name instead of IP address and still have it working? Update 2: I figured it out. Googled a little bit and ended up with the following change to my config: http { server { listen 9000; listen [::]:9000 ipv6only=on; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; # backend ip add_header Auth-Port 25; # backend port return 204; } } } Now it works. But why just "listen 9000" doesn't listen on both IP4 and IP6? Is it a bug? Yury ________________________________ From: Sergey Brester Sent: Friday, February 14, 2020 5:59 AM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I don't know what is wrong with your config... I guess your smtp server does not answer properly. Is 127.0.0.2:143 really your SMTP-server? Because port 143 is mostly an IMAP port - but you've specified protocol smtp in the server section. Anyway I tested your config with my settings (replaced name and smtp-server, here 192.0.2.222:25) and enabled debug: + error_log logs/error-mail.log debug; mail { - server_name localhost; + server_name example.com; ... http { ... - add_header Auth-Server 127.0.0.2; - add_header Auth-Port 143; + add_header Auth-Server 192.0.2.222; + add_header Auth-Port 25; it works well - I see the test incoming mail (I send to myself via 8025 port) and following output in the log (a lot of irrelevant messages are removed): >>>>>>>>> 2020/02/14 11:24:04 [debug] 121280#128244: *1 smtp mail from:"mail FROM:" ... 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request line: "GET /cgi-bin/nginxauth.cgi HTTP/1.0" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http uri: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http args: "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http exten: "cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http process request header line 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Host: localhost" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Method: none" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-User: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Pass: " 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Protocol: smtp" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-Login-Attempt: 1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-IP: 127.0.0.1" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Client-Host: [UNAVAILABLE]" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-Helo: myhost.example.com" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-From: mail FROM:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header: "Auth-SMTP-To: rcpt TO:" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http header done 2020/02/14 11:24:04 [debug] 121280#128244: *3 event timer del: 512: 1127939767 2020/02/14 11:24:04 [debug] 121280#128244: *3 generic phase: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 1 2020/02/14 11:24:04 [debug] 121280#128244: *3 search through nested static locations of "" 2020/02/14 11:24:04 [debug] 121280#128244: *3 test location: "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 using configuration "/cgi-bin/nginxauth.cgi" 2020/02/14 11:24:04 [debug] 121280#128244: *3 http cl:-1 max:1048576 2020/02/14 11:24:04 [debug] 121280#128244: *3 rewrite phase: 3 2020/02/14 11:24:04 [debug] 121280#128244: *3 http set discard body 2020/02/14 11:24:04 [debug] 121280#128244: *3 HTTP/1.1 204 No Content Server: nginx/1.17.4 Date: Fri, 14 Feb 2020 10:24:04 GMT Connection: close Auth-Status: OK Auth-Server: 192.0.2.222 Auth-Port: 25 2020/02/14 11:24:04 [debug] 121280#128244: *3 write new buf t:1 f:0 008AD6A0, pos 008AD6A0, size: 164 file: 0, size: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter: l:1 f:0 s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter limit 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 WSASend: fd:512, s:164 2020/02/14 11:24:04 [debug] 121280#128244: *3 http write filter 00000000 2020/02/14 11:24:04 [debug] 121280#128244: *3 http finalize request: 0, "/cgi-bin/nginxauth.cgi?" a:1, c:1 2020/02/14 11:24:04 [debug] 121280#128244: *3 http request count:1 blk:0 2020/02/14 11:24:04 [debug] 121280#128244: *3 http close request 2020/02/14 11:24:04 [debug] 121280#128244: *3 http log handler 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008ACC50, unused: 1161 2020/02/14 11:24:04 [debug] 121280#128244: *3 close http connection: 512 2020/02/14 11:24:04 [debug] 121280#128244: *3 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 008AC848 2020/02/14 11:24:04 [debug] 121280#128244: *3 free: 0039FDE0, unused: 28 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http read handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 164 of 1024 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process status line 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http process headers 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Server: nginx/1.17.4" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Date: Fri, 14 Feb 2020 10:24:04 GMT" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Connection: close" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Status: OK" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Server: 192.0.2.222" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header: "Auth-Port: 25" 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail auth http header done 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer del: 496: 1127939764 2020/02/14 11:24:04 [debug] 121280#128244: *1 reusable connection: 0 2020/02/14 11:24:04 [debug] 121280#128244: *1 free: 008AC040, unused: 196 2020/02/14 11:24:04 [debug] 121280#128244: *1 stream socket 496 2020/02/14 11:24:04 [debug] 121280#128244: *1 connect to 192.0.2.222:25, fd:496 #4 2020/02/14 11:24:04 [debug] 121280#128244: *1 event timer add: 496: 60000:1127939769 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 008AC040:4096 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 post event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 delete posted event 00897120 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy dummy handler 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 22 of 4096 2020/02/14 11:24:04 [debug] 121280#128244: *1 mail proxy send ehlo 2020/02/14 11:24:04 [debug] 121280#128244: *1 malloc: 0039FDE0:256 2020/02/14 11:24:04 [debug] 121280#128244: *1 WSASend: fd:496, 0, 25 of 25 2020/02/14 11:24:04 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 196 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send mail from 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 47 of 47 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 60 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy send rcpt to 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 45 of 45 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy smtp auth handler 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 63 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer add: 492: 86400000:1214280441 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer del: 496: 1127939769 2020/02/14 11:24:05 [info] 121280#128244: *1 client logged in, client: 127.0.0.1, server: 0.0.0.0:8025 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 1, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 63 of 63 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 6 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 6 of 6 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280441 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 50 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 50 of 50 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 post event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 delete posted event 008830C8 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #492 > #496 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:492 rc:0 170 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:496, 0, 170 of 170 2020/02/14 11:24:05 [debug] 121280#128244: *1 event timer: 492, old: 1214280441, new: 1214280535 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle 2020/02/14 11:24:05 [debug] 121280#128244: *1 mail proxy handler: 0, #496 > #492 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSARecv: fd:496 rc:0 56 of 4096 2020/02/14 11:24:05 [debug] 121280#128244: *1 WSASend: fd:492, 0, 56 of 56 2020/02/14 11:24:05 [debug] 121280#128244: worker cycle ... <<<<<<<<< Regards, Sergey 13.02.2020 22:45, Yury Shpakov wrote: Hi Sergey, I reconfigured the config file as follows: === === === #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } http { server { listen 9000; location /cgi-bin/nginxauth.cgi { add_header Auth-Status OK; add_header Auth-Server 127.0.0.2; # backend ip add_header Auth-Port 143; # backend port return 204; } } } === === === And now it's responding on port 9000 as expected: === === === C:\WINDOWS\system32>curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi HTTP/1.1 204 No Content Server: nginx/1.17.9 Date: Thu, 13 Feb 2020 21:30:54 GMT Connection: keep-alive Auth-Status: OK Auth-Server: 127.0.0.2 Auth-Port: 143 === === === However I'm still experiencing the same issue (in log file): === === === 2020/02/13 16:29:24 [notice] 35048#26192: signal process started 2020/02/13 16:29:34 [error] 31732#22720: *1 WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 === === === Tried under both admin and regular user. Any further ideas how to get it fixed please? Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 1:51 PM To: Yury Shpakov Cc: nginx-devel at nginx.org Subject: Re: nginx for Windows - WSASend() socket error 10057 I answered inline... 12.02.2020 18:59, Yury Shpakov wrote: Hi Sergey, Thank you for you response. I tried netstat /nabo and I don't see any reference to port 9000 at all. So a problem is to make nginx to listen on port 9000 (as server)? Or nginx is not listening on port 9000 but rather sending requests to port 9000 (as client)? With setting of `auth_http`, you are defining an URL to the service responsible for authentication (and upstream choice). Of course then you should have something that would response to the auth-requests (your own upstream, or some nginx location, or some "foreign" http-server). See https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/ for more examples. Maybe it's easier not to use auth_http at all? I was trying to remove it from configuration file but nginx was not happy. I have my own auth-module so I don't know how it can be solved in stock-nginx without this directive. Take a look here - https://serverfault.com/questions/594962/nginx-understanding-the-purpose-of-auth-http-imap-proxy - you can use some nginx location (and internal URL to same nginx instance) to specify that. Anyway it is recommended to use some auth (on nginx side), because it'd preserve the resources of mail-servers, allow you to authenticate email clients with same user/password for all mail-servers (smtp, imap, pop3, etc) as well as the same user/pwd as for some other http-services. And it is used to choose an upstream server (if multiple) for the email processing. At this point I don't need any authentication. I was told by my boss to use nginx for load testing of our service sending emails (SMTP client). I've got some SMTP Server and nginx would be used as SMTP proxy because it allows to set up delays. Well, an auth request to some nginx-location would allow you to set up delays even on authentication phase. And take into account that I REMOVED "--with-http_ssl_module" from parameters when I was building nginx. And you advised to download some nginx.exe files but I believe they were built without "--with-mail" parameter (which I need). Although, it was compiled with "--with-mail" (you can see all parameters in provided GH-link) But it would not help, because basically your issue seems to be the configuration (not the nginx.exe). Thank you, Yury ________________________________ From: Sergey Brester Sent: Wednesday, February 12, 2020 7:38 AM To: nginx-devel at nginx.org Cc: Yury Shpakov Subject: Re: nginx for Windows - WSASend() socket error 10057 It looks like your service defined in auth_http doesn't answer (or no listener on 127.0.0.1 port 9000?)... try netstat (in cmd as admin): netstat /nabo netstat /nabo | grep -A 1 ":9000\b" and check whether the listener on port 9000 is bound to 127.0.0.1 (or it is 0.0.0.0 only?) and it is the process you expect to see there (can be "reserved" by some other windows-service). additionally try to telnet or curl it: curl -H "Auth-Method: plain" -H "Auth-User: user" -H "Auth-Pass: pwd" -H "Auth-Protocol: imap" -H "Auth-Login-Attempt: 1" -i http://127.0.0.1:9000/cgi-bin/nginxauth.cgi if it does not answer, make another attempt by replace 127.0.0.1 with 0.0.0.0 (or a host-name). If it answers - see whether it is the expected response (some examples of good and bad responses are described in http://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html). But I guess if WSASend fails, it would probably (unexpected) reject the connection during the send (or even connect) process. It can be also invalid (unexpected) content-length in keep-alive connect to auth-upstream - so send but still receive is expected (or vice versa). Also follow this forum topic addressing similar issue: https://forum.nginx.org/read.php?2,257206,257207#msg-257207 Anyway it doesn't look to me like an issue of nginx (regardless windows or not), but you can also try some other ready build (for example on my GH - nginx.zip, where it works well). Regards, Sergey 12.02.2020 03:01, Yury Shpakov wrote: Hi there, Trying to make nginx work as SMTP server and/or SMTP proxy. Done everything according to: http://nginx.org/en/docs/howto_build_on_win32.html But excluded (don't care about SSL at this point so don't want to install/configure Perl now): --with-openssl=objs/lib/openssl-master \ --with-openssl-opt=no-asm \ --with-http_ssl_module \ And added: --with-mail nmake was successful and nginx.exe was created. However nginx.exe keeps failing with the error: WSASend() failed (10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied) while in http auth state, client: 127.0.0.1, server: 0.0.0.0:8025 Windows API says the following about this error: WSAENOTCONN 10057 Socket is not connected. A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using sendto) no address was supplied. Any other type of operation might also return this error?for example, setsockopt setting SO_KEEPALIVE if the connection has been reset. https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2 Windows Sockets Error Codes (Winsock2.h) - Win32 apps | Microsoft Docs Return code/value Description; WSA_INVALID_HANDLE 6: Specified event object handle is invalid. An application attempts to use an event object, but the specified handle is not valid. docs.microsoft.com Managed to debug your code in VS 2010 a little bit but it's brutal C so it's hard to figure your code out. And this debugger doesn't show you any local variables values. Any recommendation for me to make it work? Tried to play with config (commenting/uncommenting): #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } mail { server_name localhost; auth_http localhost:9000/cgi-bin/nginxauth.cgi; # auth_http none; smtp_auth none; # smtp_auth login plain cram-md5; # smtp_capabilities "SIZE 10485760" ENHANCEDSTATUSCODES 8BITMIME DSN; xclient off; server { listen 8025; protocol smtp; proxy on; proxy_pass_error_message on; } } Tried both under a regular user and under admin. Tried on 25, 1025 and 8025 ports. Thank you, Yury _______________________________________________ nginx-devel mailing list nginx-devel at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 42105 bytes Desc: image.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 9367 bytes Desc: image.png URL: From thibaultcha at fastmail.com Fri Feb 28 00:23:53 2020 From: thibaultcha at fastmail.com (Thibault Charbonnier) Date: Thu, 27 Feb 2020 16:23:53 -0800 Subject: [PATCH] Ensured SIGQUIT deletes listening UNIX socket files. In-Reply-To: <20200227152429.GI12894@mdounin.ru> References: <4b6bc48b-6c7a-7c76-51ae-1038421ed36c@fastmail.com> <20200227152429.GI12894@mdounin.ru> Message-ID: On 2/27/20 7:24 AM, Maxim Dounin wrote: > Have you checked what happens during binary upgrade with your > patch? Good call, I gave it a thought but did not bother checking, thanks for sharing the previous patch. This one is slightly simpler. Below is a new version of the patch covering binary upgrade edge-cases by relying on the existence of the nginx.oldpid file. Also attached to this email is a file I used as a test suite covering the behavior of this patch with SIGQUIT, SIGTERM, and binary upgrade scenarios. # HG changeset patch # User Thibault Charbonnier # Date 1582764433 28800 # Wed Feb 26 16:47:13 2020 -0800 # Node ID ec619d02801b925b4dad51515fb9668c1d993418 # Parent 4f18393a1d51bce6103ea2f1b2587900f349ba3d Ensured SIGQUIT deletes listening UNIX socket files. Prior to this patch, the SIGQUIT signal handling (graceful shutdown) did not remove UNIX socket files since ngx_master_process_cycle reimplemented listening socket closings in lieu of using ngx_close_listening_sockets. Since ngx_master_process_exit will call the aforementioned ngx_close_listening_sockets, we can remove the custom implementation and now expect listening sockets to be closed properly by ngx_close_listening_sockets instead. This fixes the trac issue #753 (https://trac.nginx.org/nginx/ticket/753). diff -r 4f18393a1d51 -r ec619d02801b src/core/ngx_connection.c --- a/src/core/ngx_connection.c Thu Feb 20 16:51:07 2020 +0300 +++ b/src/core/ngx_connection.c Wed Feb 26 16:47:13 2020 -0800 @@ -1023,6 +1023,10 @@ ngx_uint_t i; ngx_listening_t *ls; ngx_connection_t *c; +#if (NGX_HAVE_UNIX_DOMAIN) + ngx_core_conf_t *ccf; + ngx_fd_t fd; +#endif if (ngx_event_flags & NGX_USE_IOCP_EVENT) { return; @@ -1067,10 +1071,22 @@ } #if (NGX_HAVE_UNIX_DOMAIN) + ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module); + + fd = ngx_open_file(ccf->oldpid.data, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0); + + if (fd != NGX_INVALID_FILE) { + if (ngx_close_file(fd) == NGX_FILE_ERROR) { + ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno, + ngx_close_file_n " \"%s\" failed", + ccf->oldpid.data); + } + } if (ls[i].sockaddr->sa_family == AF_UNIX && ngx_process <= NGX_PROCESS_MASTER - && ngx_new_binary == 0) + && ngx_new_binary == 0 + && fd == NGX_INVALID_FILE) { u_char *name = ls[i].addr_text.data + sizeof("unix:") - 1; diff -r 4f18393a1d51 -r ec619d02801b src/os/unix/ngx_process_cycle.c --- a/src/os/unix/ngx_process_cycle.c Thu Feb 20 16:51:07 2020 +0300 +++ b/src/os/unix/ngx_process_cycle.c Wed Feb 26 16:47:13 2020 -0800 @@ -77,12 +77,11 @@ u_char *p; size_t size; ngx_int_t i; - ngx_uint_t n, sigio; + ngx_uint_t sigio; sigset_t set; struct itimerval itv; ngx_uint_t live; ngx_msec_t delay; - ngx_listening_t *ls; ngx_core_conf_t *ccf; > > Previous attempt to fix this was here: > > http://mailman.nginx.org/pipermail/nginx-devel/2016-December/009207.html > http://mailman.nginx.org/pipermail/nginx-devel/2016-December/009208.html > > Yet it failed to address binary upgrade case properly, see here: > > http://mailman.nginx.org/pipermail/nginx-devel/2016-December/009239.html > sigemptyset(&set); @@ -205,16 +204,6 @@ ngx_signal_worker_processes(cycle, ngx_signal_value(NGX_SHUTDOWN_SIGNAL)); - ls = cycle->listening.elts; - for (n = 0; n < cycle->listening.nelts; n++) { - if (ngx_close_socket(ls[n].fd) == -1) { - ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_socket_errno, - ngx_close_socket_n " %V failed", - &ls[n].addr_text); - } - } - cycle->listening.nelts = 0; - continue; } -------------- next part -------------- A non-text attachment was scrubbed... Name: test.sh Type: application/x-shellscript Size: 1971 bytes Desc: not available URL: From Patrik.Mada at firma.seznam.cz Fri Feb 28 09:14:31 2020 From: Patrik.Mada at firma.seznam.cz (Patrik Mada) Date: Fri, 28 Feb 2020 10:14:31 +0100 Subject: Dynamic Modules - Support CXXFLAGS Message-ID: <147FA808-5906-4B64-BFBF-3333B940BF71@firma.seznam.cz> Dear NGINX developers, we are using C++ dynamic modules in our NGINX based servers, while multiple Debian distributions have to be supported. On older Debian distributions it is impossible to use more recent C++ standard, due to build defaulting to system provided clang package. Our developers stood before decision to either create wrappers around modern C++ libraries which would be compliant with older standards or write the entire logic using obsolete featureless language. Neither of these approaches is sustainable in terms of developers' productivity and long term maintenance. Would it be possible to consider a change in dynamic modules' build system, so that C++ flags may be provided? We propose a low intrusive changes concerning 3 files. The changes are backwards compatible: if CXX is not provided by an user, the build uses CC instead (as well as CFLAGS instead of CXXFLAGS). Would you be so kind and share your opinion on those? Please find attachment provided bellow (the patch was created using Quilt). Best Regards, Patrik Mada Je dobr? v?d?t, ?e tento e-mail a p??lohy jsou d?v?rn?. Pokud spolu jedn?me o uzav?en? obchodu, vyhrazujeme si pr?vo na?e jedn?n? kdykoli ukon?it. Pro fanou?ky pr?vn? mluvy - vylu?ujeme t?m ustanoven? ob?ansk?ho z?kon?ku o p?edsmluvn? odpov?dnosti. Pravidla o tom, kdo u n?s a jak vystupuje za spole?nost a kdo m??e co a jak podepsat naleznete zde You should know that this e-mail and its attachments are confidential. If we are negotiating on the conclusion of a transaction, we reserve the right to terminate the negotiations at any time. For fans of legalese?we hereby exclude the provisions of the Civil Code on pre-contractual liability. The rules about who and how may act for the company and what are the signing procedures can be found here. -------------- next part -------------- A non-text attachment was scrubbed... Name: accept_cxx_flags.patch Type: application/octet-stream Size: 3417 bytes Desc: not available URL: From arut at nginx.com Fri Feb 28 09:43:59 2020 From: arut at nginx.com (Roman Arutyunyan) Date: Fri, 28 Feb 2020 09:43:59 +0000 Subject: [nginx] Mp4: fixed possible chunk offset overflow. Message-ID: details: https://hg.nginx.org/nginx/rev/f47f7d3d1bfa branches: changeset: 7629:f47f7d3d1bfa user: Roman Arutyunyan date: Wed Feb 26 15:10:46 2020 +0300 description: Mp4: fixed possible chunk offset overflow. In "co64" atom chunk start offset is a 64-bit unsigned integer. When trimming the "mdat" atom, chunk offsets are casted to off_t values which are typically 64-bit signed integers. A specially crafted mp4 file with huge chunk offsets may lead to off_t overflow and result in negative trim boundaries. The consequences of the overflow are: - Incorrect Content-Length header value in the response. - Negative left boundary of the response file buffer holding the trimmed "mdat". This leads to pread()/sendfile() errors followed by closing the client connection. On rare systems where off_t is a 32-bit integer, this scenario is also feasible with the "stco" atom. The fix is to add checks which make sure data chunks referenced by each track are within the mp4 file boundaries. Additionally a few more checks are added to ensure mp4 file consistency and log errors. diffstat: src/http/modules/ngx_http_mp4_module.c | 75 +++++++++++++++++++++++++++++---- 1 files changed, 64 insertions(+), 11 deletions(-) diffs (134 lines): diff -r 2e3bfd696ecb -r f47f7d3d1bfa src/http/modules/ngx_http_mp4_module.c --- a/src/http/modules/ngx_http_mp4_module.c Thu Feb 27 19:03:21 2020 +0300 +++ b/src/http/modules/ngx_http_mp4_module.c Wed Feb 26 15:10:46 2020 +0300 @@ -3116,6 +3116,13 @@ ngx_http_mp4_update_stsz_atom(ngx_http_m "chunk samples sizes:%uL", trak->start_chunk_samples_size); + if (trak->start_chunk_samples_size > (uint64_t) mp4->end) { + ngx_log_error(NGX_LOG_ERR, mp4->file.log, 0, + "too large mp4 start samples size in \"%s\"", + mp4->file.name.data); + return NGX_ERROR; + } + if (mp4->length) { if (trak->end_sample - trak->start_sample > entries) { ngx_log_error(NGX_LOG_ERR, mp4->file.log, 0, @@ -3135,6 +3142,13 @@ ngx_http_mp4_update_stsz_atom(ngx_http_m ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "mp4 stsz end_chunk_samples_size:%uL", trak->end_chunk_samples_size); + + if (trak->end_chunk_samples_size > (uint64_t) mp4->end) { + ngx_log_error(NGX_LOG_ERR, mp4->file.log, 0, + "too large mp4 end samples size in \"%s\"", + mp4->file.name.data); + return NGX_ERROR; + } } atom_size = sizeof(ngx_mp4_stsz_atom_t) + (data->last - data->pos); @@ -3226,6 +3240,7 @@ ngx_http_mp4_update_stco_atom(ngx_http_m { size_t atom_size; uint32_t entries; + uint64_t chunk_offset, samples_size; ngx_buf_t *atom, *data; ngx_mp4_stco_atom_t *stco_atom; @@ -3256,8 +3271,19 @@ ngx_http_mp4_update_stco_atom(ngx_http_m data->pos += trak->start_chunk * sizeof(uint32_t); - trak->start_offset = ngx_mp4_get_32value(data->pos); - trak->start_offset += trak->start_chunk_samples_size; + chunk_offset = ngx_mp4_get_32value(data->pos); + samples_size = trak->start_chunk_samples_size; + + if (chunk_offset > (uint64_t) mp4->end - samples_size + || chunk_offset + samples_size > NGX_MAX_UINT32_VALUE) + { + ngx_log_error(NGX_LOG_ERR, mp4->file.log, 0, + "too large chunk offset in \"%s\"", + mp4->file.name.data); + return NGX_ERROR; + } + + trak->start_offset = chunk_offset + samples_size; ngx_mp4_set_32value(data->pos, trak->start_offset); ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, @@ -3276,9 +3302,19 @@ ngx_http_mp4_update_stco_atom(ngx_http_m data->last = data->pos + entries * sizeof(uint32_t); if (entries) { - trak->end_offset = - ngx_mp4_get_32value(data->last - sizeof(uint32_t)); - trak->end_offset += trak->end_chunk_samples_size; + chunk_offset = ngx_mp4_get_32value(data->last - sizeof(uint32_t)); + samples_size = trak->end_chunk_samples_size; + + if (chunk_offset > (uint64_t) mp4->end - samples_size + || chunk_offset + samples_size > NGX_MAX_UINT32_VALUE) + { + ngx_log_error(NGX_LOG_ERR, mp4->file.log, 0, + "too large chunk offset in \"%s\"", + mp4->file.name.data); + return NGX_ERROR; + } + + trak->end_offset = chunk_offset + samples_size; ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "end chunk offset:%O", trak->end_offset); @@ -3409,7 +3445,7 @@ ngx_http_mp4_update_co64_atom(ngx_http_m ngx_http_mp4_trak_t *trak) { size_t atom_size; - uint64_t entries; + uint64_t entries, chunk_offset, samples_size; ngx_buf_t *atom, *data; ngx_mp4_co64_atom_t *co64_atom; @@ -3440,8 +3476,17 @@ ngx_http_mp4_update_co64_atom(ngx_http_m data->pos += trak->start_chunk * sizeof(uint64_t); - trak->start_offset = ngx_mp4_get_64value(data->pos); - trak->start_offset += trak->start_chunk_samples_size; + chunk_offset = ngx_mp4_get_64value(data->pos); + samples_size = trak->start_chunk_samples_size; + + if (chunk_offset > (uint64_t) mp4->end - samples_size) { + ngx_log_error(NGX_LOG_ERR, mp4->file.log, 0, + "too large chunk offset in \"%s\"", + mp4->file.name.data); + return NGX_ERROR; + } + + trak->start_offset = chunk_offset + samples_size; ngx_mp4_set_64value(data->pos, trak->start_offset); ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, @@ -3460,9 +3505,17 @@ ngx_http_mp4_update_co64_atom(ngx_http_m data->last = data->pos + entries * sizeof(uint64_t); if (entries) { - trak->end_offset = - ngx_mp4_get_64value(data->last - sizeof(uint64_t)); - trak->end_offset += trak->end_chunk_samples_size; + chunk_offset = ngx_mp4_get_64value(data->last - sizeof(uint64_t)); + samples_size = trak->end_chunk_samples_size; + + if (chunk_offset > (uint64_t) mp4->end - samples_size) { + ngx_log_error(NGX_LOG_ERR, mp4->file.log, 0, + "too large chunk offset in \"%s\"", + mp4->file.name.data); + return NGX_ERROR; + } + + trak->end_offset = chunk_offset + samples_size; ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "end chunk offset:%O", trak->end_offset); From xeioex at nginx.com Fri Feb 28 12:15:49 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 28 Feb 2020 12:15:49 +0000 Subject: [njs] Fixed non-native module importing. Message-ID: details: https://hg.nginx.org/njs/rev/ee5fa0d312df branches: changeset: 1341:ee5fa0d312df user: hongzhidao date: Thu Jan 30 21:14:30 2020 +0800 description: Fixed non-native module importing. Previously, string from "import statement" was used as a key in a hash to keep track of already loaded modules. This works fine for built-in modules. It can cause an issue when different modules from different paths but with identical names are loaded. This patch avoids the issue by using a full path to a file as a key. This closes #282 issue on GitHub. diffstat: src/njs_module.c | 20 ++++++++++++++------ test/module/lib1.js | 3 ++- test/module/libs/hash.js | 3 ++- test/module/libs/name.js | 1 + test/module/name.js | 1 + test/module/normal.js | 10 ++++++++++ test/module/recursive.js | 4 +--- test/njs_expect_test.exp | 8 ++++---- 8 files changed, 35 insertions(+), 15 deletions(-) diffs (176 lines): diff -r 3313d0d593a0 -r ee5fa0d312df src/njs_module.c --- a/src/njs_module.c Thu Feb 27 14:30:14 2020 +0300 +++ b/src/njs_module.c Thu Jan 30 21:14:30 2020 +0800 @@ -113,7 +113,7 @@ njs_parser_module(njs_vm_t *vm, njs_pars parser->node = NULL; module = njs_module_find(vm, &name, 0); - if (module != NULL) { + if (module != NULL && module->function.native) { goto found; } @@ -138,18 +138,24 @@ njs_parser_module(njs_vm_t *vm, njs_pars goto fail; } + module = njs_module_find(vm, &info.file, 0); + if (module != NULL) { + (void) close(info.fd); + goto found; + } + ret = njs_module_read(vm, info.fd, &text); (void) close(info.fd); if (njs_slow_path(ret != NJS_OK)) { - njs_internal_error(vm, "while reading \"%V\" module", &name); + njs_internal_error(vm, "while reading \"%V\" module", &info.file); goto fail; } if (njs_module_realpath_equal(&prev->file, &info.file)) { njs_parser_syntax_error(vm, parser, "Cannot import itself \"%V\"", - &name); + &info.file); goto fail; } @@ -171,7 +177,7 @@ njs_parser_module(njs_vm_t *vm, njs_pars goto fail; } - module = njs_module_add(vm, &name); + module = njs_module_add(vm, &info.file); if (njs_slow_path(module == NULL)) { goto fail; } @@ -229,7 +235,8 @@ njs_module_lookup(njs_vm_t *vm, const nj } ret = njs_module_relative_path(vm, cwd, info); - if (ret == NJS_OK) { + + if (ret != NJS_DECLINED) { return ret; } @@ -241,7 +248,8 @@ njs_module_lookup(njs_vm_t *vm, const nj for (i = 0; i < vm->paths->items; i++) { ret = njs_module_relative_path(vm, path, info); - if (ret == NJS_OK) { + + if (ret != NJS_DECLINED) { return ret; } diff -r 3313d0d593a0 -r ee5fa0d312df test/module/lib1.js --- a/test/module/lib1.js Thu Feb 27 14:30:14 2020 +0300 +++ b/test/module/lib1.js Thu Jan 30 21:14:30 2020 +0800 @@ -10,6 +10,7 @@ function hash() { return v; } +import hashlib from 'hash.js'; import crypto from 'crypto'; var state = {count:0} @@ -22,4 +23,4 @@ function get() { return state.count; } -export default {hash, inc, get}; +export default {hash, inc, get, name: hashlib.name} diff -r 3313d0d593a0 -r ee5fa0d312df test/module/libs/hash.js --- a/test/module/libs/hash.js Thu Feb 27 14:30:14 2020 +0300 +++ b/test/module/libs/hash.js Thu Jan 30 21:14:30 2020 +0800 @@ -4,6 +4,7 @@ function hash() { return v; } +import name from 'name.js'; import crypto from 'crypto'; -export default {hash}; +export default {hash, name}; diff -r 3313d0d593a0 -r ee5fa0d312df test/module/libs/name.js --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test/module/libs/name.js Thu Jan 30 21:14:30 2020 +0800 @@ -0,0 +1,1 @@ +export default 'libs.name'; diff -r 3313d0d593a0 -r ee5fa0d312df test/module/name.js --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test/module/name.js Thu Jan 30 21:14:30 2020 +0800 @@ -0,0 +1,1 @@ +export default 'name'; diff -r 3313d0d593a0 -r ee5fa0d312df test/module/normal.js --- a/test/module/normal.js Thu Feb 27 14:30:14 2020 +0300 +++ b/test/module/normal.js Thu Jan 30 21:14:30 2020 +0800 @@ -1,3 +1,4 @@ +import name from 'name.js'; import lib1 from 'lib1.js'; import lib2 from 'lib2.js'; import lib1_2 from 'lib1.js'; @@ -7,6 +8,15 @@ var h = crypto.createHash('md5'); var hash = h.update('AB').digest('hex'); var fails = 0; + +if (name != 'name') { + fails++; +} + +if (lib1.name != 'libs.name') { + fails++; +} + if (lib1.hash() != hash) { fails++; } diff -r 3313d0d593a0 -r ee5fa0d312df test/module/recursive.js --- a/test/module/recursive.js Thu Feb 27 14:30:14 2020 +0300 +++ b/test/module/recursive.js Thu Jan 30 21:14:30 2020 +0800 @@ -1,3 +1,1 @@ - - -import lib from './recursive.js'; +import lib from 'recursive.js'; diff -r 3313d0d593a0 -r ee5fa0d312df test/njs_expect_test.exp --- a/test/njs_expect_test.exp Thu Feb 27 14:30:14 2020 +0300 +++ b/test/njs_expect_test.exp Thu Jan 30 21:14:30 2020 +0800 @@ -757,13 +757,13 @@ njs_run {"-p" "test/module" "-p" "test/m "passed!" njs_run {"./test/module/normal.js"} \ - "SyntaxError: Cannot find module \"hash.js\" in sub2.js:5" + "SyntaxError: Cannot find module \"hash.js\" in lib1.js:13" njs_run {"-p" "test/module/libs" "./test/module/exception.js"} \ "at error \\(sub1.js:5\\)" njs_run {"-p" "test/module" "./test/module/recursive.js"} \ - "SyntaxError: Cannot import itself \"./recursive.js\" in recursive.js:3" + "SyntaxError: Cannot import itself \"./test/module/recursive.js\" in recursive.js:1" # CLI OPTIONS @@ -843,7 +843,7 @@ njs_test { "Error: loading exception\r\n at module \\(loading_exception.js:1\\)"} {"import lib3 from 'lib1.js'\r\n" "undefined\r\n"} -} "-p test/module/" +} "-p test/module/ -p test/module/libs/" njs_test { {"import m from 'export_name.js'\r\n" @@ -861,7 +861,7 @@ njs_test { } "-p test/module/" njs_run {"-q" "./test/module/normal.js"} \ - "SyntaxError: Cannot find module \"hash.js\" in 5" + "SyntaxError: Cannot find module \"hash.js\" in 13" njs_run {"-p" "test/module/libs/" "-d" "./test/module/normal.js"} \ "passed!" From mdounin at mdounin.ru Fri Feb 28 13:25:57 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 28 Feb 2020 16:25:57 +0300 Subject: nginx for Windows - WSASend() socket error 10057 In-Reply-To: References: <5528c341ee8af2cd16fc29fd9814eae5@sebres.de> <20891f45d8b2d29e42b9660acba9ddba@sebres.de> Message-ID: <20200228132557.GJ12894@mdounin.ru> Hello! On Thu, Feb 27, 2020 at 07:40:38PM +0000, Yury Shpakov wrote: > I wrote my own auth_http server. And it was hit only one time for 1000 SMTP requests. > So auth_http is requested once per session/connection. There is no such thing as "SMTP requests". And yes, that's expected that's auth_http is called once per connection: auth_http is only called for authentication, non for each SMTP command. > For this example it's hit once: > > using (var smtpClient = new SmtpClient("localhost", 8025) { Timeout = 60 * 60 * 1000 }) > { > for (int i = 0; i < 1000; i++) > { > smtpClient.Send("noreply at wmata.com", "ys at wmata.com", "Email subject", "Email body"); > } > } > > > For this example it's hit 1000 times: > > for (int i = 0; i < 1000; i++) > { > using (var smtpClient = new SmtpClient("localhost", 8025) { Timeout = 60 * 60 * 1000 }) > { > smtpClient.Send("noreply at wmata.com", "ys at wmata.com", "Email subject", "Email body"); > } > } The difference is that in the first example you establish one connection and use it to send 1000 messages, while in the second example you establish 1000 connections - and each of them calls auth_http. > Evidently by the same reason this one didn't work either: > smtp_greeting_delay 10s; This directive delays SMTP greeting. It is expected to happen only once per connection, see here for an overview of the SMTP protocol: https://tools.ietf.org/html/rfc5321 > This options is not gonna work either: > - Maxim's ngx_http_delay (I used it more for development purposes, like test or simulation of load etc); That's expected as well, as the delay module is to delay HTTP requsets, not SMTP commands. > If I do this I don't need nginx at all. But my boss said: "Use nginx. Period.": > - some "slow" upstream backend that doing nothing, just waiting (preferably asynchronous). > > So the question is still open: how to make nginx wait on SMTP > processing to model latency for my load tests? If you are trying to model latency within a single SMTP connection, the answer is: you can't do this with nginx, as nginx does not try to do anything with an established and authenticated connection SMTP connnection, and doesn't even try to parse what happens on such a connection. It merely proxies bytes between the client and the backend server. Either way, this thread looks like an off-topic for the nginx-devel@ mailing list. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Fri Feb 28 14:21:55 2020 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 28 Feb 2020 14:21:55 +0000 Subject: [nginx] Added default overwrite in error_page 494. Message-ID: details: https://hg.nginx.org/nginx/rev/f001d9384293 branches: changeset: 7630:f001d9384293 user: Maxim Dounin date: Fri Feb 28 17:21:18 2020 +0300 description: Added default overwrite in error_page 494. We used to have default error_page overwrite for 495, 496, and 497, so a configuration like error_page 495 /error; will result in error 400, much like without any error_page configured. The 494 status code was introduced later (in 3848:de59ad6bf557, nginx 0.9.4), and relevant changes to ngx_http_core_error_page() were missed, resulting in inconsistent behaviour of "error_page 494" - with error_page configured it results in 494 being returned instead of 400. Reported by Frank Liu, http://mailman.nginx.org/pipermail/nginx/2020-February/058957.html. diffstat: src/http/ngx_http_core_module.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r f47f7d3d1bfa -r f001d9384293 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Wed Feb 26 15:10:46 2020 +0300 +++ b/src/http/ngx_http_core_module.c Fri Feb 28 17:21:18 2020 +0300 @@ -4687,6 +4687,7 @@ ngx_http_core_error_page(ngx_conf_t *cf, case NGX_HTTP_TO_HTTPS: case NGX_HTTPS_CERT_ERROR: case NGX_HTTPS_NO_CERT: + case NGX_HTTP_REQUEST_HEADER_TOO_LARGE: err->overwrite = NGX_HTTP_BAD_REQUEST; } } From xeioex at nginx.com Fri Feb 28 15:56:46 2020 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Fri, 28 Feb 2020 15:56:46 +0000 Subject: [njs] Fixed njs_date_string(). Message-ID: details: https://hg.nginx.org/njs/rev/3f094214cd64 branches: changeset: 1342:3f094214cd64 user: Dmitry Volyntsev date: Fri Feb 28 18:56:24 2020 +0300 description: Fixed njs_date_string(). This closes #292 issue on Github. diffstat: src/njs_date.c | 2 +- src/test/njs_unit_test.c | 3 +++ 2 files changed, 4 insertions(+), 1 deletions(-) diffs (25 lines): diff -r ee5fa0d312df -r 3f094214cd64 src/njs_date.c --- a/src/njs_date.c Thu Jan 30 21:14:30 2020 +0800 +++ b/src/njs_date.c Fri Feb 28 18:56:24 2020 +0300 @@ -1157,7 +1157,7 @@ njs_date_string(njs_vm_t *vm, njs_value_ "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" }; if (njs_slow_path(isnan(time))) { - vm->retval = njs_string_invalid_date; + *retval = njs_string_invalid_date; return NJS_OK; } diff -r ee5fa0d312df -r 3f094214cd64 src/test/njs_unit_test.c --- a/src/test/njs_unit_test.c Thu Jan 30 21:14:30 2020 +0800 +++ b/src/test/njs_unit_test.c Fri Feb 28 18:56:24 2020 +0300 @@ -13270,6 +13270,9 @@ static njs_unit_test_t njs_test[] = { njs_str("new Date(8.65e15)"), njs_str("Invalid Date") }, + { njs_str("njs.dump([new Date(8.65e15)])"), + njs_str("[Invalid Date]") }, + { njs_str("new Date(0e0.o0)"), njs_str("Invalid Date") },