From mdounin at mdounin.ru  Tue Sep  1 13:19:19 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 1 Sep 2020 16:19:19 +0300
Subject: possible problem with ngx_palloc_small()
In-Reply-To: <CAFPOs6r+byuon4epRa_f=-sxd5o3D-b-x8u=Kpp9zkf8rrw5bA@mail.gmail.com>
References: <CAFPOs6r+byuon4epRa_f=-sxd5o3D-b-x8u=Kpp9zkf8rrw5bA@mail.gmail.com>
Message-ID: <20200901131919.GZ12747@mdounin.ru>

Hello!

On Mon, Aug 31, 2020 at 11:08:13AM -0700, Maksim Yevmenkin wrote:

> Hello,
> 
> a colleague of mine sent me this
> 
> ==
> 
> There is a problem in ngx_palloc_small() if it is called with arg
> 'align' set true when the small buffer is almost exhausted such that
> there are less bytes available in that buffer than the change in
> alignment consumes
> 
> In that case, 'm' (the alignment adjusted start of the remainder of
> the buffer) may move beyond the 'end' marker, meaning that p->d.end -
> m becomes -ve.
> 
> Unfortunately, that subtraction is cast to a size_t (unsigned) and so
> its comparison to '>= size' is very likely true, meaning that the
> p->d.last is advanced beyond p->d.end and so memory already utilised
> is returned. iI that happens to trample over bytes used for say the
> p->large->next...->next chain, then a BUS error is likely
> 
> It seems that this can be addressed by :
> 
>  @@ -160,7 +160,7 @@ ngx_palloc_small(ngx_pool_t *pool, size_t size,
> ngx_uint_t align)
>              m = ngx_align_ptr(m, NGX_ALIGNMENT);
>          }
> 
> -        if ((size_t) (p->d.end - m) >= size) {
> +        if (p->d.end >= (size + m)) {
>              p->d.last = m + size;
> 
>              return m;
> ==
> 
> can someone please share thoughts, comments, etc?

https://trac.nginx.org/nginx/ticket/686

-- 
Maxim Dounin
http://mdounin.ru/

From xeioex at nginx.com  Tue Sep  1 17:14:17 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 01 Sep 2020 17:14:17 +0000
Subject: [njs] Added DataView object.
Message-ID: <hg.d00a6ec5a39a.1598980457.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/d00a6ec5a39a
branches:  
changeset: 1513:d00a6ec5a39a
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Tue Sep 01 16:37:33 2020 +0000
description:
Added DataView object.

diffstat:

 src/njs_builtin.c        |   11 +
 src/njs_object_hash.h    |   12 +
 src/njs_typed_array.c    |  558 ++++++++++++++++++++++++++++++++++++++++++++++-
 src/njs_typed_array.h    |    2 +
 src/njs_utils.h          |   32 ++
 src/njs_value.c          |    1 +
 src/njs_value.h          |   20 +
 src/njs_vm.h             |    3 +-
 src/test/njs_unit_test.c |   86 +++++++
 9 files changed, 721 insertions(+), 4 deletions(-)

diffs (894 lines):

diff -r d3e1f95c0c5c -r d00a6ec5a39a src/njs_builtin.c
--- a/src/njs_builtin.c	Fri Aug 28 11:51:35 2020 +0000
+++ b/src/njs_builtin.c	Tue Sep 01 16:37:33 2020 +0000
@@ -71,6 +71,7 @@ static const njs_object_type_init_t *con
     &njs_date_type_init,
     &njs_promise_type_init,
     &njs_array_buffer_type_init,
+    &njs_data_view_type_init,
     &njs_text_decoder_type_init,
     &njs_text_encoder_type_init,
 
@@ -1285,6 +1286,16 @@ static const njs_object_prop_t  njs_glob
 
     {
         .type = NJS_PROPERTY_HANDLER,
+        .name = njs_string("DataView"),
+        .value = njs_prop_handler2(njs_top_level_constructor,
+                                   NJS_OBJ_TYPE_DATA_VIEW,
+                                   NJS_DATA_VIEW_HASH),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY_HANDLER,
         .name = njs_string("TextDecoder"),
         .value = njs_prop_handler2(njs_top_level_constructor,
                                    NJS_OBJ_TYPE_TEXT_DECODER,
diff -r d3e1f95c0c5c -r d00a6ec5a39a src/njs_object_hash.h
--- a/src/njs_object_hash.h	Fri Aug 28 11:51:35 2020 +0000
+++ b/src/njs_object_hash.h	Tue Sep 01 16:37:33 2020 +0000
@@ -611,6 +611,18 @@
         'A'), 'r'), 'r'), 'a'), 'y'), 'B'), 'u'), 'f'), 'f'), 'e'), 'r')
 
 
+#define NJS_DATA_VIEW_HASH                                                    \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(NJS_DJB_HASH_INIT,                                       \
+        'D'), 'a'), 't'), 'a'), 'V'), 'i'), 'e'), 'w')
+
+
 #define NJS_UINT8ARRAY_HASH                                                   \
     njs_djb_hash_add(                                                         \
     njs_djb_hash_add(                                                         \
diff -r d3e1f95c0c5c -r d00a6ec5a39a src/njs_typed_array.c
--- a/src/njs_typed_array.c	Fri Aug 28 11:51:35 2020 +0000
+++ b/src/njs_typed_array.c	Tue Sep 01 16:37:33 2020 +0000
@@ -1,6 +1,7 @@
 
 /*
  * Copyright (C) Igor Sysoev
+ * Copyright (C) Dmitry Volyntsev
  * Copyright (C) NGINX, Inc.
  */
 
@@ -369,7 +370,7 @@ njs_typed_array_prototype_buffer(njs_vm_
     njs_typed_array_t  *array;
 
     this = njs_argument(args, 0);
-    if (!njs_is_typed_array(this)) {
+    if (!njs_is_typed_array(this) && !njs_is_data_view(this)) {
         njs_type_error(vm, "Method TypedArray.prototype.buffer called "
                        "on incompatible receiver");
         return NJS_ERROR;
@@ -391,7 +392,7 @@ njs_typed_array_prototype_byte_length(nj
     njs_typed_array_t  *array;
 
     this = njs_argument(args, 0);
-    if (!njs_is_typed_array(this)) {
+    if (!njs_is_typed_array(this) && !njs_is_data_view(this)) {
         njs_type_error(vm, "Method TypedArray.prototype.byteLength called "
                        "on incompatible receiver");
         return NJS_ERROR;
@@ -413,7 +414,7 @@ njs_typed_array_prototype_byte_offset(nj
     njs_typed_array_t  *array;
 
     this = njs_argument(args, 0);
-    if (!njs_is_typed_array(this)) {
+    if (!njs_is_typed_array(this) && !njs_is_data_view(this)) {
         njs_type_error(vm, "Method TypedArray.prototype.byteOffset called "
                        "on incompatible receiver");
         return NJS_ERROR;
@@ -2269,6 +2270,557 @@ const njs_object_type_init_t  njs_typed_
 };
 
 
+static njs_int_t
+njs_data_view_constructor(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    uint64_t            size, offset;
+    njs_int_t           ret;
+    njs_data_view_t     *view;
+    njs_array_buffer_t  *buffer;
+
+    if (!vm->top_frame->ctor) {
+        njs_type_error(vm, "Constructor of DataView requires 'new'");
+        return NJS_ERROR;
+    }
+
+    if (!njs_is_array_buffer(njs_arg(args, nargs, 1))) {
+        njs_type_error(vm, "buffer is not an ArrayBuffer");
+        return NJS_ERROR;
+    }
+
+    size = 0;
+    offset = 0;
+
+    buffer = njs_array_buffer(njs_argument(args, 1));
+
+    ret = njs_value_to_index(vm, njs_arg(args, nargs, 2), &offset);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
+    if (!njs_is_undefined(njs_arg(args, nargs, 3))) {
+        ret = njs_value_to_index(vm, njs_argument(args, 3), &size);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_ERROR;
+        }
+
+        if (njs_slow_path((offset + size) > buffer->size)) {
+            njs_range_error(vm, "Invalid DataView length: %uL", size);
+            return NJS_ERROR;
+        }
+
+    } else {
+        if (offset > buffer->size) {
+            njs_range_error(vm, "byteOffset %uL is outside the bound of "
+                            "the buffer", offset);
+            return NJS_ERROR;
+        }
+
+        size = buffer->size - offset;
+    }
+
+    view = njs_mp_zalloc(vm->mem_pool, sizeof(njs_data_view_t));
+    if (njs_slow_path(view == NULL)) {
+        goto memory_error;
+    }
+
+    view->buffer = buffer;
+    view->offset = offset;
+    view->byte_length = size;
+    view->type = NJS_OBJ_TYPE_DATA_VIEW;
+
+    njs_lvlhsh_init(&view->object.hash);
+    njs_lvlhsh_init(&view->object.shared_hash);
+    view->object.__proto__ = &vm->prototypes[view->type].object;
+    view->object.type = NJS_DATA_VIEW;
+    view->object.extensible = 1;
+
+    njs_set_data_view(&vm->retval, view);
+
+    return NJS_OK;
+
+memory_error:
+
+    njs_memory_error(vm);
+
+    return NJS_ERROR;
+}
+
+
+static const njs_object_prop_t  njs_data_view_constructor_props[] =
+{
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("name"),
+        .value = njs_string("DataView"),
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("length"),
+        .value = njs_value(NJS_NUMBER, 1, 1.0),
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY_HANDLER,
+        .name = njs_string("prototype"),
+        .value = njs_prop_handler(njs_object_prototype_create),
+    },
+};
+
+
+static const njs_object_init_t  njs_data_view_constructor_init = {
+    njs_data_view_constructor_props,
+    njs_nitems(njs_data_view_constructor_props),
+};
+
+
+typedef union {
+    float       f;
+    uint32_t    u;
+} njs_conv_f32_t;
+
+
+typedef union {
+    double      f;
+    uint64_t    u;
+} njs_conv_f64_t;
+
+
+static njs_int_t
+njs_data_view_prototype_get(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t type)
+{
+    double              v;
+    uint32_t            u32;
+    uint64_t            index;
+    njs_int_t           ret;
+    njs_bool_t          swap;
+    njs_value_t         *this;
+    const uint8_t       *u8;
+    njs_conv_f32_t      conv_f32;
+    njs_conv_f64_t      conv_f64;
+    njs_data_view_t     *view;
+    njs_array_buffer_t  *buffer;
+
+    this = njs_argument(args, 0);
+    if (njs_slow_path(!njs_is_data_view(this))) {
+        njs_type_error(vm, "this is not a DataView");
+        return NJS_ERROR;
+    }
+
+    ret = njs_value_to_index(vm, njs_arg(args, nargs, 1), &index);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
+    swap = njs_bool(njs_arg(args, nargs, 2));
+
+#if NJS_HAVE_LITTLE_ENDIAN
+    swap = !swap;
+#endif
+
+    view = njs_data_view(this);
+
+    if (njs_typed_array_element_size(type) + index > view->byte_length) {
+        njs_range_error(vm, "index %uL is outside the bound of the buffer",
+                        index);
+        return NJS_ERROR;
+    }
+
+    buffer = view->buffer;
+    u8 = &buffer->u.u8[index + view->offset];
+
+    switch (type) {
+    case NJS_OBJ_TYPE_UINT8_ARRAY:
+        v = *u8;
+        break;
+
+    case NJS_OBJ_TYPE_INT8_ARRAY:
+        v = (int8_t) *u8;
+        break;
+
+    case NJS_OBJ_TYPE_UINT16_ARRAY:
+        u32 = *((uint16_t *) u8);
+
+        if (swap) {
+            u32 = njs_bswap_u16(u32);
+        }
+
+        v = u32;
+        break;
+
+    case NJS_OBJ_TYPE_INT16_ARRAY:
+        u32 = *((uint16_t *) u8);
+
+        if (swap) {
+            u32 = njs_bswap_u16(u32);
+        }
+
+        v = (int16_t) u32;
+        break;
+
+    case NJS_OBJ_TYPE_UINT32_ARRAY:
+    case NJS_OBJ_TYPE_INT32_ARRAY:
+    case NJS_OBJ_TYPE_FLOAT32_ARRAY:
+        u32 = *((uint32_t *) u8);
+
+        if (swap) {
+            u32 = njs_bswap_u32(u32);
+        }
+
+        switch (type) {
+        case NJS_OBJ_TYPE_UINT32_ARRAY:
+            v = u32;
+            break;
+
+        case NJS_OBJ_TYPE_INT32_ARRAY:
+            v = (int32_t) u32;
+            break;
+
+        default:
+            conv_f32.u = u32;
+            v = conv_f32.f;
+        }
+
+        break;
+
+    default:
+        /* NJS_OBJ_TYPE_FLOAT64_ARRAY. */
+
+        conv_f64.u = *((uint64_t *) u8);
+
+        if (swap) {
+            conv_f64.u = njs_bswap_u64(conv_f64.u);
+        }
+
+        v = conv_f64.f;
+    }
+
+    njs_set_number(&vm->retval, v);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_data_view_prototype_set(njs_vm_t *vm, njs_value_t *args,
+    njs_uint_t nargs, njs_index_t type)
+{
+    double              v;
+    uint8_t             *u8;
+    uint32_t            u32;
+    uint64_t            index;
+    njs_int_t           ret;
+    njs_bool_t          swap;
+    njs_value_t         *this;
+    njs_conv_f32_t      conv_f32;
+    njs_conv_f64_t      conv_f64;
+    njs_data_view_t     *view;
+    njs_array_buffer_t  *buffer;
+
+    this = njs_argument(args, 0);
+    if (njs_slow_path(!njs_is_data_view(this))) {
+        njs_type_error(vm, "this is not a DataView");
+        return NJS_ERROR;
+    }
+
+    ret = njs_value_to_index(vm, njs_arg(args, nargs, 1), &index);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
+    ret = njs_value_to_number(vm, njs_arg(args, nargs, 2), &v);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
+    swap = njs_bool(njs_arg(args, nargs, 3));
+
+#if NJS_HAVE_LITTLE_ENDIAN
+    swap = !swap;
+#endif
+
+    view = njs_data_view(this);
+
+    if (njs_typed_array_element_size(type) + index > view->byte_length) {
+        njs_range_error(vm, "index %uL is outside the bound of the buffer",
+                        index);
+        return NJS_ERROR;
+    }
+
+    buffer = view->buffer;
+    u8 = &buffer->u.u8[index + view->offset];
+
+    switch (type) {
+    case NJS_OBJ_TYPE_UINT8_ARRAY:
+    case NJS_OBJ_TYPE_INT8_ARRAY:
+        *u8 = njs_number_to_int32(v);
+        break;
+
+    case NJS_OBJ_TYPE_UINT16_ARRAY:
+    case NJS_OBJ_TYPE_INT16_ARRAY:
+        u32 = (uint16_t) njs_number_to_int32(v);
+
+        if (swap) {
+            u32 = njs_bswap_u16(u32);
+        }
+
+        *((uint16_t *) u8) = u32;
+        break;
+
+    case NJS_OBJ_TYPE_UINT32_ARRAY:
+    case NJS_OBJ_TYPE_INT32_ARRAY:
+        u32 = njs_number_to_int32(v);
+
+        if (swap) {
+            u32 = njs_bswap_u32(u32);
+        }
+
+        *((uint32_t *) u8) = u32;
+        break;
+
+    case NJS_OBJ_TYPE_FLOAT32_ARRAY:
+        conv_f32.f = (float) v;
+
+        if (swap) {
+            conv_f32.u = njs_bswap_u32(conv_f32.u);
+        }
+
+        *((uint32_t *) u8) = conv_f32.u;
+        break;
+
+    default:
+        /* NJS_OBJ_TYPE_FLOAT64_ARRAY. */
+
+        conv_f64.f = v;
+
+        if (swap) {
+            conv_f64.u = njs_bswap_u64(conv_f64.u);
+        }
+
+        *((uint64_t *) u8) = conv_f64.u;
+    }
+
+    njs_set_undefined(&vm->retval);
+
+    return NJS_OK;
+}
+
+
+static const njs_object_prop_t  njs_data_view_prototype_properties[] =
+{
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_wellknown_symbol(NJS_SYMBOL_TO_STRING_TAG),
+        .value = njs_string("DataView"),
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY_HANDLER,
+        .name = njs_string("constructor"),
+        .value = njs_prop_handler(njs_object_prototype_create_constructor),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("buffer"),
+        .value = njs_value(NJS_INVALID, 1, NAN),
+        .getter = njs_native_function(njs_typed_array_prototype_buffer, 0),
+        .setter = njs_value(NJS_UNDEFINED, 0, NAN),
+        .writable = NJS_ATTRIBUTE_UNSET,
+        .configurable = 1,
+        .enumerable = 0,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("byteLength"),
+        .value = njs_value(NJS_INVALID, 1, NAN),
+        .getter = njs_native_function(njs_typed_array_prototype_byte_length, 0),
+        .setter = njs_value(NJS_UNDEFINED, 0, NAN),
+        .writable = NJS_ATTRIBUTE_UNSET,
+        .configurable = 1,
+        .enumerable = 0,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("byteOffset"),
+        .value = njs_value(NJS_INVALID, 1, NAN),
+        .getter = njs_native_function(njs_typed_array_prototype_byte_offset, 0),
+        .setter = njs_value(NJS_UNDEFINED, 0, NAN),
+        .writable = NJS_ATTRIBUTE_UNSET,
+        .configurable = 1,
+        .enumerable = 0,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("getUint8"),
+        .value = njs_native_function2(njs_data_view_prototype_get, 1,
+                                      NJS_OBJ_TYPE_UINT8_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("getInt8"),
+        .value = njs_native_function2(njs_data_view_prototype_get, 1,
+                                      NJS_OBJ_TYPE_INT8_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("getUint16"),
+        .value = njs_native_function2(njs_data_view_prototype_get, 1,
+                                      NJS_OBJ_TYPE_UINT16_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("getInt16"),
+        .value = njs_native_function2(njs_data_view_prototype_get, 1,
+                                      NJS_OBJ_TYPE_INT16_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("getUint32"),
+        .value = njs_native_function2(njs_data_view_prototype_get, 1,
+                                      NJS_OBJ_TYPE_UINT32_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("getInt32"),
+        .value = njs_native_function2(njs_data_view_prototype_get, 1,
+                                      NJS_OBJ_TYPE_INT32_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("getFloat32"),
+        .value = njs_native_function2(njs_data_view_prototype_get, 1,
+                                      NJS_OBJ_TYPE_FLOAT32_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("getFloat64"),
+        .value = njs_native_function2(njs_data_view_prototype_get, 1,
+                                      NJS_OBJ_TYPE_FLOAT64_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("setUint8"),
+        .value = njs_native_function2(njs_data_view_prototype_set, 2,
+                                      NJS_OBJ_TYPE_UINT8_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("setInt8"),
+        .value = njs_native_function2(njs_data_view_prototype_set, 2,
+                                      NJS_OBJ_TYPE_INT8_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("setUint16"),
+        .value = njs_native_function2(njs_data_view_prototype_set, 2,
+                                      NJS_OBJ_TYPE_UINT16_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("setInt16"),
+        .value = njs_native_function2(njs_data_view_prototype_set, 2,
+                                      NJS_OBJ_TYPE_INT16_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("setUint32"),
+        .value = njs_native_function2(njs_data_view_prototype_set, 2,
+                                      NJS_OBJ_TYPE_UINT32_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("setInt32"),
+        .value = njs_native_function2(njs_data_view_prototype_set, 2,
+                                      NJS_OBJ_TYPE_INT32_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("setFloat32"),
+        .value = njs_native_function2(njs_data_view_prototype_set, 2,
+                                      NJS_OBJ_TYPE_FLOAT32_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("setFloat64"),
+        .value = njs_native_function2(njs_data_view_prototype_set, 2,
+                                      NJS_OBJ_TYPE_FLOAT64_ARRAY),
+        .writable = 1,
+        .configurable = 1,
+    },
+};
+
+
+static const njs_object_init_t  njs_data_view_prototype_init = {
+    njs_data_view_prototype_properties,
+    njs_nitems(njs_data_view_prototype_properties),
+};
+
+
+const njs_object_type_init_t  njs_data_view_type_init = {
+    .constructor = njs_native_ctor(njs_data_view_constructor, 1, 0),
+    .prototype_props = &njs_data_view_prototype_init,
+    .constructor_props = &njs_data_view_constructor_init,
+    .prototype_value = { .object = { .type = NJS_OBJECT } },
+};
+
+
 static const njs_object_prop_t  njs_typed_array_u8_constructor_props[] =
 {
     {
diff -r d3e1f95c0c5c -r d00a6ec5a39a src/njs_typed_array.h
--- a/src/njs_typed_array.h	Fri Aug 28 11:51:35 2020 +0000
+++ b/src/njs_typed_array.h	Tue Sep 01 16:37:33 2020 +0000
@@ -24,6 +24,7 @@ njs_inline unsigned
 njs_typed_array_element_size(njs_object_type_t type)
 {
     switch (type) {
+    case NJS_OBJ_TYPE_DATA_VIEW:
     case NJS_OBJ_TYPE_UINT8_ARRAY:
     case NJS_OBJ_TYPE_UINT8_CLAMPED_ARRAY:
     case NJS_OBJ_TYPE_INT8_ARRAY:
@@ -96,6 +97,7 @@ njs_typed_array_prop(const njs_typed_arr
 
 
 extern const njs_object_type_init_t  njs_typed_array_type_init;
+extern const njs_object_type_init_t  njs_data_view_type_init;
 extern const njs_object_type_init_t  njs_typed_array_u8_type_init;
 extern const njs_object_type_init_t  njs_typed_array_u8clamped_type_init;
 extern const njs_object_type_init_t  njs_typed_array_i8_type_init;
diff -r d3e1f95c0c5c -r d00a6ec5a39a src/njs_utils.h
--- a/src/njs_utils.h	Fri Aug 28 11:51:35 2020 +0000
+++ b/src/njs_utils.h	Tue Sep 01 16:37:33 2020 +0000
@@ -72,4 +72,36 @@ njs_swap_u64(void *a, void *b, size_t si
 }
 
 
+njs_inline uint16_t
+njs_bswap_u16(uint16_t u16)
+{
+    return (u16 >> 8)
+           | (u16 << 8);
+}
+
+
+njs_inline uint32_t
+njs_bswap_u32(uint32_t u32)
+{
+    return ((u32 & 0xff000000) >> 24)
+           | ((u32 & 0x00ff0000) >> 8)
+           | ((u32 & 0x0000ff00) << 8)
+           | ((u32 & 0x000000ff) << 24);
+}
+
+
+njs_inline uint64_t
+njs_bswap_u64(uint64_t u64)
+{
+    return ((u64 & 0xff00000000000000ULL) >> 56)
+           | ((u64 & 0x00ff000000000000ULL) >> 40)
+           | ((u64 & 0x0000ff0000000000ULL) >> 24)
+           | ((u64 & 0x000000ff00000000ULL) >> 8)
+           | ((u64 & 0x00000000ff000000ULL) << 8)
+           | ((u64 & 0x0000000000ff0000ULL) << 24)
+           | ((u64 & 0x000000000000ff00ULL) << 40)
+           | ((u64 & 0x00000000000000ffULL) << 56);
+}
+
+
 #endif /* _NJS_UTILS_H_INCLUDED_ */
diff -r d3e1f95c0c5c -r d00a6ec5a39a src/njs_value.c
--- a/src/njs_value.c	Fri Aug 28 11:51:35 2020 +0000
+++ b/src/njs_value.c	Tue Sep 01 16:37:33 2020 +0000
@@ -524,6 +524,7 @@ njs_property_query(njs_vm_t *vm, njs_pro
     case NJS_OBJECT:
     case NJS_ARRAY:
     case NJS_ARRAY_BUFFER:
+    case NJS_DATA_VIEW:
     case NJS_TYPED_ARRAY:
     case NJS_OBJECT_BOOLEAN:
     case NJS_OBJECT_NUMBER:
diff -r d3e1f95c0c5c -r d00a6ec5a39a src/njs_value.h
--- a/src/njs_value.h	Fri Aug 28 11:51:35 2020 +0000
+++ b/src/njs_value.h	Tue Sep 01 16:37:33 2020 +0000
@@ -72,6 +72,7 @@ typedef enum {
     NJS_PROMISE,
     NJS_OBJECT_VALUE,
     NJS_ARRAY_BUFFER,
+    NJS_DATA_VIEW,
     NJS_VALUE_TYPE_MAX
 } njs_value_type_t;
 
@@ -95,6 +96,7 @@ typedef struct njs_regexp_pattern_s   nj
 typedef struct njs_array_s            njs_array_t;
 typedef struct njs_array_buffer_s     njs_array_buffer_t;
 typedef struct njs_typed_array_s      njs_typed_array_t;
+typedef struct njs_typed_array_s      njs_data_view_t;
 typedef struct njs_regexp_s           njs_regexp_t;
 typedef struct njs_date_s             njs_date_t;
 typedef struct njs_object_value_s     njs_promise_t;
@@ -141,6 +143,7 @@ union njs_value_s {
             njs_array_t               *array;
             njs_array_buffer_t        *array_buffer;
             njs_typed_array_t         *typed_array;
+            njs_data_view_t           *data_view;
             njs_object_value_t        *object_value;
             njs_function_t            *function;
             njs_function_lambda_t     *lambda;
@@ -656,6 +659,10 @@ typedef struct {
     ((value)->type == NJS_TYPED_ARRAY)
 
 
+#define njs_is_data_view(value)                                               \
+    ((value)->type == NJS_DATA_VIEW)
+
+
 #define njs_is_typed_array_uint8(value)                                       \
     (njs_is_typed_array(value)                                                \
      && njs_typed_array(value)->type == NJS_OBJ_TYPE_UINT8_ARRAY)
@@ -733,6 +740,10 @@ typedef struct {
     ((value)->data.u.array_buffer)
 
 
+#define njs_data_view(value)                                                  \
+    ((value)->data.u.data_view)
+
+
 #define njs_typed_array(value)                                                \
     ((value)->data.u.typed_array)
 
@@ -929,6 +940,15 @@ njs_set_typed_array(njs_value_t *value, 
 
 
 njs_inline void
+njs_set_data_view(njs_value_t *value, njs_data_view_t *array)
+{
+    value->data.u.data_view = array;
+    value->type = NJS_DATA_VIEW;
+    value->data.truth = 1;
+}
+
+
+njs_inline void
 njs_set_function(njs_value_t *value, njs_function_t *function)
 {
     value->data.u.function = function;
diff -r d3e1f95c0c5c -r d00a6ec5a39a src/njs_vm.h
--- a/src/njs_vm.h	Fri Aug 28 11:51:35 2020 +0000
+++ b/src/njs_vm.h	Tue Sep 01 16:37:33 2020 +0000
@@ -84,11 +84,12 @@ typedef enum {
     NJS_OBJ_TYPE_DATE,
     NJS_OBJ_TYPE_PROMISE,
     NJS_OBJ_TYPE_ARRAY_BUFFER,
+    NJS_OBJ_TYPE_DATA_VIEW,
     NJS_OBJ_TYPE_TEXT_DECODER,
     NJS_OBJ_TYPE_TEXT_ENCODER,
 
+#define NJS_OBJ_TYPE_HIDDEN_MIN    (NJS_OBJ_TYPE_FS_DIRENT)
     NJS_OBJ_TYPE_FS_DIRENT,
-#define NJS_OBJ_TYPE_HIDDEN_MIN    (NJS_OBJ_TYPE_FS_DIRENT)
     NJS_OBJ_TYPE_CRYPTO_HASH,
     NJS_OBJ_TYPE_CRYPTO_HMAC,
     NJS_OBJ_TYPE_TYPED_ARRAY,
diff -r d3e1f95c0c5c -r d00a6ec5a39a src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Fri Aug 28 11:51:35 2020 +0000
+++ b/src/test/njs_unit_test.c	Tue Sep 01 16:37:33 2020 +0000
@@ -6232,6 +6232,92 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("(new Float64Array([255,255,NaN,3,NaN,Infinity,3,-Infinity,0,-0,2,1,-5])).slice(2).sort()"),
       njs_str("-Infinity,-5,0,0,1,2,3,3,Infinity,NaN,NaN") },
 
+    { njs_str("(new DataView(new ArrayBuffer(3)))"),
+      njs_str("[object DataView]") },
+
+    { njs_str("(new DataView(new ArrayBuffer(3))).buffer"),
+      njs_str("[object ArrayBuffer]") },
+
+    { njs_str("(new DataView(new ArrayBuffer(3))).byteLength"),
+      njs_str("3") },
+
+    { njs_str("(new DataView(new ArrayBuffer(3), 1)).byteLength"),
+      njs_str("2") },
+
+    { njs_str("(new DataView(new ArrayBuffer(3), 3)).byteLength"),
+      njs_str("0") },
+
+    { njs_str("(new DataView(new ArrayBuffer(3), 1)).byteOffset"),
+      njs_str("1") },
+
+    { njs_str("(new DataView(new ArrayBuffer(3), 1, 1)).byteLength"),
+      njs_str("1") },
+
+    { njs_str("(new DataView(new ArrayBuffer(3), 4))"),
+      njs_str("RangeError: byteOffset 4 is outside the bound of the buffer") },
+
+    { njs_str("(new DataView(new ArrayBuffer(3), 1,3))"),
+      njs_str("RangeError: Invalid DataView length: 3") },
+
+    { njs_str("var u8 = new Uint8Array([255, 129, 130, 131, 4, 5, 6, 7, 8, 9, 255]); "
+              "var dv = new DataView(u8.buffer, 1); "
+              "['getUint8', 'getInt8',"
+              " 'getUint16', 'getInt16',"
+              " 'getUint32', 'getInt32',"
+              " 'getFloat32','getFloat64'"
+              "]"
+              ".map(fn => [dv[fn](0), dv[fn](0,1), dv[fn](1), dv[fn](1,1)])"),
+      njs_str("129,129,130,130,"
+              "-127,-127,-126,-126,"
+              "33154,33409,33411,33666,"
+              "-32382,-32127,-32125,-31870,"
+              "2172814084,75727489,2189624325,84181890,"
+              "-2122153212,75727489,-2105342971,84181890,"
+              "-4.794245620412925e-38,3.091780090135418e-36,-1.9251027092506622e-37,6.230764342760857e-36,"
+              "-2.159546358334202e-301,5.447603729090798e-270,-1.4538065947240604e-296,3.72581468952343e-265") },
+
+    { njs_str("var u8 = new Uint8Array(10);"
+              "var dv = new DataView(u8.buffer, 1);"
+              "var u8view = new Uint8Array(u8.buffer, 1);"
+              "function run(test) {"
+              "     var fn = test[0];"
+              "     var val = test[1];"
+              "     var size = parseInt(fn.match(/\\d+/)) / 8;"
+              "     "
+              "     return  [[0], [0,1],[1], [1,1]].map(args => {"
+              "           var offset = args[0];"
+              "           var le = args[1];"
+              "           u8.fill(0); "
+              "           dv[fn].apply(dv, [offset, val, le]);"
+              "           return `[${u8view.subarray(0, offset + size)}]`;"
+              "     })"
+              "};"
+              "["
+              " ['setUint8',    129],"
+              " ['setInt8',    -127],"
+              " ['setUint16', 33154],"
+              " ['setInt16', -32382],"
+              " ['setUint32', 2172814084],"
+              " ['setInt32', -2122153212],"
+              " ['setFloat32', -4.794245620412925e-38],"
+              " ['setFloat64', -2.159546358334202e-301],"
+              "]"
+              ".map(t => run(t))"),
+      njs_str("[129],[129],[0,129],[0,129],"
+              "[129],[129],[0,129],[0,129],"
+              "[129,130],[130,129],[0,129,130],[0,130,129],"
+              "[129,130],[130,129],[0,129,130],[0,130,129],"
+              "[129,130,131,4],[4,131,130,129],[0,129,130,131,4],[0,4,131,130,129],"
+              "[129,130,131,4],[4,131,130,129],[0,129,130,131,4],[0,4,131,130,129],"
+              "[129,130,131,4],[4,131,130,129],[0,129,130,131,4],[0,4,131,130,129],"
+              "[129,130,131,4,5,6,7,8],[8,7,6,5,4,131,130,129],[0,129,130,131,4,5,6,7,8],[0,8,7,6,5,4,131,130,129]"
+              ) },
+
+    { njs_str("var u8 = new Uint8Array([1,2,3]); "
+              "var dv = new DataView(u8.buffer); "
+              "dv.getUint16(2)"),
+      njs_str("RangeError: index 2 is outside the bound of the buffer") },
+
 #if NJS_HAVE_LARGE_STACK
     { njs_str("var o = Object({length: 3});"
                  "Object.defineProperty(o, '0', {set: function(v){this[0] = 2 * v}});"

From xeioex at nginx.com  Tue Sep  1 17:32:51 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 01 Sep 2020 17:32:51 +0000
Subject: [njs] Fixed RegExpBuiltinExec() with global flag and byte-strings.
Message-ID: <hg.e03701b6e8aa.1598981571.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/e03701b6e8aa
branches:  
changeset: 1514:e03701b6e8aa
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Tue Sep 01 17:25:33 2020 +0000
description:
Fixed RegExpBuiltinExec() with global flag and byte-strings.

The issue was introduced in f9082cd59ba6 (0.4.2).  Since 1c729f765cfb
(0.4.2) RegExp.prototype[Symbol.replace] with a regexp with the global
flag may result in an endless loop for byte-strings with broken UTF8
encoding.

diffstat:

 src/njs_regexp.c         |  20 +++++++++++++++++---
 src/test/njs_unit_test.c |  14 ++++++++++++--
 2 files changed, 29 insertions(+), 5 deletions(-)

diffs (70 lines):

diff -r d00a6ec5a39a -r e03701b6e8aa src/njs_regexp.c
--- a/src/njs_regexp.c	Tue Sep 01 16:37:33 2020 +0000
+++ b/src/njs_regexp.c	Tue Sep 01 17:25:33 2020 +0000
@@ -929,6 +929,7 @@ njs_regexp_exec_result(njs_vm_t *vm, njs
     int                   *captures;
     u_char                *start;
     int32_t               size, length;
+    uint32_t              index;
     njs_int_t             ret;
     njs_uint_t            i, n;
     njs_array_t           *array;
@@ -982,11 +983,24 @@ njs_regexp_exec_result(njs_vm_t *vm, njs
         goto fail;
     }
 
-    njs_set_number(&prop->value, njs_string_index(string, captures[0]));
+    if (type == NJS_REGEXP_UTF8) {
+        index = njs_string_index(string, captures[0]);
+
+    } else {
+        index = captures[0];
+    }
+
+    njs_set_number(&prop->value, index);
 
     if (pattern->global) {
-        njs_set_number(&regexp->last_index,
-                       njs_string_index(string, captures[1]));
+        if (type == NJS_REGEXP_UTF8) {
+            index = njs_string_index(string, captures[1]);
+
+        } else {
+            index = captures[1];
+        }
+
+        njs_set_number(&regexp->last_index, index);
     }
 
     lhq.key_hash = NJS_INDEX_HASH;
diff -r d00a6ec5a39a -r e03701b6e8aa src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Tue Sep 01 16:37:33 2020 +0000
+++ b/src/test/njs_unit_test.c	Tue Sep 01 17:25:33 2020 +0000
@@ -8307,6 +8307,12 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("RegExp.prototype[Symbol.replace].call(/b/, 'abc','B')"),
       njs_str("aBc") },
 
+    { njs_str("String.bytesFrom([253,242,141,10]).replace(/\\s/g, 'X')[3]"),
+      njs_str("X") },
+
+    { njs_str("String.bytesFrom([255,149,15,97,95]).replace(/_/g, 'X')[4]"),
+      njs_str("X") },
+
     { njs_str("/]/"),
       njs_str("/\\]/") },
 
@@ -10327,8 +10333,12 @@ static njs_unit_test_t  njs_test[] =
 #endif
 
     { njs_str("var r = /\\x80/g; r.exec('\\u0081\\u0080'.toBytes());"
-                 "r.lastIndex +' '+ r.source +' '+ r.source.length +' '+ r"),
-      njs_str("1 \\x80 4 /\\x80/g") },
+              "r.lastIndex +' '+ r.source +' '+ r.source.length +' '+ r"),
+      njs_str("2 \\x80 4 /\\x80/g") },
+
+    { njs_str("var r = /_/g; var index = r.exec(String.bytesFrom([255,149,15,97,95])).index;"
+              "[index, r.lastIndex]"),
+      njs_str("4,5") },
 
     { njs_str("var descs = Object.getOwnPropertyDescriptors(RegExp('a'));"
               "Object.keys(descs)"),

From pluknet at nginx.com  Wed Sep  2 20:24:28 2020
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Wed, 02 Sep 2020 20:24:28 +0000
Subject: [nginx] HTTP/2: rejecting invalid stream identifiers with
 PROTOCOL_ERROR.
Message-ID: <hg.da5e3f5b1673.1599078268.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/da5e3f5b1673
branches:  
changeset: 7703:da5e3f5b1673
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Wed Sep 02 23:13:36 2020 +0300
description:
HTTP/2: rejecting invalid stream identifiers with PROTOCOL_ERROR.

Prodded by Xu Yang.

diffstat:

 src/http/v2/ngx_http_v2.c |  34 +++++++++++++++++++++++++++++++---
 1 files changed, 31 insertions(+), 3 deletions(-)

diffs (72 lines):

diff -r 7015f26aef90 -r da5e3f5b1673 src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Wed Jul 29 13:28:04 2020 +0300
+++ b/src/http/v2/ngx_http_v2.c	Wed Sep 02 23:13:36 2020 +0300
@@ -953,6 +953,13 @@ ngx_http_v2_state_data(ngx_http_v2_conne
     ngx_log_debug0(NGX_LOG_DEBUG_HTTP, h2c->connection->log, 0,
                    "http2 DATA frame");
 
+    if (h2c->state.sid == 0) {
+        ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
+                      "client sent DATA frame with incorrect identifier");
+
+        return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_PROTOCOL_ERROR);
+    }
+
     if (size > h2c->recv_window) {
         ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
                       "client violated connection flow control: "
@@ -2095,6 +2102,16 @@ static u_char *
 ngx_http_v2_state_settings(ngx_http_v2_connection_t *h2c, u_char *pos,
     u_char *end)
 {
+    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, h2c->connection->log, 0,
+                   "http2 SETTINGS frame");
+
+    if (h2c->state.sid) {
+        ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
+                      "client sent SETTINGS frame with incorrect identifier");
+
+        return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_PROTOCOL_ERROR);
+    }
+
     if (h2c->state.flags == NGX_HTTP_V2_ACK_FLAG) {
 
         if (h2c->state.length != 0) {
@@ -2118,9 +2135,6 @@ ngx_http_v2_state_settings(ngx_http_v2_c
         return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_SIZE_ERROR);
     }
 
-    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, h2c->connection->log, 0,
-                   "http2 SETTINGS frame");
-
     return ngx_http_v2_state_settings_params(h2c, pos, end);
 }
 
@@ -2269,6 +2283,13 @@ ngx_http_v2_state_ping(ngx_http_v2_conne
     ngx_log_debug0(NGX_LOG_DEBUG_HTTP, h2c->connection->log, 0,
                    "http2 PING frame");
 
+    if (h2c->state.sid) {
+        ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
+                      "client sent PING frame with incorrect identifier");
+
+        return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_PROTOCOL_ERROR);
+    }
+
     if (h2c->state.flags & NGX_HTTP_V2_ACK_FLAG) {
         return ngx_http_v2_state_skip(h2c, pos, end);
     }
@@ -2310,6 +2331,13 @@ ngx_http_v2_state_goaway(ngx_http_v2_con
         return ngx_http_v2_state_save(h2c, pos, end, ngx_http_v2_state_goaway);
     }
 
+    if (h2c->state.sid) {
+        ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
+                      "client sent GOAWAY frame with incorrect identifier");
+
+        return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_PROTOCOL_ERROR);
+    }
+
 #if (NGX_DEBUG)
     h2c->state.length -= NGX_HTTP_V2_GOAWAY_SIZE;
 

From xeioex at nginx.com  Thu Sep  3 13:30:50 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 03 Sep 2020 13:30:50 +0000
Subject: [njs] Fixed RegExp.prototype[Symbol.replace] when replace val is
 function.
Message-ID: <hg.2bae88c33583.1599139850.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/2bae88c33583
branches:  
changeset: 1515:2bae88c33583
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Thu Sep 03 13:30:15 2020 +0000
description:
Fixed RegExp.prototype[Symbol.replace] when replace val is function.

Previously, a custom function received garbage value as the first
argument.

diffstat:

 src/njs_regexp.c         |  1 +
 src/test/njs_unit_test.c |  4 ++++
 2 files changed, 5 insertions(+), 0 deletions(-)

diffs (25 lines):

diff -r e03701b6e8aa -r 2bae88c33583 src/njs_regexp.c
--- a/src/njs_regexp.c	Tue Sep 01 17:25:33 2020 +0000
+++ b/src/njs_regexp.c	Thu Sep 03 13:30:15 2020 +0000
@@ -1347,6 +1347,7 @@ njs_regexp_prototype_symbol_replace(njs_
             array = njs_array(r);
 
             arguments = array->start;
+            arguments[0] = matched;
             ncaptures = array->length;
 
             for (n = 1; n < ncaptures; n++) {
diff -r e03701b6e8aa -r 2bae88c33583 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Tue Sep 01 17:25:33 2020 +0000
+++ b/src/test/njs_unit_test.c	Thu Sep 03 13:30:15 2020 +0000
@@ -8307,6 +8307,10 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("RegExp.prototype[Symbol.replace].call(/b/, 'abc','B')"),
       njs_str("aBc") },
 
+    { njs_str("var m; var r = /./; r.exec = function() { return []; };"
+              "r[Symbol.replace]('foo', function() {m = arguments[0]}); [m, typeof m]"),
+      njs_str("undefined,string") },
+
     { njs_str("String.bytesFrom([253,242,141,10]).replace(/\\s/g, 'X')[3]"),
       njs_str("X") },
 

From xeioex at nginx.com  Thu Sep  3 13:30:52 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 03 Sep 2020 13:30:52 +0000
Subject: [njs] Fixed TypedArraySpeciesCreate().
Message-ID: <hg.95faab343e26.1599139852.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/95faab343e26
branches:  
changeset: 1516:95faab343e26
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Thu Sep 03 13:30:16 2020 +0000
description:
Fixed TypedArraySpeciesCreate().

According to the spec, it is expected to verify that created
typed-array instance has appropriate length.

diffstat:

 src/njs_typed_array.c    |  11 ++++++++++-
 src/test/njs_unit_test.c |  12 ++++++++++++
 2 files changed, 22 insertions(+), 1 deletions(-)

diffs (55 lines):

diff -r 2bae88c33583 -r 95faab343e26 src/njs_typed_array.c
--- a/src/njs_typed_array.c	Thu Sep 03 13:30:15 2020 +0000
+++ b/src/njs_typed_array.c	Thu Sep 03 13:30:16 2020 +0000
@@ -801,12 +801,21 @@ njs_typed_array_species_create(njs_vm_t 
         return NJS_ERROR;
     }
 
-    if (!njs_is_typed_array(retval)) {
+    if (njs_slow_path(!njs_is_typed_array(retval))) {
         njs_type_error(vm, "Derived TypedArray constructor "
                        "returned not a typed array");
         return NJS_ERROR;
     }
 
+    if (njs_slow_path(nargs == 1 && njs_is_number(&args[0])
+                      && njs_typed_array_length(njs_typed_array(retval))
+                         < njs_number(&args[0])))
+    {
+        njs_type_error(vm, "Derived TypedArray constructor "
+                       "returned too short array");
+        return NJS_ERROR;
+    }
+
     return NJS_OK;
 }
 
diff -r 2bae88c33583 -r 95faab343e26 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Thu Sep 03 13:30:15 2020 +0000
+++ b/src/test/njs_unit_test.c	Thu Sep 03 13:30:16 2020 +0000
@@ -5821,6 +5821,13 @@ static njs_unit_test_t  njs_test[] =
       njs_str("true") },
 
     { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{var a = new v(2); "
+              "           a.constructor = {}; "
+              "           a.constructor[Symbol.species] = function() { return new v()};"
+              "           try {a.filter(v=>true)} catch(e) {return e.name == 'TypeError'}})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
               ".every(v=>{var a = new v([1,2,3]); "
               "           var r = a.slice(1,3);"
               "           return  a.buffer !== r.buffer;})"),
@@ -5859,6 +5866,11 @@ static njs_unit_test_t  njs_test[] =
       njs_str("true") },
 
     { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{var a = new v([1,2,3]); "
+              "           return  a.subarray(3).length === 0;})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
               ".every(v=>{var a = new v([1,2,3,4]); a.copyWithin(2); "
               "           return a.toString() === '1,2,1,2'})"),
       njs_str("true") },

From xeioex at nginx.com  Thu Sep  3 13:30:54 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 03 Sep 2020 13:30:54 +0000
Subject: [njs] Fixed TextDecoder() test on big-endian platforms.
Message-ID: <hg.f4fe4b4b9730.1599139854.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/f4fe4b4b9730
branches:  
changeset: 1517:f4fe4b4b9730
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Thu Sep 03 13:30:17 2020 +0000
description:
Fixed TextDecoder() test on big-endian platforms.

diffstat:

 src/test/njs_unit_test.c |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff -r 95faab343e26 -r f4fe4b4b9730 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Thu Sep 03 13:30:16 2020 +0000
+++ b/src/test/njs_unit_test.c	Thu Sep 03 13:30:17 2020 +0000
@@ -18205,8 +18205,8 @@ static njs_unit_test_t  njs_test[] =
 
     { njs_str("var de = new TextDecoder();"
               "var u8arr = new Uint8Array([240, 160, 174, 183]);"
-              "var u16arr = new Uint16Array([41200, 47022]);"
-              "var u32arr = new Uint32Array([3081674992]);"
+              "var u16arr = new Uint16Array(u8arr.buffer);"
+              "var u32arr = new Uint32Array(u8arr.buffer);"
               "[u8arr, u16arr, u32arr].map(v=>de.decode(v)).join(',')"),
       njs_str("?,?,?") },
 

From kasei at kasei.im  Mon Sep  7 12:06:06 2020
From: kasei at kasei.im (Kasei Wang)
Date: Mon, 7 Sep 2020 20:06:06 +0800
Subject: [PATCH] HTTP/2: premalloc http2 recv_buffer in init_conf
Message-ID: <CALjvqog3aBqCOruKPgPNvqD7DqRTymXQsfVRdJDj1BXwLp0szA@mail.gmail.com>

# HG changeset patch
# User Kasei Wang <kasei at kasei.im>
# Date 1599480224 -28800
#      Mon Sep 07 20:03:44 2020 +0800
# Node ID 1532a8fca7b24c3c3aa5cddd3362402d15f57a4f
# Parent  87d2ea860f380dc8418c97c0163412f53c2d008e
HTTP/2: premalloc http2 recv_buffer in init_conf

There is no need to check if h2mcf->recv_buffer malloced every time.

diff -r 87d2ea860f38 -r 1532a8fca7b2 src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c    Mon Sep 10 18:57:39 2018 +0300
+++ b/src/http/v2/ngx_http_v2.c    Mon Sep 07 20:03:44 2020 +0800
@@ -233,7 +233,6 @@
     ngx_pool_cleanup_t        *cln;
     ngx_http_connection_t     *hc;
     ngx_http_v2_srv_conf_t    *h2scf;
-    ngx_http_v2_main_conf_t   *h2mcf;
     ngx_http_v2_connection_t  *h2c;

     c = rev->data;
@@ -243,17 +242,6 @@

     c->log->action = "processing HTTP/2 connection";

-    h2mcf = ngx_http_get_module_main_conf(hc->conf_ctx, ngx_http_v2_module);
-
-    if (h2mcf->recv_buffer == NULL) {
-        h2mcf->recv_buffer = ngx_palloc(ngx_cycle->pool,
-                                        h2mcf->recv_buffer_size);
-        if (h2mcf->recv_buffer == NULL) {
-            ngx_http_close_connection(c);
-            return;
-        }
-    }
-
     h2c = ngx_pcalloc(c->pool, sizeof(ngx_http_v2_connection_t));
     if (h2c == NULL) {
         ngx_http_close_connection(c);
diff -r 87d2ea860f38 -r 1532a8fca7b2 src/http/v2/ngx_http_v2_module.c
--- a/src/http/v2/ngx_http_v2_module.c    Mon Sep 10 18:57:39 2018 +0300
+++ b/src/http/v2/ngx_http_v2_module.c    Mon Sep 07 20:03:44 2020 +0800
@@ -334,6 +334,10 @@
     ngx_http_v2_main_conf_t *h2mcf = conf;

     ngx_conf_init_size_value(h2mcf->recv_buffer_size, 256 * 1024);
+    h2mcf->recv_buffer = ngx_pcalloc(cf->pool, h2mcf->recv_buffer_size);
+    if (h2mcf->recv_buffer == NULL) {
+        return NGX_CONF_ERROR;
+    }

     return NGX_CONF_OK;
 }

From xeioex at nginx.com  Mon Sep  7 12:45:11 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 07 Sep 2020 12:45:11 +0000
Subject: [njs] Added %TypedArray% remaining methods.
Message-ID: <hg.0e927892f0bf.1599482711.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/0e927892f0bf
branches:  
changeset: 1518:0e927892f0bf
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Mon Sep 07 12:44:07 2020 +0000
description:
Added %TypedArray% remaining methods.

The following methods were added: of(), from().

diffstat:

 src/njs_typed_array.c    |  255 +++++++++++++++++++++++++++++++++++++---------
 src/njs_value.h          |    4 +
 src/test/njs_unit_test.c |   52 +++++++++
 3 files changed, 260 insertions(+), 51 deletions(-)

diffs (355 lines):

diff -r f4fe4b4b9730 -r 0e927892f0bf src/njs_typed_array.c
--- a/src/njs_typed_array.c	Thu Sep 03 13:30:17 2020 +0000
+++ b/src/njs_typed_array.c	Mon Sep 07 12:44:07 2020 +0000
@@ -268,6 +268,194 @@ njs_typed_array_constructor(njs_vm_t *vm
 
 
 static njs_int_t
+njs_typed_array_create(njs_vm_t *vm, njs_value_t *constructor,
+    njs_value_t *args, njs_uint_t nargs, njs_value_t *retval)
+{
+    njs_int_t     ret;
+    njs_value_t   this;
+    njs_object_t  *object;
+
+    object = njs_function_new_object(vm, constructor);
+    if (njs_slow_path(object == NULL)) {
+        return NJS_ERROR;
+    }
+
+    njs_set_object(&this, object);
+
+    ret = njs_function_call2(vm, njs_function(constructor), &this, args,
+                             nargs, retval, 1);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
+    if (njs_slow_path(!njs_is_typed_array(retval))) {
+        njs_type_error(vm, "Derived TypedArray constructor "
+                       "returned not a typed array");
+        return NJS_ERROR;
+    }
+
+    if (njs_slow_path(nargs == 1 && njs_is_number(&args[0])
+                      && njs_typed_array_length(njs_typed_array(retval))
+                         < njs_number(&args[0])))
+    {
+        njs_type_error(vm, "Derived TypedArray constructor "
+                       "returned too short array");
+        return NJS_ERROR;
+    }
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_typed_array_species_create(njs_vm_t *vm, njs_value_t *exemplar,
+    njs_value_t *args, njs_uint_t nargs, njs_value_t *retval)
+{
+    njs_int_t          ret;
+    njs_value_t        constructor;
+    njs_typed_array_t  *array;
+
+    array = njs_typed_array(exemplar);
+
+    njs_set_function(&constructor, &vm->constructors[array->type]);
+
+    ret = njs_value_species_constructor(vm, exemplar, &constructor,
+                                        &constructor);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    return njs_typed_array_create(vm, &constructor, args, nargs, retval);
+}
+
+
+static njs_int_t
+njs_typed_array_of(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    double             num;
+    uint32_t           length, i;
+    njs_int_t          ret;
+    njs_value_t        *this;
+    njs_value_t        argument;
+    njs_typed_array_t  *array;
+
+    this = njs_argument(args, 0);
+
+    if (njs_slow_path(!njs_is_constructor(this))) {
+        njs_type_error(vm, "%s is not a constructor",
+                       njs_type_string(this->type));
+        return NJS_ERROR;
+    }
+
+    length = nargs - 1;
+
+    njs_set_number(&argument, length);
+    ret = njs_typed_array_create(vm, this, &argument, 1, &vm->retval);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
+    array = njs_typed_array(&vm->retval);
+
+    for (i = 0; i < length; i++) {
+        ret = njs_value_to_number(vm, njs_argument(args, i + 1), &num);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_ERROR;
+        }
+
+        njs_typed_array_prop_set(vm, array, i, num);
+    }
+
+    njs_set_typed_array(&vm->retval, array);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_typed_array_from(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    double             num;
+    int64_t            length, i;
+    njs_int_t          ret;
+    njs_value_t        *this, *source, *mapfn;
+    njs_value_t        arguments[3], retval;
+    njs_function_t     *function;
+    njs_typed_array_t  *array;
+
+    this = njs_argument(args, 0);
+
+    if (njs_slow_path(!njs_is_constructor(this))) {
+        njs_type_error(vm, "%s is not a constructor",
+                       njs_type_string(this->type));
+        return NJS_ERROR;
+    }
+
+    mapfn = njs_arg(args, nargs, 2);
+
+    if (njs_slow_path(!njs_is_function_or_undefined(mapfn))) {
+        njs_type_error(vm, "\"mapfn\" argument is not callable");
+        return NJS_ERROR;
+    }
+
+    function = NULL;
+    if (njs_is_function(mapfn)) {
+        function = njs_function(mapfn);
+    }
+
+    source = njs_arg(args, nargs, 1);
+
+    ret = njs_value_to_object(vm, source);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    ret = njs_object_length(vm, source, &length);
+    if (njs_slow_path(ret == NJS_ERROR)) {
+        return ret;
+    }
+
+    njs_set_number(&arguments[0], length);
+    ret = njs_typed_array_create(vm, this, arguments, 1, &vm->retval);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
+    array = njs_typed_array(&vm->retval);
+    arguments[0] = *njs_arg(args, nargs, 3);
+
+    for (i = 0; i < length; i++) {
+        ret = njs_value_property_i64(vm, source, i, &retval);
+        if (njs_slow_path(ret == NJS_ERROR)) {
+            return NJS_ERROR;
+        }
+
+        if (function != NULL) {
+            arguments[1] = retval;
+            njs_set_number(&arguments[2], i);
+            ret = njs_function_apply(vm, function, arguments, 3, &retval);
+            if (njs_slow_path(ret != NJS_OK)) {
+                return NJS_ERROR;
+            }
+        }
+
+        ret = njs_value_to_number(vm, &retval, &num);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_ERROR;
+        }
+
+        njs_typed_array_prop_set(vm, array, i, num);
+    }
+
+    njs_set_typed_array(&vm->retval, array);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
 njs_typed_array_get_this(njs_vm_t *vm, njs_value_t *args,
     njs_uint_t nargs, njs_index_t unused)
 {
@@ -770,57 +958,6 @@ njs_typed_array_prototype_fill(njs_vm_t 
 
 
 static njs_int_t
-njs_typed_array_species_create(njs_vm_t *vm, njs_value_t *exemplar,
-    njs_value_t *args, njs_uint_t nargs, njs_value_t *retval)
-{
-    njs_int_t          ret;
-    njs_value_t        this, constructor;
-    njs_object_t       *object;
-    njs_typed_array_t  *array;
-
-    array = njs_typed_array(exemplar);
-
-    njs_set_function(&constructor, &vm->constructors[array->type]);
-
-    ret = njs_value_species_constructor(vm, exemplar, &constructor,
-                                        &constructor);
-    if (njs_slow_path(ret != NJS_OK)) {
-        return ret;
-    }
-
-    object = njs_function_new_object(vm, &constructor);
-    if (njs_slow_path(object == NULL)) {
-        return NJS_ERROR;
-    }
-
-    njs_set_object(&this, object);
-
-    ret = njs_function_call2(vm, njs_function(&constructor), &this,
-                             args, nargs, retval, 1);
-    if (njs_slow_path(ret != NJS_OK)) {
-        return NJS_ERROR;
-    }
-
-    if (njs_slow_path(!njs_is_typed_array(retval))) {
-        njs_type_error(vm, "Derived TypedArray constructor "
-                       "returned not a typed array");
-        return NJS_ERROR;
-    }
-
-    if (njs_slow_path(nargs == 1 && njs_is_number(&args[0])
-                      && njs_typed_array_length(njs_typed_array(retval))
-                         < njs_number(&args[0])))
-    {
-        njs_type_error(vm, "Derived TypedArray constructor "
-                       "returned too short array");
-        return NJS_ERROR;
-    }
-
-    return NJS_OK;
-}
-
-
-static njs_int_t
 njs_typed_array_prototype_slice(njs_vm_t *vm, njs_value_t *args,
     njs_uint_t nargs, njs_index_t copy)
 {
@@ -2013,6 +2150,22 @@ static const njs_object_prop_t  njs_type
         .configurable = 1,
         .enumerable = 0,
     },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("of"),
+        .value = njs_native_function(njs_typed_array_of, 0),
+        .writable = 1,
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_string("from"),
+        .value = njs_native_function(njs_typed_array_from, 1),
+        .writable = 1,
+        .configurable = 1,
+    },
 };
 
 
diff -r f4fe4b4b9730 -r 0e927892f0bf src/njs_value.h
--- a/src/njs_value.h	Thu Sep 03 13:30:17 2020 +0000
+++ b/src/njs_value.h	Mon Sep 07 12:44:07 2020 +0000
@@ -676,6 +676,10 @@ typedef struct {
     ((value)->type == NJS_FUNCTION || (value)->type == NJS_UNDEFINED)
 
 
+#define njs_is_constructor(value)                                             \
+    (njs_is_function(value) && njs_function(value)->ctor)
+
+
 #define njs_is_regexp(value)                                                  \
     ((value)->type == NJS_REGEXP)
 
diff -r f4fe4b4b9730 -r 0e927892f0bf src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Thu Sep 03 13:30:17 2020 +0000
+++ b/src/test/njs_unit_test.c	Mon Sep 07 12:44:07 2020 +0000
@@ -5473,6 +5473,58 @@ static njs_unit_test_t  njs_test[] =
       njs_str("true") },
 
     { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{var a = v.of(); return njs.dump(a) === `${v.name} []`})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{var a = v.of(1); return njs.dump(a) === `${v.name} [1]`})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{var a = v.of(1,2,3,4,5); return njs.dump(a) === `${v.name} [1,2,3,4,5]`})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{ try{ v.of(Symbol()); } catch (e) { return e.name === 'TypeError'}})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{ try{ v.of.call(()=>1); } catch (e) { return e.name === 'TypeError'}})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{ try{ v.of.call(function(){}); } catch (e) { return e.name === 'TypeError'}})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{var a = v.from([1,2]); return njs.dump(a) === `${v.name} [1,2]`})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{var a = v.from([1,2], v=>2*v); return njs.dump(a) === `${v.name} [2,4]`})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{var a = v.from([1,2], function(v){return v * this.m}, {m:3}); "
+              "           return njs.dump(a) === `${v.name} [3,6]`})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
+              ".every(v=>{var a = v.from([1,2], function(v){return v * this.m}, {m:3}); "
+              "           return njs.dump(a) === `${v.name} [3,6]`})"),
+      njs_str("true") },
+
+    { njs_str(NJS_INT_TYPED_ARRAY_LIST
+              ".every(v=>{var a = v.from({length:3, 0:1, 2:'a'});"
+              "           return njs.dump(a) === `${v.name} [1,0,0]`})"),
+      njs_str("true") },
+
+    { njs_str(NJS_FLOAT_TYPED_ARRAY_LIST
+              ".every(v=>{var a = v.from({length:3, 0:1, 2:'a'});"
+              "           return njs.dump(a) === `${v.name} [1,NaN,NaN]`})"),
+      njs_str("true") },
+
+    { njs_str(NJS_TYPED_ARRAY_LIST
               ".every(v=>{var a = new v(4); a.fill(42); return (a[0] === 42 && a.length == 4)})"),
       njs_str("true") },
 

From mdounin at mdounin.ru  Mon Sep  7 13:25:15 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 7 Sep 2020 16:25:15 +0300
Subject: [PATCH] HTTP/2: premalloc http2 recv_buffer in init_conf
In-Reply-To: <CALjvqog3aBqCOruKPgPNvqD7DqRTymXQsfVRdJDj1BXwLp0szA@mail.gmail.com>
References: <CALjvqog3aBqCOruKPgPNvqD7DqRTymXQsfVRdJDj1BXwLp0szA@mail.gmail.com>
Message-ID: <20200907132515.GD18881@mdounin.ru>

Hello!

On Mon, Sep 07, 2020 at 08:06:06PM +0800, Kasei Wang wrote:

> # HG changeset patch
> # User Kasei Wang <kasei at kasei.im>
> # Date 1599480224 -28800
> #      Mon Sep 07 20:03:44 2020 +0800
> # Node ID 1532a8fca7b24c3c3aa5cddd3362402d15f57a4f
> # Parent  87d2ea860f380dc8418c97c0163412f53c2d008e
> HTTP/2: premalloc http2 recv_buffer in init_conf
> 
> There is no need to check if h2mcf->recv_buffer malloced every time.
> 
> diff -r 87d2ea860f38 -r 1532a8fca7b2 src/http/v2/ngx_http_v2.c
> --- a/src/http/v2/ngx_http_v2.c    Mon Sep 10 18:57:39 2018 +0300
> +++ b/src/http/v2/ngx_http_v2.c    Mon Sep 07 20:03:44 2020 +0800
> @@ -233,7 +233,6 @@
>      ngx_pool_cleanup_t        *cln;
>      ngx_http_connection_t     *hc;
>      ngx_http_v2_srv_conf_t    *h2scf;
> -    ngx_http_v2_main_conf_t   *h2mcf;
>      ngx_http_v2_connection_t  *h2c;
> 
>      c = rev->data;
> @@ -243,17 +242,6 @@
> 
>      c->log->action = "processing HTTP/2 connection";
> 
> -    h2mcf = ngx_http_get_module_main_conf(hc->conf_ctx, ngx_http_v2_module);
> -
> -    if (h2mcf->recv_buffer == NULL) {
> -        h2mcf->recv_buffer = ngx_palloc(ngx_cycle->pool,
> -                                        h2mcf->recv_buffer_size);
> -        if (h2mcf->recv_buffer == NULL) {
> -            ngx_http_close_connection(c);
> -            return;
> -        }
> -    }
> -
>      h2c = ngx_pcalloc(c->pool, sizeof(ngx_http_v2_connection_t));
>      if (h2c == NULL) {
>          ngx_http_close_connection(c);
> diff -r 87d2ea860f38 -r 1532a8fca7b2 src/http/v2/ngx_http_v2_module.c
> --- a/src/http/v2/ngx_http_v2_module.c    Mon Sep 10 18:57:39 2018 +0300
> +++ b/src/http/v2/ngx_http_v2_module.c    Mon Sep 07 20:03:44 2020 +0800
> @@ -334,6 +334,10 @@
>      ngx_http_v2_main_conf_t *h2mcf = conf;
> 
>      ngx_conf_init_size_value(h2mcf->recv_buffer_size, 256 * 1024);
> +    h2mcf->recv_buffer = ngx_pcalloc(cf->pool, h2mcf->recv_buffer_size);
> +    if (h2mcf->recv_buffer == NULL) {
> +        return NGX_CONF_ERROR;
> +    }
> 
>      return NGX_CONF_OK;
>  }

Thank you for the patch.

While current approach implies some run-time costs, these costs 
are believed to be very small and not even measurable.  On the 
other hand, it saves relatively large buffer allocation if HTTP/2 
is compiled in but not used, and this might be important in some 
setups.

-- 
Maxim Dounin
http://mdounin.ru/

From alexander.borisov at nginx.com  Mon Sep  7 14:57:00 2020
From: alexander.borisov at nginx.com (Alexander Borisov)
Date: Mon, 07 Sep 2020 14:57:00 +0000
Subject: [njs] Separating string length counting and encoding conversion.
Message-ID: <hg.9a25433b4c76.1599490620.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/9a25433b4c76
branches:  
changeset: 1519:9a25433b4c76
user:      Alexander Borisov <alexander.borisov at nginx.com>
date:      Mon Sep 07 17:54:47 2020 +0300
description:
Separating string length counting and encoding conversion.

diffstat:

 src/njs_string.c         |  378 +++++++++++++++++++++++++++++++---------------
 src/njs_string.h         |   13 +
 src/test/njs_unit_test.c |    6 +
 3 files changed, 273 insertions(+), 124 deletions(-)

diffs (537 lines):

diff -r 0e927892f0bf -r 9a25433b4c76 src/njs_string.c
--- a/src/njs_string.c	Mon Sep 07 12:44:07 2020 +0000
+++ b/src/njs_string.c	Mon Sep 07 17:54:47 2020 +0300
@@ -12,10 +12,52 @@
 #define NJS_TRIM_END    2
 
 
+static u_char   njs_basis64[] = {
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 62, 77, 77, 77, 63,
+    52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 77, 77, 77, 77, 77, 77,
+    77,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14,
+    15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 77, 77, 77, 77, 77,
+    77, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
+    41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 77, 77, 77, 77, 77,
+
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77
+};
+
+
+static u_char   njs_basis64url[] = {
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 62, 77, 77,
+    52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 77, 77, 77, 77, 77, 77,
+    77,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14,
+    15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 77, 77, 77, 77, 63,
+    77, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
+    41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 77, 77, 77, 77, 77,
+
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
+    77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77
+};
+
+
 static void njs_encode_base64_core(njs_str_t *dst, const njs_str_t *src,
     const u_char *basis, njs_uint_t padding);
-static njs_int_t njs_decode_base64_core(njs_vm_t *vm,
-    njs_value_t *value, const njs_str_t *src, const u_char *basis);
+static njs_int_t njs_string_decode_base64_core(njs_vm_t *vm,
+    njs_value_t *value, const njs_str_t *src, njs_bool_t url);
 static njs_int_t njs_string_slice_prop(njs_vm_t *vm, njs_string_prop_t *string,
     njs_slice_prop_t *slice, njs_value_t *args, njs_uint_t nargs);
 static njs_int_t njs_string_slice_args(njs_vm_t *vm, njs_slice_prop_t *slice,
@@ -34,8 +76,8 @@ static njs_int_t njs_string_split_part_a
     njs_utf8_t utf8, const u_char *start, size_t size);
 
 
-#define njs_base64_encoded_length(len)  (((len + 2) / 3) * 4)
-#define njs_base64_decoded_length(len)  (((len + 3) / 4) * 3)
+#define njs_base64_encoded_length(len)       (((len + 2) / 3) * 4)
+#define njs_base64_decoded_length(len, pad)  (((len / 4) * 3) - pad)
 
 
 njs_int_t
@@ -206,9 +248,26 @@ njs_string_truncate(njs_value_t *value, 
 njs_int_t
 njs_string_hex(njs_vm_t *vm, njs_value_t *value, const njs_str_t *src)
 {
+    size_t     length;
+    njs_str_t  dst;
+
+    length = njs_encode_hex_length(src, &dst.length);
+
+    dst.start = njs_string_alloc(vm, value, dst.length, length);
+    if (njs_fast_path(dst.start != NULL)) {
+        njs_encode_hex(&dst, src);
+        return NJS_OK;
+    }
+
+    return NJS_ERROR;
+}
+
+
+void
+njs_encode_hex(njs_str_t *dst, const njs_str_t *src)
+{
     u_char        *p, c;
-    size_t        len;
-    njs_uint_t    i;
+    size_t        i, len;
     const u_char  *start;
 
     static const u_char  hex[16] = "0123456789abcdef";
@@ -216,23 +275,32 @@ njs_string_hex(njs_vm_t *vm, njs_value_t
     len = src->length;
     start = src->start;
 
-    p = njs_string_alloc(vm, value, len * 2, len * 2);
-
-    if (njs_fast_path(p != NULL)) {
-        for (i = 0; i < len; i++) {
-            c = start[i];
-            *p++ = hex[c >> 4];
-            *p++ = hex[c & 0x0f];
-        }
-
-        return NJS_OK;
+    p = dst->start;
+
+    for (i = 0; i < len; i++) {
+        c = start[i];
+        *p++ = hex[c >> 4];
+        *p++ = hex[c & 0x0f];
     }
-
-    return NJS_ERROR;
 }
 
 
-static void
+size_t
+njs_encode_hex_length(const njs_str_t *src, size_t *out_size)
+{
+    size_t  size;
+
+    size = src->length * 2;
+
+    if (out_size != NULL) {
+        *out_size = size;
+    }
+
+    return size;
+}
+
+
+void
 njs_encode_base64(njs_str_t *dst, const njs_str_t *src)
 {
     static u_char   basis64[] =
@@ -242,6 +310,21 @@ njs_encode_base64(njs_str_t *dst, const 
 }
 
 
+size_t
+njs_encode_base64_length(const njs_str_t *src, size_t *out_size)
+{
+    size_t  size;
+
+    size = (src->length == 0) ? 0 : njs_base64_encoded_length(src->length);
+
+    if (out_size != NULL) {
+        *out_size = size;
+    }
+
+    return size;
+}
+
+
 static void
 njs_encode_base64url(njs_str_t *dst, const njs_str_t *src)
 {
@@ -308,16 +391,17 @@ njs_encode_base64_core(njs_str_t *dst, c
 njs_int_t
 njs_string_base64(njs_vm_t *vm, njs_value_t *value, const njs_str_t *src)
 {
+    size_t     length;
     njs_str_t  dst;
 
-    if (njs_slow_path(src->length == 0)) {
+    length = njs_encode_base64_length(src, &dst.length);
+
+    if (njs_slow_path(dst.length == 0)) {
         vm->retval = njs_string_empty;
         return NJS_OK;
     }
 
-    dst.length = njs_base64_encoded_length(src->length);
-
-    dst.start = njs_string_alloc(vm, value, dst.length, dst.length);
+    dst.start = njs_string_alloc(vm, value, dst.length, length);
     if (njs_slow_path(dst.start == NULL)) {
         return NJS_ERROR;
     }
@@ -1666,30 +1750,31 @@ njs_string_bytes_from_string(njs_vm_t *v
 }
 
 
-njs_int_t
-njs_string_decode_hex(njs_vm_t *vm, njs_value_t *value, const njs_str_t *src)
+size_t
+njs_decode_hex_length(const njs_str_t *src, size_t *out_size)
 {
-    u_char        *p, *dst;
+    if (out_size != NULL) {
+        *out_size = src->length / 2;
+    }
+
+    return 0;
+}
+
+
+void
+njs_decode_hex(njs_str_t *dst, const njs_str_t *src)
+{
+    u_char        *p;
     size_t        len;
     njs_int_t     c;
     njs_uint_t    i, n;
     const u_char  *start;
 
-    len = src->length;
+    n = 0;
+    p = dst->start;
+
     start = src->start;
-
-    if (njs_slow_path(len == 0)) {
-        vm->retval = njs_string_empty;
-        return NJS_OK;
-    }
-
-    dst = njs_string_alloc(vm, value, len / 2, 0);
-    if (njs_slow_path(dst == NULL)) {
-        return NJS_ERROR;
-    }
-
-    n = 0;
-    p = dst;
+    len = src->length;
 
     for (i = 0; i < len; i++) {
         c = njs_char_to_hex(start[i]);
@@ -1705,81 +1790,47 @@ njs_string_decode_hex(njs_vm_t *vm, njs_
         }
     }
 
-    if (njs_slow_path((size_t) (p - dst) != (len / 2))) {
-        njs_string_truncate(value, p - dst, 0);
-    }
-
-    return NJS_OK;
-}
-
-
-njs_int_t
-njs_string_decode_base64(njs_vm_t *vm, njs_value_t *value, const njs_str_t *src)
-{
-    static u_char   basis64[] = {
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 62, 77, 77, 77, 63,
-        52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 77, 77, 77, 77, 77, 77,
-        77,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14,
-        15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 77, 77, 77, 77, 77,
-        77, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
-        41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 77, 77, 77, 77, 77,
-
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77
-    };
-
-    return njs_decode_base64_core(vm, value, src, basis64);
+    dst->length -= (dst->start + dst->length) - p;
 }
 
 
 njs_int_t
-njs_string_decode_base64url(njs_vm_t *vm, njs_value_t *value,
-    const njs_str_t *src)
+njs_string_decode_hex(njs_vm_t *vm, njs_value_t *value, const njs_str_t *src)
 {
-    static u_char   basis64[] = {
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 62, 77, 77,
-        52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 77, 77, 77, 77, 77, 77,
-        77,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14,
-        15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 77, 77, 77, 77, 63,
-        77, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
-        41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 77, 77, 77, 77, 77,
-
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
-        77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77
-    };
-
-    return njs_decode_base64_core(vm, value, src, basis64);
-}
-
-
-static njs_int_t
-njs_decode_base64_core(njs_vm_t *vm, njs_value_t *value, const njs_str_t *src,
-    const u_char *basis)
-{
-    size_t  len, dst_len;
-    u_char  *d, *s, *dst;
-
-    if (njs_slow_path(src->length == 0)) {
+    size_t     size, length;
+    njs_str_t  dst;
+
+    length = njs_decode_hex_length(src, &size);
+
+    if (njs_slow_path(size == 0)) {
         vm->retval = njs_string_empty;
         return NJS_OK;
     }
 
+    dst.start = njs_string_alloc(vm, value, size, length);
+    if (njs_slow_path(dst.start == NULL)) {
+        return NJS_ERROR;
+    }
+
+    dst.length = size;
+
+    njs_decode_hex(&dst, src);
+
+    if (njs_slow_path(dst.length != size)) {
+        njs_string_truncate(value, dst.length, 0);
+    }
+
+    return NJS_OK;
+}
+
+
+static size_t
+njs_decode_base64_length_core(const njs_str_t *src, const u_char *basis,
+    size_t *out_size)
+{
+    uint    pad;
+    size_t  len;
+
     for (len = 0; len < src->length; len++) {
         if (src->start[len] == '=') {
             break;
@@ -1790,46 +1841,125 @@ njs_decode_base64_core(njs_vm_t *vm, njs
         }
     }
 
-    if (len % 4 == 1) {
-        /* Rounding down to integer multiple of 4. */
-        len -= 1;
+    pad = 0;
+
+    if (len % 4 != 0) {
+        pad = 4 - (len % 4);
+        len += pad;
+    }
+
+    len = njs_base64_decoded_length(len, pad);
+
+    if (out_size != NULL) {
+        *out_size = len;
     }
 
-    dst_len = njs_base64_decoded_length(len);
-
-    dst = njs_string_alloc(vm, value, dst_len, 0);
-    if (njs_slow_path(dst == NULL)) {
-        return NJS_ERROR;
-    }
+    return 0;
+}
+
+
+size_t
+njs_decode_base64_length(const njs_str_t *src, size_t *out_size)
+{
+    return njs_decode_base64_length_core(src, njs_basis64, out_size);
+}
+
+
+size_t
+njs_decode_base64url_length(const njs_str_t *src, size_t *out_size)
+{
+    return njs_decode_base64_length_core(src, njs_basis64url, out_size);
+}
+
+
+static void
+njs_decode_base64_core(njs_str_t *dst, const njs_str_t *src,
+    const u_char *basis)
+{
+    size_t  len;
+    u_char  *d, *s;
 
     s = src->start;
-    d = dst;
-
-    while (len > 3) {
+    d = dst->start;
+
+    len = dst->length;
+
+    while (len >= 3) {
         *d++ = (u_char) (basis[s[0]] << 2 | basis[s[1]] >> 4);
         *d++ = (u_char) (basis[s[1]] << 4 | basis[s[2]] >> 2);
         *d++ = (u_char) (basis[s[2]] << 6 | basis[s[3]]);
 
         s += 4;
-        len -= 4;
+        len -= 3;
     }
 
-    if (len > 1) {
+    if (len >= 1) {
         *d++ = (u_char) (basis[s[0]] << 2 | basis[s[1]] >> 4);
     }
 
-    if (len > 2) {
+    if (len >= 2) {
         *d++ = (u_char) (basis[s[1]] << 4 | basis[s[2]] >> 2);
     }
-
-    if (njs_slow_path((size_t) (d - dst) != dst_len)) {
-        njs_string_truncate(value, d - dst, 0);
+}
+
+
+void
+njs_decode_base64(njs_str_t *dst, const njs_str_t *src)
+{
+    njs_decode_base64_core(dst, src, njs_basis64);
+}
+
+
+void
+njs_decode_base64url(njs_str_t *dst, const njs_str_t *src)
+{
+    njs_decode_base64_core(dst, src, njs_basis64url);
+}
+
+
+static njs_int_t
+njs_string_decode_base64_core(njs_vm_t *vm, njs_value_t *value,
+    const njs_str_t *src, njs_bool_t url)
+{
+    size_t     length;
+    const u_char *basis;
+    njs_str_t  dst;
+
+    basis = (url) ? njs_basis64url : njs_basis64;
+
+    length = njs_decode_base64_length_core(src, basis, &dst.length);
+
+    if (njs_slow_path(dst.length == 0)) {
+        vm->retval = njs_string_empty;
+        return NJS_OK;
     }
 
+    dst.start = njs_string_alloc(vm, value, dst.length, length);
+    if (njs_slow_path(dst.start == NULL)) {
+        return NJS_ERROR;
+    }
+
+    njs_decode_base64_core(&dst, src, basis);
+
     return NJS_OK;
 }
 
 
+njs_int_t
+njs_string_decode_base64(njs_vm_t *vm, njs_value_t *value, const njs_str_t *src)
+{
+    return njs_string_decode_base64_core(vm, value, src, 0);
+}
+
+
+njs_int_t
+njs_string_decode_base64url(njs_vm_t *vm, njs_value_t *value,
+    const njs_str_t *src)
+{
+    return njs_string_decode_base64_core(vm, value, src, 1);
+}
+
+
 static njs_int_t
 njs_string_from_char_code(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t is_point)
diff -r 0e927892f0bf -r 9a25433b4c76 src/njs_string.h
--- a/src/njs_string.h	Mon Sep 07 12:44:07 2020 +0000
+++ b/src/njs_string.h	Mon Sep 07 17:54:47 2020 +0300
@@ -185,6 +185,19 @@ u_char *njs_string_alloc(njs_vm_t *vm, n
     uint64_t length);
 njs_int_t njs_string_new(njs_vm_t *vm, njs_value_t *value, const u_char *start,
     uint32_t size, uint32_t length);
+
+void njs_encode_hex(njs_str_t *dst, const njs_str_t *src);
+size_t njs_encode_hex_length(const njs_str_t *src, size_t *out_size);
+void njs_encode_base64(njs_str_t *dst, const njs_str_t *src);
+size_t njs_encode_base64_length(const njs_str_t *src, size_t *out_size);
+
+void njs_decode_hex(njs_str_t *dst, const njs_str_t *src);
+size_t njs_decode_hex_length(const njs_str_t *src, size_t *out_size);
+void njs_decode_base64(njs_str_t *dst, const njs_str_t *src);
+size_t njs_decode_base64_length(const njs_str_t *src, size_t *out_size);
+void njs_decode_base64url(njs_str_t *dst, const njs_str_t *src);
+size_t njs_decode_base64url_length(const njs_str_t *src, size_t *out_size);
+
 njs_int_t njs_string_hex(njs_vm_t *vm, njs_value_t *value,
     const njs_str_t *src);
 njs_int_t njs_string_base64(njs_vm_t *vm, njs_value_t *value,
diff -r 0e927892f0bf -r 9a25433b4c76 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Mon Sep 07 12:44:07 2020 +0000
+++ b/src/test/njs_unit_test.c	Mon Sep 07 17:54:47 2020 +0300
@@ -8824,9 +8824,15 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("String.bytesFrom('QQ==', 'base64')"),
       njs_str("A") },
 
+    { njs_str("String.bytesFrom('QQ=', 'base64')"),
+      njs_str("A") },
+
     { njs_str("String.bytesFrom('QQ', 'base64')"),
       njs_str("A") },
 
+    { njs_str("String.bytesFrom('Q', 'base64')"),
+      njs_str("") },
+
     { njs_str("String.bytesFrom('QUI=', 'base64')"),
       njs_str("AB") },
 

From alexander.borisov at nginx.com  Mon Sep  7 14:57:02 2020
From: alexander.borisov at nginx.com (Alexander Borisov)
Date: Mon, 07 Sep 2020 14:57:02 +0000
Subject: [njs] Added support for ArrayBuffer in TextDecoder.prototype.decode().
Message-ID: <hg.4ef85ee61700.1599490622.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/4ef85ee61700
branches:  
changeset: 1520:4ef85ee61700
user:      Alexander Borisov <alexander.borisov at nginx.com>
date:      Mon Sep 07 17:55:10 2020 +0300
description:
Added support for ArrayBuffer in TextDecoder.prototype.decode().

This closes #331 issue on Github.

diffstat:

 src/njs_encoding.c       |  43 ++++++++++++++++++++++++++-----------------
 src/test/njs_unit_test.c |   6 ++++++
 2 files changed, 32 insertions(+), 17 deletions(-)

diffs (80 lines):

diff -r 9a25433b4c76 -r 4ef85ee61700 src/njs_encoding.c
--- a/src/njs_encoding.c	Mon Sep 07 17:54:47 2020 +0300
+++ b/src/njs_encoding.c	Mon Sep 07 17:55:10 2020 +0300
@@ -532,16 +532,17 @@ static njs_int_t
 njs_text_decoder_decode(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t unused)
 {
-    u_char                   *dst;
-    size_t                   size;
-    ssize_t                  length;
-    njs_int_t                ret;
-    njs_bool_t               stream;
-    njs_value_t              retval, *this, *typed_array, *options;
-    const u_char             *start, *end;
-    njs_unicode_decode_t     ctx;
-    njs_encoding_decode_t    *data;
-    const njs_typed_array_t  *array;
+    u_char                    *dst;
+    size_t                    size;
+    ssize_t                   length;
+    njs_int_t                 ret;
+    njs_bool_t                stream;
+    njs_value_t               retval, *this, *value, *options;
+    const u_char              *start, *end;
+    njs_unicode_decode_t      ctx;
+    njs_encoding_decode_t     *data;
+    const njs_typed_array_t   *array;
+    const njs_array_buffer_t  *buffer;
 
     static const njs_value_t  stream_str = njs_string("stream");
 
@@ -558,17 +559,25 @@ njs_text_decoder_decode(njs_vm_t *vm, nj
     }
 
     if (njs_fast_path(nargs > 1)) {
-        typed_array = njs_argument(args, 1);
-        if (njs_slow_path(!njs_is_typed_array(typed_array))) {
+        value = njs_argument(args, 1);
+
+        if (njs_is_typed_array(value)) {
+            array = njs_typed_array(value);
+
+            start = array->buffer->u.u8;
+            end = start + array->byte_length;
+
+        } else if (njs_is_array_buffer(value)) {
+            buffer = njs_array_buffer(value);
+
+            start = buffer->u.u8;
+            end = start + buffer->size;
+
+        } else {
             njs_type_error(vm, "The \"input\" argument must be an instance "
                            "of TypedArray");
             return NJS_ERROR;
         }
-
-        array = njs_typed_array(typed_array);
-
-        start = array->buffer->u.u8;
-        end = start + array->byte_length;
     }
 
     if (nargs > 2) {
diff -r 9a25433b4c76 -r 4ef85ee61700 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Mon Sep 07 17:54:47 2020 +0300
+++ b/src/test/njs_unit_test.c	Mon Sep 07 17:55:10 2020 +0300
@@ -18327,6 +18327,12 @@ static njs_unit_test_t  njs_test[] =
 
     { njs_str("TextDecoder.prototype.decode.apply({}, new Uint8Array([1]))"),
       njs_str("TypeError: \"this\" is not a TextDecoder") },
+
+    { njs_str("var de = new TextDecoder();"
+              "var buf = new Uint32Array([1,2,3]).buffer;"
+              "var en = new TextEncoder();"
+              "njs.dump(en.encode(de.decode(buf)))"),
+      njs_str("Uint8Array [1,0,0,0,2,0,0,0,3,0,0,0]") },
 };
 
 

From alexander.borisov at nginx.com  Mon Sep  7 14:57:04 2020
From: alexander.borisov at nginx.com (Alexander Borisov)
Date: Mon, 07 Sep 2020 14:57:04 +0000
Subject: [njs] Fixed TextDecoder.prototype.decode() with non-zero TypedArray
 offset.
Message-ID: <hg.f8c8e23d2bbd.1599490624.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/f8c8e23d2bbd
branches:  
changeset: 1521:f8c8e23d2bbd
user:      Alexander Borisov <alexander.borisov at nginx.com>
date:      Mon Sep 07 17:55:24 2020 +0300
description:
Fixed TextDecoder.prototype.decode() with non-zero TypedArray offset.

diffstat:

 src/njs_encoding.c       |   4 ++--
 src/njs_typed_array.c    |   3 +--
 src/njs_typed_array.h    |  14 ++++++++++++++
 src/test/njs_unit_test.c |  11 +++++++++++
 4 files changed, 28 insertions(+), 4 deletions(-)

diffs (86 lines):

diff -r 4ef85ee61700 -r f8c8e23d2bbd src/njs_encoding.c
--- a/src/njs_encoding.c	Mon Sep 07 17:55:10 2020 +0300
+++ b/src/njs_encoding.c	Mon Sep 07 17:55:24 2020 +0300
@@ -214,7 +214,7 @@ njs_text_encoder_encode_into(njs_vm_t *v
     end = start + str.length;
 
     array = njs_typed_array(dest);
-    to = njs_typed_array_buffer(array)->u.u8;
+    to = njs_typed_array_start(array);
     to_end = to + array->byte_length;
 
     cp = 0;
@@ -564,7 +564,7 @@ njs_text_decoder_decode(njs_vm_t *vm, nj
         if (njs_is_typed_array(value)) {
             array = njs_typed_array(value);
 
-            start = array->buffer->u.u8;
+            start = njs_typed_array_start(array);
             end = start + array->byte_length;
 
         } else if (njs_is_array_buffer(value)) {
diff -r 4ef85ee61700 -r f8c8e23d2bbd src/njs_typed_array.c
--- a/src/njs_typed_array.c	Mon Sep 07 17:55:10 2020 +0300
+++ b/src/njs_typed_array.c	Mon Sep 07 17:55:24 2020 +0300
@@ -610,8 +610,7 @@ njs_typed_array_prototype_byte_offset(nj
 
     array = njs_typed_array(this);
 
-    njs_set_number(&vm->retval, array->offset
-                                * njs_typed_array_element_size(array->type));
+    njs_set_number(&vm->retval, njs_typed_array_offset(array));
 
     return NJS_OK;
 }
diff -r 4ef85ee61700 -r f8c8e23d2bbd src/njs_typed_array.h
--- a/src/njs_typed_array.h	Mon Sep 07 17:55:10 2020 +0300
+++ b/src/njs_typed_array.h	Mon Sep 07 17:55:24 2020 +0300
@@ -55,6 +55,20 @@ njs_typed_array_length(const njs_typed_a
 }
 
 
+njs_inline uint32_t
+njs_typed_array_offset(const njs_typed_array_t *array)
+{
+    return array->offset * njs_typed_array_element_size(array->type);
+}
+
+
+njs_inline u_char *
+njs_typed_array_start(const njs_typed_array_t *array)
+{
+    return &array->buffer->u.u8[njs_typed_array_offset(array)];
+}
+
+
 njs_inline double
 njs_typed_array_prop(const njs_typed_array_t *array, uint32_t index)
 {
diff -r 4ef85ee61700 -r f8c8e23d2bbd src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Mon Sep 07 17:55:10 2020 +0300
+++ b/src/test/njs_unit_test.c	Mon Sep 07 17:55:24 2020 +0300
@@ -18226,6 +18226,11 @@ static njs_unit_test_t  njs_test[] =
               "var res = en.encodeInto('?????', utf8); njs.dump(res)"),
       njs_str("{read:5,written:10}") },
 
+    { njs_str("var en = new TextEncoder();"
+              "var utf8 = new Uint8Array(10);"
+              "en.encodeInto('?????', utf8.subarray(2)); utf8[0]"),
+      njs_str("0") },
+
     { njs_str("var str = String.bytesFrom([0xCE]);"
               "var en = new TextEncoder();"
               "var utf8 = new Uint8Array(3);"
@@ -18333,6 +18338,12 @@ static njs_unit_test_t  njs_test[] =
               "var en = new TextEncoder();"
               "njs.dump(en.encode(de.decode(buf)))"),
       njs_str("Uint8Array [1,0,0,0,2,0,0,0,3,0,0,0]") },
+
+    { njs_str("var de = new TextDecoder();"
+              "var buf = new Uint32Array([1,2,3]).subarray(1,2);"
+              "var en = new TextEncoder();"
+              "njs.dump(en.encode(de.decode(buf)))"),
+      njs_str("Uint8Array [2,0,0,0]") },
 };
 
 

From qiao.liu at intel.com  Thu Sep 10 05:22:12 2020
From: qiao.liu at intel.com (Liu, Qiao)
Date: Thu, 10 Sep 2020 05:22:12 +0000
Subject: [PATCH] Use BPF to distribute packet to different work thread.
Message-ID: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>

# HG changeset patch
# User Liu Qiao <qiao.liu at intel.com>
# Date 1599735293 14400
#      Thu Sep 10 06:54:53 2020 -0400
# Node ID f79d524a2cc0093c53490f947564e42371cf944f
# Parent  da5e3f5b16733167142b599b6af3ce9469a07d52
Use BPF to distribute packet to different work thread.
Use Berkeley Packet Filter to get packet queue_mapping number,
and use this queue_mapping number to distribute the packet to
different work thread, this will improve CPU utilization and http
latency.
Author: Samudrala, Sridhar <sridhar.samudrala at intel.com>

diff -r da5e3f5b1673 -r f79d524a2cc0 auto/os/linux
--- a/auto/os/linux           Wed Sep 02 23:13:36 2020 +0300
+++ b/auto/os/linux        Thu Sep 10 06:54:53 2020 -0400
@@ -32,6 +32,10 @@
     have=NGX_HAVE_POSIX_FADVISE . auto/nohave
fi
+if [ $version -lt 263680 ]; then
+    have=NGX_HAVE_REUSEPORT_CBPF . auto/nohave
+fi
+
# epoll, EPOLLET version
 ngx_feature="epoll"
diff -r da5e3f5b1673 -r f79d524a2cc0 auto/unix
--- a/auto/unix   Wed Sep 02 23:13:36 2020 +0300
+++ b/auto/unix               Thu Sep 10 06:54:53 2020 -0400
@@ -331,6 +331,17 @@
ngx_feature_test="setsockopt(0, SOL_SOCKET, SO_REUSEPORT, NULL, 0)"
. auto/feature
+ngx_feature="SO_REUSEPORT_CBPF"
+ngx_feature_name="NGX_HAVE_REUSEPORT_CBPF"
+ngx_feature_run=no
+ngx_feature_incs="#include <sys/socket.h>
+                  #include <linux/filter.h>
+                  #include <error.h>"
+ngx_feature_path=
+ngx_feature_libs=
+ngx_feature_test="setsockopt(0, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, NULL, 0)"
+. auto/feature
+
 ngx_feature="SO_ACCEPTFILTER"
ngx_feature_name="NGX_HAVE_DEFERRED_ACCEPT"
diff -r da5e3f5b1673 -r f79d524a2cc0 src/core/ngx_connection.c
--- a/src/core/ngx_connection.c Wed Sep 02 23:13:36 2020 +0300
+++ b/src/core/ngx_connection.c              Thu Sep 10 06:54:53 2020 -0400
@@ -8,7 +8,10 @@
#include <ngx_config.h>
#include <ngx_core.h>
#include <ngx_event.h>
-
+#if (NGX_HAVE_REUSEPORT_CBPF)
+#include <linux/filter.h>
+#include <error.h>
+#endif
 ngx_os_io_t  ngx_io;
@@ -708,6 +711,35 @@
     return NGX_OK;
}
+#if(NGX_HAVE_REUSEPORT)
+#if(NGX_HAVE_REUSEPORT_CBPF)
+#ifndef ARRAY_SIZE
+#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
+#endif
+
+static void attach_bpf(int fd, uint16_t n)
+{
+             struct sock_filter code[] = {
+                            /* A = skb->queue_mapping */
+                            { BPF_LD  | BPF_W | BPF_ABS, 0, 0, SKF_AD_OFF + SKF_AD_QUEUE },
+                            /* A = A % n */
+                            { BPF_ALU | BPF_MOD, 0, 0, n },
+                            /* return A */
+                            { BPF_RET | BPF_A, 0, 0, 0 },
+             };
+             struct sock_fprog p = {
+                            .len = ARRAY_SIZE(code),
+                            .filter = code,
+             };
+
+             if (setsockopt(fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, &p, sizeof(p)))
+                            printf("failed to set SO_ATTACH_REUSEPORT_CBPF");
+             else
+                            printf("bpf prog attached to fd:%d\n", fd);
+}
+#endif
+#endif
+
 void
ngx_configure_listening_sockets(ngx_cycle_t *cycle)
@@ -719,6 +751,11 @@
#if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER)
     struct accept_filter_arg   af;
#endif
+#if (NGX_HAVE_REUSEPORT)
+#if (NGX_HAVE_REUSEPORT_CBPF)
+        ngx_core_conf_t* ccf ;
+#endif
+#endif
     ls = cycle->listening.elts;
     for (i = 0; i < cycle->listening.nelts; i++) {
@@ -1011,6 +1048,16 @@
         }
 #endif
+#if (NGX_HAVE_REUSEPORT)
+#if (NGX_HAVE_REUSEPORT_CBPF)
+    if(ls[i].reuseport)
+    {
+        ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,ngx_core_module);
+        if(ccf)
+            attach_bpf(ls[i].fd, ccf->worker_processes);
+    }
+#endif
+#endif
     }
     return;

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20200910/9d7e3d3e/attachment.htm>

From vl at nginx.com  Thu Sep 10 10:11:32 2020
From: vl at nginx.com (Vladimir Homutov)
Date: Thu, 10 Sep 2020 13:11:32 +0300
Subject: [PATCH] Use BPF to distribute packet to different work thread.
In-Reply-To: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
References: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
Message-ID: <20200910101132.GA38727@vl.krasnogorsk.ru>

On Thu, Sep 10, 2020 at 05:22:12AM +0000, Liu, Qiao wrote:
> # HG changeset patch
> # User Liu Qiao <qiao.liu at intel.com>
> # Date 1599735293 14400
> #      Thu Sep 10 06:54:53 2020 -0400
> # Node ID f79d524a2cc0093c53490f947564e42371cf944f
> # Parent  da5e3f5b16733167142b599b6af3ce9469a07d52
> Use BPF to distribute packet to different work thread.
> Use Berkeley Packet Filter to get packet queue_mapping number,
> and use this queue_mapping number to distribute the packet to
> different work thread, this will improve CPU utilization and http
> latency.
> Author: Samudrala, Sridhar <sridhar.samudrala at intel.com>

Do you have any real measurements with proper testing methodology?

Also, why not bind NIC queues to cores system-wide?

>
> diff -r da5e3f5b1673 -r f79d524a2cc0 auto/os/linux
> --- a/auto/os/linux           Wed Sep 02 23:13:36 2020 +0300
> +++ b/auto/os/linux        Thu Sep 10 06:54:53 2020 -0400
> @@ -32,6 +32,10 @@
>      have=NGX_HAVE_POSIX_FADVISE . auto/nohave
> fi
> +if [ $version -lt 263680 ]; then
> +    have=NGX_HAVE_REUSEPORT_CBPF . auto/nohave
> +fi
> +
> # epoll, EPOLLET version
>  ngx_feature="epoll"
> diff -r da5e3f5b1673 -r f79d524a2cc0 auto/unix
> --- a/auto/unix   Wed Sep 02 23:13:36 2020 +0300
> +++ b/auto/unix               Thu Sep 10 06:54:53 2020 -0400
> @@ -331,6 +331,17 @@
> ngx_feature_test="setsockopt(0, SOL_SOCKET, SO_REUSEPORT, NULL, 0)"
> . auto/feature
> +ngx_feature="SO_REUSEPORT_CBPF"
> +ngx_feature_name="NGX_HAVE_REUSEPORT_CBPF"
> +ngx_feature_run=no
> +ngx_feature_incs="#include <sys/socket.h>
> +                  #include <linux/filter.h>
> +                  #include <error.h>"
> +ngx_feature_path=
> +ngx_feature_libs=
> +ngx_feature_test="setsockopt(0, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, NULL, 0)"
> +. auto/feature
> +
>  ngx_feature="SO_ACCEPTFILTER"
> ngx_feature_name="NGX_HAVE_DEFERRED_ACCEPT"
> diff -r da5e3f5b1673 -r f79d524a2cc0 src/core/ngx_connection.c
> --- a/src/core/ngx_connection.c Wed Sep 02 23:13:36 2020 +0300
> +++ b/src/core/ngx_connection.c              Thu Sep 10 06:54:53 2020 -0400
> @@ -8,7 +8,10 @@
> #include <ngx_config.h>
> #include <ngx_core.h>
> #include <ngx_event.h>
> -
> +#if (NGX_HAVE_REUSEPORT_CBPF)
> +#include <linux/filter.h>
> +#include <error.h>
> +#endif
>  ngx_os_io_t  ngx_io;
> @@ -708,6 +711,35 @@
>      return NGX_OK;
> }
> +#if(NGX_HAVE_REUSEPORT)
> +#if(NGX_HAVE_REUSEPORT_CBPF)
> +#ifndef ARRAY_SIZE
> +#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
> +#endif
> +
> +static void attach_bpf(int fd, uint16_t n)
> +{
> +             struct sock_filter code[] = {
> +                            /* A = skb->queue_mapping */
> +                            { BPF_LD  | BPF_W | BPF_ABS, 0, 0, SKF_AD_OFF + SKF_AD_QUEUE },
> +                            /* A = A % n */
> +                            { BPF_ALU | BPF_MOD, 0, 0, n },
> +                            /* return A */
> +                            { BPF_RET | BPF_A, 0, 0, 0 },
> +             };
> +             struct sock_fprog p = {
> +                            .len = ARRAY_SIZE(code),
> +                            .filter = code,
> +             };
> +
> +             if (setsockopt(fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, &p, sizeof(p)))
> +                            printf("failed to set SO_ATTACH_REUSEPORT_CBPF");
> +             else
> +                            printf("bpf prog attached to fd:%d\n", fd);
> +}
> +#endif
> +#endif
> +
>  void
> ngx_configure_listening_sockets(ngx_cycle_t *cycle)
> @@ -719,6 +751,11 @@
> #if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER)
>      struct accept_filter_arg   af;
> #endif
> +#if (NGX_HAVE_REUSEPORT)
> +#if (NGX_HAVE_REUSEPORT_CBPF)
> +        ngx_core_conf_t* ccf ;
> +#endif
> +#endif
>      ls = cycle->listening.elts;
>      for (i = 0; i < cycle->listening.nelts; i++) {
> @@ -1011,6 +1048,16 @@
>          }
>  #endif
> +#if (NGX_HAVE_REUSEPORT)
> +#if (NGX_HAVE_REUSEPORT_CBPF)
> +    if(ls[i].reuseport)
> +    {
> +        ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,ngx_core_module);
> +        if(ccf)
> +            attach_bpf(ls[i].fd, ccf->worker_processes);
> +    }
> +#endif
> +#endif
>      }
>      return;
>

> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel


From qiao.liu at intel.com  Fri Sep 11 05:41:47 2020
From: qiao.liu at intel.com (Liu, Qiao)
Date: Fri, 11 Sep 2020 05:41:47 +0000
Subject: [PATCH] Use BPF to distribute packet to different work thread.
In-Reply-To: <20200910101132.GA38727@vl.krasnogorsk.ru>
References: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200910101132.GA38727@vl.krasnogorsk.ru>
Message-ID: <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>

Hi, Vladimir Homutov:
The below is our WRK test result output with BPF enable 

  112 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   608.23ms  820.71ms  10.00s    87.48%
    Connect    16.52ms   54.53ms   1.99s    94.73%
    Delay     153.13ms  182.17ms   2.00s    90.74%
    Req/Sec   244.79    142.32     1.99k    68.40%
  Latency Distribution
  50.00%  293.50ms
  75.00%  778.33ms
  90.00%    1.61s
  99.00%    3.71s
  99.90%    7.03s
  99.99%    8.94s
  Connect Distribution
  50.00%    1.93ms
  75.00%    2.85ms
  90.00%   55.76ms
  99.00%  229.19ms
  99.90%  656.79ms
  99.99%    1.43s
  Delay Distribution
  50.00%  110.96ms
  75.00%  193.67ms
  90.00%  321.77ms
  99.00%  959.27ms
  99.90%    1.57s
  99.99%    1.91s
Compared with no BPF but enable reuseport as below

112 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   680.50ms  943.69ms  10.00s    87.18%
    Connect    58.44ms  238.08ms   2.00s    94.58%
    Delay     158.84ms  256.28ms   2.00s    90.92%
    Req/Sec   244.51    151.00     1.41k    69.67%
  Latency Distribution
  50.00%  317.61ms
  75.00%  913.52ms
  90.00%    1.90s
  99.00%    4.30s
  99.90%    6.52s
  99.99%    8.80s
  Connect Distribution
  50.00%    1.88ms
  75.00%    2.21ms
  90.00%   55.94ms
  99.00%    1.45s
  99.90%    1.95s
  99.99%    2.00s
  Delay Distribution
  50.00%   73.01ms
  75.00%  190.40ms
  90.00%  387.01ms
  99.00%    1.34s
  99.90%    1.86s
  99.99%    1.99s


>From the above results, there shows almost 20% percent latency reduction. P99 latency of BPF is 3.71s , but without BPF is 4.3s.

For question2 why not bind queues to cores, as what we think ,bind to thread is different from bind to core. Bind to threads is user space work to distribute the load, bind to core is NIC hardware level. 
We do have ADQ for NIC, that will do a lot bind to cores things
Thanks
LQ
-----Original Message-----
From: nginx-devel <nginx-devel-bounces at nginx.org> On Behalf Of Vladimir Homutov
Sent: Thursday, September 10, 2020 6:12 PM
To: nginx-devel at nginx.org
Subject: Re: [PATCH] Use BPF to distribute packet to different work thread.

On Thu, Sep 10, 2020 at 05:22:12AM +0000, Liu, Qiao wrote:
> # HG changeset patch
> # User Liu Qiao <qiao.liu at intel.com>
> # Date 1599735293 14400
> #      Thu Sep 10 06:54:53 2020 -0400
> # Node ID f79d524a2cc0093c53490f947564e42371cf944f
> # Parent  da5e3f5b16733167142b599b6af3ce9469a07d52
> Use BPF to distribute packet to different work thread.
> Use Berkeley Packet Filter to get packet queue_mapping number, and use 
> this queue_mapping number to distribute the packet to different work 
> thread, this will improve CPU utilization and http latency.
> Author: Samudrala, Sridhar <sridhar.samudrala at intel.com>

Do you have any real measurements with proper testing methodology?

Also, why not bind NIC queues to cores system-wide?

>
> diff -r da5e3f5b1673 -r f79d524a2cc0 auto/os/linux
> --- a/auto/os/linux           Wed Sep 02 23:13:36 2020 +0300
> +++ b/auto/os/linux        Thu Sep 10 06:54:53 2020 -0400
> @@ -32,6 +32,10 @@
>      have=NGX_HAVE_POSIX_FADVISE . auto/nohave fi
> +if [ $version -lt 263680 ]; then
> +    have=NGX_HAVE_REUSEPORT_CBPF . auto/nohave fi
> +
> # epoll, EPOLLET version
>  ngx_feature="epoll"
> diff -r da5e3f5b1673 -r f79d524a2cc0 auto/unix
> --- a/auto/unix   Wed Sep 02 23:13:36 2020 +0300
> +++ b/auto/unix               Thu Sep 10 06:54:53 2020 -0400
> @@ -331,6 +331,17 @@
> ngx_feature_test="setsockopt(0, SOL_SOCKET, SO_REUSEPORT, NULL, 0)"
> . auto/feature
> +ngx_feature="SO_REUSEPORT_CBPF"
> +ngx_feature_name="NGX_HAVE_REUSEPORT_CBPF"
> +ngx_feature_run=no
> +ngx_feature_incs="#include <sys/socket.h>
> +                  #include <linux/filter.h>
> +                  #include <error.h>"
> +ngx_feature_path=
> +ngx_feature_libs=
> +ngx_feature_test="setsockopt(0, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, NULL, 0)"
> +. auto/feature
> +
>  ngx_feature="SO_ACCEPTFILTER"
> ngx_feature_name="NGX_HAVE_DEFERRED_ACCEPT"
> diff -r da5e3f5b1673 -r f79d524a2cc0 src/core/ngx_connection.c
> --- a/src/core/ngx_connection.c Wed Sep 02 23:13:36 2020 +0300
> +++ b/src/core/ngx_connection.c              Thu Sep 10 06:54:53 2020 -0400
> @@ -8,7 +8,10 @@
> #include <ngx_config.h>
> #include <ngx_core.h>
> #include <ngx_event.h>
> -
> +#if (NGX_HAVE_REUSEPORT_CBPF)
> +#include <linux/filter.h>
> +#include <error.h>
> +#endif
>  ngx_os_io_t  ngx_io;
> @@ -708,6 +711,35 @@
>      return NGX_OK;
> }
> +#if(NGX_HAVE_REUSEPORT)
> +#if(NGX_HAVE_REUSEPORT_CBPF)
> +#ifndef ARRAY_SIZE
> +#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0])) #endif
> +
> +static void attach_bpf(int fd, uint16_t n) {
> +             struct sock_filter code[] = {
> +                            /* A = skb->queue_mapping */
> +                            { BPF_LD  | BPF_W | BPF_ABS, 0, 0, SKF_AD_OFF + SKF_AD_QUEUE },
> +                            /* A = A % n */
> +                            { BPF_ALU | BPF_MOD, 0, 0, n },
> +                            /* return A */
> +                            { BPF_RET | BPF_A, 0, 0, 0 },
> +             };
> +             struct sock_fprog p = {
> +                            .len = ARRAY_SIZE(code),
> +                            .filter = code,
> +             };
> +
> +             if (setsockopt(fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, &p, sizeof(p)))
> +                            printf("failed to set SO_ATTACH_REUSEPORT_CBPF");
> +             else
> +                            printf("bpf prog attached to fd:%d\n", 
> +fd); } #endif #endif
> +
>  void
> ngx_configure_listening_sockets(ngx_cycle_t *cycle) @@ -719,6 +751,11 
> @@ #if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER)
>      struct accept_filter_arg   af;
> #endif
> +#if (NGX_HAVE_REUSEPORT)
> +#if (NGX_HAVE_REUSEPORT_CBPF)
> +        ngx_core_conf_t* ccf ;
> +#endif
> +#endif
>      ls = cycle->listening.elts;
>      for (i = 0; i < cycle->listening.nelts; i++) { @@ -1011,6 
> +1048,16 @@
>          }
>  #endif
> +#if (NGX_HAVE_REUSEPORT)
> +#if (NGX_HAVE_REUSEPORT_CBPF)
> +    if(ls[i].reuseport)
> +    {
> +        ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,ngx_core_module);
> +        if(ccf)
> +            attach_bpf(ls[i].fd, ccf->worker_processes);
> +    }
> +#endif
> +#endif
>      }
>      return;
>

> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

From xeioex at nginx.com  Fri Sep 11 13:57:44 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Fri, 11 Sep 2020 13:57:44 +0000
Subject: [njs] HTTP: added support for http/2 and http/3 in r.httpVersion.
Message-ID: <hg.e1dccc5a2f05.1599832664.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/e1dccc5a2f05
branches:  
changeset: 1522:e1dccc5a2f05
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Fri Sep 11 13:54:57 2020 +0000
description:
HTTP: added support for http/2 and http/3 in r.httpVersion.

diffstat:

 nginx/ngx_http_js_module.c |  16 +++++++++++++++-
 1 files changed, 15 insertions(+), 1 deletions(-)

diffs (28 lines):

diff -r f8c8e23d2bbd -r e1dccc5a2f05 nginx/ngx_http_js_module.c
--- a/nginx/ngx_http_js_module.c	Mon Sep 07 17:55:24 2020 +0300
+++ b/nginx/ngx_http_js_module.c	Fri Sep 11 13:54:57 2020 +0000
@@ -1912,9 +1912,23 @@ ngx_http_js_ext_get_http_version(njs_vm_
         ngx_str_set(&v, "1.0");
         break;
 
-    default: /* NGX_HTTP_VERSION_11 */
+    case NGX_HTTP_VERSION_11:
         ngx_str_set(&v, "1.1");
         break;
+
+    case NGX_HTTP_VERSION_20:
+        ngx_str_set(&v, "2.0");
+        break;
+
+#if (NGX_HTTP_VERSION_30)
+    case NGX_HTTP_VERSION_30:
+        ngx_str_set(&v, "3.0");
+        break;
+#endif
+
+    default:
+        ngx_str_set(&v, "");
+        break;
     }
 
     return njs_vm_value_string_set(vm, retval, v.data, v.len);

From pluknet at nginx.com  Fri Sep 11 19:16:43 2020
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 11 Sep 2020 19:16:43 +0000
Subject: [nginx] Cache: reset c->body_start when reading a variant on Vary
 mismatch.
Message-ID: <hg.847fd35f94de.1599851803.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/847fd35f94de
branches:  
changeset: 7704:847fd35f94de
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Fri Aug 04 19:37:37 2017 +0300
description:
Cache: reset c->body_start when reading a variant on Vary mismatch.

Previously, a variant not present in shared memory and stored on disk using a
secondary key was read using c->body_start from a variant stored with a main
key.  This could result in critical errors "cache file .. has too long header".

diffstat:

 src/http/ngx_http_cache.h      |  1 +
 src/http/ngx_http_file_cache.c |  4 +++-
 2 files changed, 4 insertions(+), 1 deletions(-)

diffs (32 lines):

diff -r da5e3f5b1673 -r 847fd35f94de src/http/ngx_http_cache.h
--- a/src/http/ngx_http_cache.h	Wed Sep 02 23:13:36 2020 +0300
+++ b/src/http/ngx_http_cache.h	Fri Aug 04 19:37:37 2017 +0300
@@ -80,6 +80,7 @@ struct ngx_http_cache_s {
     ngx_str_t                        vary;
     u_char                           variant[NGX_HTTP_CACHE_KEY_LEN];
 
+    size_t                           buffer_size;
     size_t                           header_start;
     size_t                           body_start;
     off_t                            length;
diff -r da5e3f5b1673 -r 847fd35f94de src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c	Wed Sep 02 23:13:36 2020 +0300
+++ b/src/http/ngx_http_file_cache.c	Fri Aug 04 19:37:37 2017 +0300
@@ -294,6 +294,8 @@ ngx_http_file_cache_open(ngx_http_reques
         cln->data = c;
     }
 
+    c->buffer_size = c->body_start;
+
     rc = ngx_http_file_cache_exists(cache, c);
 
     ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
@@ -1230,7 +1232,7 @@ ngx_http_file_cache_reopen(ngx_http_requ
 
     c->secondary = 1;
     c->file.name.len = 0;
-    c->body_start = c->buf->end - c->buf->start;
+    c->body_start = c->buffer_size;
 
     ngx_memcpy(c->key, c->variant, NGX_HTTP_CACHE_KEY_LEN);
 

From pluknet at nginx.com  Fri Sep 11 19:16:46 2020
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Fri, 11 Sep 2020 19:16:46 +0000
Subject: [nginx] Cache: keep c->body_start when Vary changes (ticket #2029).
Message-ID: <hg.3781de64e747.1599851806.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/3781de64e747
branches:  
changeset: 7705:3781de64e747
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Wed Sep 09 19:26:27 2020 +0300
description:
Cache: keep c->body_start when Vary changes (ticket #2029).

If the variant hash doesn't match one we used as a secondary cache key,
we switch back to the original key.  In this case, c->body_start was kept
updated from an existing cache node overwriting the new response value.
After file cache update, it led to discrepancy between a cache node and
cache file seen as critical errors "file cache .. has too long header".

diffstat:

 src/http/ngx_http_cache.h      |  1 +
 src/http/ngx_http_file_cache.c |  3 ++-
 2 files changed, 3 insertions(+), 1 deletions(-)

diffs (31 lines):

diff -r 847fd35f94de -r 3781de64e747 src/http/ngx_http_cache.h
--- a/src/http/ngx_http_cache.h	Fri Aug 04 19:37:37 2017 +0300
+++ b/src/http/ngx_http_cache.h	Wed Sep 09 19:26:27 2020 +0300
@@ -117,6 +117,7 @@ struct ngx_http_cache_s {
     unsigned                         purged:1;
     unsigned                         reading:1;
     unsigned                         secondary:1;
+    unsigned                         update_variant:1;
     unsigned                         background:1;
 
     unsigned                         stale_updating:1;
diff -r 847fd35f94de -r 3781de64e747 src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c	Fri Aug 04 19:37:37 2017 +0300
+++ b/src/http/ngx_http_file_cache.c	Wed Sep 09 19:26:27 2020 +0300
@@ -854,7 +854,7 @@ ngx_http_file_cache_exists(ngx_http_file
         if (fcn->exists || fcn->uses >= c->min_uses) {
 
             c->exists = fcn->exists;
-            if (fcn->body_start) {
+            if (fcn->body_start && !c->update_variant) {
                 c->body_start = fcn->body_start;
             }
 
@@ -1339,6 +1339,7 @@ ngx_http_file_cache_update_variant(ngx_h
     ngx_shmtx_unlock(&cache->shpool->mutex);
 
     c->file.name.len = 0;
+    c->update_variant = 1;
 
     ngx_memcpy(c->key, c->main, NGX_HTTP_CACHE_KEY_LEN);
 

From gabriel at augendre.info  Sat Sep 12 14:02:18 2020
From: gabriel at augendre.info (Gabriel Augendre)
Date: Sat, 12 Sep 2020 16:02:18 +0200
Subject: Why isn't TLS 1.3 included in the defaults ssl_protocols ?
Message-ID: <e4a492fc-ab6a-f637-59d9-e431392f1860@augendre.info>

Hello,

After searching in the archives of this mailing list, I couldn't find 
why the default `ssl_protocols` only include TLSv1 to TLSv1.2 and omit 
TLSv1.3.

Could TLSv1.3 be included in the defaults ssl_protocols?

Cheers,

-- 
Gabriel Augendre

From maxim at nginx.com  Sun Sep 13 09:03:58 2020
From: maxim at nginx.com (Maxim Konovalov)
Date: Sun, 13 Sep 2020 12:03:58 +0300
Subject: [PATCH] Use BPF to distribute packet to different work thread.
In-Reply-To: <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>
References: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200910101132.GA38727@vl.krasnogorsk.ru>
 <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>
Message-ID: <dba3bf90-c223-e2cb-469a-02395e82af96@nginx.com>

Hi LQ,

On 11.09.2020 08:41, Liu, Qiao wrote:
> Hi, Vladimir Homutov:
> The below is our WRK test result output with BPF enable 
> [...]

Thanks for sharing the benchmark results.

Have you verified them on the hosts without ADQ-capable cards and/or in
virtual environments such as modern clouds?

Maxim

-- 
Maxim Konovalov

From mdounin at mdounin.ru  Sun Sep 13 22:25:12 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 14 Sep 2020 01:25:12 +0300
Subject: Why isn't TLS 1.3 included in the defaults ssl_protocols ?
In-Reply-To: <e4a492fc-ab6a-f637-59d9-e431392f1860@augendre.info>
References: <e4a492fc-ab6a-f637-59d9-e431392f1860@augendre.info>
Message-ID: <20200913222512.GT18881@mdounin.ru>

Hello!

On Sat, Sep 12, 2020 at 04:02:18PM +0200, Gabriel Augendre wrote:

> Hello,
> 
> After searching in the archives of this mailing list, I couldn't find 
> why the default `ssl_protocols` only include TLSv1 to TLSv1.2 and omit 
> TLSv1.3.
> 
> Could TLSv1.3 be included in the defaults ssl_protocols?

http://mailman.nginx.org/pipermail/nginx/2018-November/057194.html
http://mailman.nginx.org/pipermail/nginx/2020-June/059541.html

-- 
Maxim Dounin
http://mdounin.ru/

From mdounin at mdounin.ru  Sun Sep 13 23:39:36 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 14 Sep 2020 02:39:36 +0300
Subject: [PATCH] Use BPF to distribute packet to different work thread.
In-Reply-To: <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>
References: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200910101132.GA38727@vl.krasnogorsk.ru>
 <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>
Message-ID: <20200913233936.GV18881@mdounin.ru>

Hello!

On Fri, Sep 11, 2020 at 05:41:47AM +0000, Liu, Qiao wrote:

> Hi, Vladimir Homutov:
> The below is our WRK test result output with BPF enable 
> 
>   112 threads and 10000 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   608.23ms  820.71ms  10.00s    87.48%
>     Connect    16.52ms   54.53ms   1.99s    94.73%
>     Delay     153.13ms  182.17ms   2.00s    90.74%
>     Req/Sec   244.79    142.32     1.99k    68.40%
>   Latency Distribution
>   50.00%  293.50ms
>   75.00%  778.33ms
>   90.00%    1.61s
>   99.00%    3.71s
>   99.90%    7.03s
>   99.99%    8.94s
>   Connect Distribution
>   50.00%    1.93ms
>   75.00%    2.85ms
>   90.00%   55.76ms
>   99.00%  229.19ms
>   99.90%  656.79ms
>   99.99%    1.43s
>   Delay Distribution
>   50.00%  110.96ms
>   75.00%  193.67ms
>   90.00%  321.77ms
>   99.00%  959.27ms
>   99.90%    1.57s
>   99.99%    1.91s
> Compared with no BPF but enable reuseport as below
> 
> 112 threads and 10000 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   680.50ms  943.69ms  10.00s    87.18%
>     Connect    58.44ms  238.08ms   2.00s    94.58%
>     Delay     158.84ms  256.28ms   2.00s    90.92%
>     Req/Sec   244.51    151.00     1.41k    69.67%
>   Latency Distribution
>   50.00%  317.61ms
>   75.00%  913.52ms
>   90.00%    1.90s
>   99.00%    4.30s
>   99.90%    6.52s
>   99.99%    8.80s
>   Connect Distribution
>   50.00%    1.88ms
>   75.00%    2.21ms
>   90.00%   55.94ms
>   99.00%    1.45s
>   99.90%    1.95s
>   99.99%    2.00s
>   Delay Distribution
>   50.00%   73.01ms
>   75.00%  190.40ms
>   90.00%  387.01ms
>   99.00%    1.34s
>   99.90%    1.86s
>   99.99%    1.99s
> 
> 
> From the above results, there shows almost 20% percent latency 
> reduction. P99 latency of BPF is 3.71s , but without BPF is 
> 4.3s.

Thank you for the results.

Given that latency stdev is way higher than the average latency, 
I don't think the "20% percent latency reduction" observed is 
statistically significant.  Please try running several tests and 
use ministat(1) to check the results.

Also, the latency values look very high, and request rate very 
low.  What's on the server side?

-- 
Maxim Dounin
http://mdounin.ru/

From qiao.liu at intel.com  Mon Sep 14 01:17:32 2020
From: qiao.liu at intel.com (Liu, Qiao)
Date: Mon, 14 Sep 2020 01:17:32 +0000
Subject: [PATCH] Use BPF to distribute packet to different work thread.
In-Reply-To: <20200913233936.GV18881@mdounin.ru>
References: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200910101132.GA38727@vl.krasnogorsk.ru>
 <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200913233936.GV18881@mdounin.ru>
Message-ID: <SN6PR11MB3517F8A7E457669E49D3483C87230@SN6PR11MB3517.namprd11.prod.outlook.com>

Hi, Maxim Dounin:
Thanks for your reply, this server is random selected, we just do BPF and no-BPF test, I think the latency based on server configuration, not related with BPF patch, also the NIC
of the server is Mellanox, not ADQ capable hardware , we will do more test
Thanks
LQ

-----Original Message-----
From: nginx-devel <nginx-devel-bounces at nginx.org> On Behalf Of Maxim Dounin
Sent: Monday, September 14, 2020 7:40 AM
To: nginx-devel at nginx.org
Subject: Re: [PATCH] Use BPF to distribute packet to different work thread.

Hello!

On Fri, Sep 11, 2020 at 05:41:47AM +0000, Liu, Qiao wrote:

> Hi, Vladimir Homutov:
> The below is our WRK test result output with BPF enable
> 
>   112 threads and 10000 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   608.23ms  820.71ms  10.00s    87.48%
>     Connect    16.52ms   54.53ms   1.99s    94.73%
>     Delay     153.13ms  182.17ms   2.00s    90.74%
>     Req/Sec   244.79    142.32     1.99k    68.40%
>   Latency Distribution
>   50.00%  293.50ms
>   75.00%  778.33ms
>   90.00%    1.61s
>   99.00%    3.71s
>   99.90%    7.03s
>   99.99%    8.94s
>   Connect Distribution
>   50.00%    1.93ms
>   75.00%    2.85ms
>   90.00%   55.76ms
>   99.00%  229.19ms
>   99.90%  656.79ms
>   99.99%    1.43s
>   Delay Distribution
>   50.00%  110.96ms
>   75.00%  193.67ms
>   90.00%  321.77ms
>   99.00%  959.27ms
>   99.90%    1.57s
>   99.99%    1.91s
> Compared with no BPF but enable reuseport as below
> 
> 112 threads and 10000 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   680.50ms  943.69ms  10.00s    87.18%
>     Connect    58.44ms  238.08ms   2.00s    94.58%
>     Delay     158.84ms  256.28ms   2.00s    90.92%
>     Req/Sec   244.51    151.00     1.41k    69.67%
>   Latency Distribution
>   50.00%  317.61ms
>   75.00%  913.52ms
>   90.00%    1.90s
>   99.00%    4.30s
>   99.90%    6.52s
>   99.99%    8.80s
>   Connect Distribution
>   50.00%    1.88ms
>   75.00%    2.21ms
>   90.00%   55.94ms
>   99.00%    1.45s
>   99.90%    1.95s
>   99.99%    2.00s
>   Delay Distribution
>   50.00%   73.01ms
>   75.00%  190.40ms
>   90.00%  387.01ms
>   99.00%    1.34s
>   99.90%    1.86s
>   99.99%    1.99s
> 
> 
> From the above results, there shows almost 20% percent latency 
> reduction. P99 latency of BPF is 3.71s , but without BPF is 4.3s.

Thank you for the results.

Given that latency stdev is way higher than the average latency, I don't think the "20% percent latency reduction" observed is statistically significant.  Please try running several tests and use ministat(1) to check the results.

Also, the latency values look very high, and request rate very low.  What's on the server side?

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

From qiao.liu at intel.com  Mon Sep 14 01:19:51 2020
From: qiao.liu at intel.com (Liu, Qiao)
Date: Mon, 14 Sep 2020 01:19:51 +0000
Subject: [PATCH] Use BPF to distribute packet to different work thread.
In-Reply-To: <dba3bf90-c223-e2cb-469a-02395e82af96@nginx.com>
References: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200910101132.GA38727@vl.krasnogorsk.ru>
 <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>
 <dba3bf90-c223-e2cb-469a-02395e82af96@nginx.com>
Message-ID: <SN6PR11MB35177235BF8CBFE2251460CC87230@SN6PR11MB3517.namprd11.prod.outlook.com>

Hi, Maxim Konovalov:
Yes, this host has no ADQ-capable cards, the server use Mellanox NIC, if use ADQ-capable cards, the latency improvement will be 40%
Thanks
LQ
-----Original Message-----
From: Maxim Konovalov <maxim at nginx.com> 
Sent: Sunday, September 13, 2020 5:04 PM
To: nginx-devel at nginx.org; Liu, Qiao <qiao.liu at intel.com>
Subject: Re: [PATCH] Use BPF to distribute packet to different work thread.

Hi LQ,

On 11.09.2020 08:41, Liu, Qiao wrote:
> Hi, Vladimir Homutov:
> The below is our WRK test result output with BPF enable [...]

Thanks for sharing the benchmark results.

Have you verified them on the hosts without ADQ-capable cards and/or in virtual environments such as modern clouds?

Maxim

--
Maxim Konovalov

From qiao.liu at intel.com  Tue Sep 15 02:08:45 2020
From: qiao.liu at intel.com (Liu, Qiao)
Date: Tue, 15 Sep 2020 02:08:45 +0000
Subject: [PATCH] Use BPF to distribute packet to different work thread.
In-Reply-To: <SN6PR11MB3517F8A7E457669E49D3483C87230@SN6PR11MB3517.namprd11.prod.outlook.com>
References: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200910101132.GA38727@vl.krasnogorsk.ru>
 <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200913233936.GV18881@mdounin.ru>
 <SN6PR11MB3517F8A7E457669E49D3483C87230@SN6PR11MB3517.namprd11.prod.outlook.com>
Message-ID: <SN6PR11MB3517E97F77028B2A11C3B9EF87200@SN6PR11MB3517.namprd11.prod.outlook.com>

Below is 5 times test result compare, 112 threads, 10000 connections, 1M object http request. Seems P99 have great improvement, and Max is also reduced



                                   AVG          Stdev            Max        P99
                  test 1      1.32s        447.09ms     5.48s      2.82s
BPF           test 2      1.39s        513.8ms       9.42s      3.1s
                  test 3      1.4s          341.38ms     5.63s      2.55s
                  test 4      1.41s        407.45ms     6.96s      2.77s
                  test 5      1.29s        644.81ms     9.45s      3.74s
                 Average  1.362s      470.906ms   7.388s    2.996s

NonBPF   test 1      1.48s         916.88ms     9.44s       5.08s
                 test 2      1.43s         658.48ms     9.54s       3.92s
                 test 3      1.41s         650.38ms     8.63s       3.59s
                 test 4      1.29s         1010ms        10s           5.21s
                 test 5      1.31s         875.01ms     9.53s       4.39s
             Average     1.384s        822.15ms    9.428s    4.438s


Thanks
LQ
-----Original Message-----
From: nginx-devel <nginx-devel-bounces at nginx.org> On Behalf Of Liu, Qiao
Sent: Monday, September 14, 2020 9:18 AM
To: nginx-devel at nginx.org
Subject: RE: [PATCH] Use BPF to distribute packet to different work thread.

Hi, Maxim Dounin:
Thanks for your reply, this server is random selected, we just do BPF and no-BPF test, I think the latency based on server configuration, not related with BPF patch, also the NIC of the server is Mellanox, not ADQ capable hardware , we will do more test Thanks LQ

-----Original Message-----
From: nginx-devel <nginx-devel-bounces at nginx.org> On Behalf Of Maxim Dounin
Sent: Monday, September 14, 2020 7:40 AM
To: nginx-devel at nginx.org
Subject: Re: [PATCH] Use BPF to distribute packet to different work thread.

Hello!

On Fri, Sep 11, 2020 at 05:41:47AM +0000, Liu, Qiao wrote:

> Hi, Vladimir Homutov:
> The below is our WRK test result output with BPF enable
> 
>   112 threads and 10000 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   608.23ms  820.71ms  10.00s    87.48%
>     Connect    16.52ms   54.53ms   1.99s    94.73%
>     Delay     153.13ms  182.17ms   2.00s    90.74%
>     Req/Sec   244.79    142.32     1.99k    68.40%
>   Latency Distribution
>   50.00%  293.50ms
>   75.00%  778.33ms
>   90.00%    1.61s
>   99.00%    3.71s
>   99.90%    7.03s
>   99.99%    8.94s
>   Connect Distribution
>   50.00%    1.93ms
>   75.00%    2.85ms
>   90.00%   55.76ms
>   99.00%  229.19ms
>   99.90%  656.79ms
>   99.99%    1.43s
>   Delay Distribution
>   50.00%  110.96ms
>   75.00%  193.67ms
>   90.00%  321.77ms
>   99.00%  959.27ms
>   99.90%    1.57s
>   99.99%    1.91s
> Compared with no BPF but enable reuseport as below
> 
> 112 threads and 10000 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   680.50ms  943.69ms  10.00s    87.18%
>     Connect    58.44ms  238.08ms   2.00s    94.58%
>     Delay     158.84ms  256.28ms   2.00s    90.92%
>     Req/Sec   244.51    151.00     1.41k    69.67%
>   Latency Distribution
>   50.00%  317.61ms
>   75.00%  913.52ms
>   90.00%    1.90s
>   99.00%    4.30s
>   99.90%    6.52s
>   99.99%    8.80s
>   Connect Distribution
>   50.00%    1.88ms
>   75.00%    2.21ms
>   90.00%   55.94ms
>   99.00%    1.45s
>   99.90%    1.95s
>   99.99%    2.00s
>   Delay Distribution
>   50.00%   73.01ms
>   75.00%  190.40ms
>   90.00%  387.01ms
>   99.00%    1.34s
>   99.90%    1.86s
>   99.99%    1.99s
> 
> 
> From the above results, there shows almost 20% percent latency 
> reduction. P99 latency of BPF is 3.71s , but without BPF is 4.3s.

Thank you for the results.

Given that latency stdev is way higher than the average latency, I don't think the "20% percent latency reduction" observed is statistically significant.  Please try running several tests and use ministat(1) to check the results.

Also, the latency values look very high, and request rate very low.  What's on the server side?

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

From alexander.borisov at nginx.com  Wed Sep 16 14:23:05 2020
From: alexander.borisov at nginx.com (Alexander Borisov)
Date: Wed, 16 Sep 2020 14:23:05 +0000
Subject: [njs] Introduced Buffer implementation.
Message-ID: <hg.27bb9caf186c.1600266185.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/27bb9caf186c
branches:  
changeset: 1523:27bb9caf186c
user:      Alexander Borisov <alexander.borisov at nginx.com>
date:      Wed Sep 16 17:22:01 2020 +0300
description:
Introduced Buffer implementation.

diffstat:

 auto/sources                  |     1 +
 src/njs.h                     |     6 +-
 src/njs_array_buffer.c        |    16 +-
 src/njs_array_buffer.h        |     5 +-
 src/njs_buffer.c              |  2944 +++++++++++++++++++++++++++++++++++++++++
 src/njs_buffer.h              |    19 +
 src/njs_builtin.c             |    14 +
 src/njs_encoding.c            |     4 +-
 src/njs_main.h                |     1 +
 src/njs_object_hash.h         |    10 +
 src/njs_string.c              |    42 +
 src/njs_string.h              |     4 +
 src/njs_typed_array.c         |    67 +-
 src/njs_typed_array.h         |     7 +-
 src/njs_utils.h               |    12 +
 src/njs_value.c               |     1 +
 src/njs_value.h               |     1 +
 src/njs_vm.c                  |     6 +-
 src/njs_vm.h                  |     1 +
 src/test/njs_externals_test.c |     9 +-
 src/test/njs_unit_test.c      |   936 ++++++++++++-
 21 files changed, 4011 insertions(+), 95 deletions(-)

diffs (truncated from 4450 to 1000 lines):

diff -r e1dccc5a2f05 -r 27bb9caf186c auto/sources
--- a/auto/sources	Fri Sep 11 13:54:57 2020 +0000
+++ b/auto/sources	Wed Sep 16 17:22:01 2020 +0300
@@ -57,6 +57,7 @@ NJS_LIB_SRCS=" \
    src/njs_promise.c \
    src/njs_query_string.c \
    src/njs_encoding.c \
+   src/njs_buffer.c \
 "
 
 NJS_LIB_TEST_SRCS=" \
diff -r e1dccc5a2f05 -r 27bb9caf186c src/njs.h
--- a/src/njs.h	Fri Sep 11 13:54:57 2020 +0000
+++ b/src/njs.h	Wed Sep 16 17:22:01 2020 +0300
@@ -317,11 +317,11 @@ NJS_EXPORT njs_int_t njs_vm_value_string
     njs_value_t *value, uintptr_t *next);
 
 /*
- * Sets a Uint8Array value.
+ * Sets a Buffer value.
  *   start data is not copied and should not be freed.
  */
-NJS_EXPORT njs_int_t njs_vm_value_typed_array_uint8_set(njs_vm_t *vm,
-    njs_value_t *value, const u_char *start, uint32_t size);
+NJS_EXPORT njs_int_t njs_vm_value_buffer_set(njs_vm_t *vm, njs_value_t *value,
+    const u_char *start, uint32_t size);
 
 /*
  * Converts a value to string.
diff -r e1dccc5a2f05 -r 27bb9caf186c src/njs_array_buffer.c
--- a/src/njs_array_buffer.c	Fri Sep 11 13:54:57 2020 +0000
+++ b/src/njs_array_buffer.c	Wed Sep 16 17:22:01 2020 +0300
@@ -8,7 +8,7 @@
 
 
 njs_array_buffer_t *
-njs_array_buffer_alloc(njs_vm_t *vm, uint64_t size)
+njs_array_buffer_alloc(njs_vm_t *vm, uint64_t size, njs_bool_t zeroing)
 {
     njs_object_t        *proto;
     njs_array_buffer_t  *array;
@@ -22,11 +22,15 @@ njs_array_buffer_alloc(njs_vm_t *vm, uin
         goto memory_error;
     }
 
-    if (size > 0) {
+    if (zeroing) {
         array->u.data = njs_mp_zalloc(vm->mem_pool, size);
-        if (njs_slow_path(array->u.data == NULL)) {
-            goto memory_error;
-        }
+
+    } else {
+        array->u.data = njs_mp_alloc(vm->mem_pool, size);
+    }
+
+    if (njs_slow_path(array->u.data == NULL)) {
+        goto memory_error;
     }
 
     proto = &vm->prototypes[NJS_OBJ_TYPE_ARRAY_BUFFER].object;
@@ -80,7 +84,7 @@ njs_array_buffer_constructor(njs_vm_t *v
         return NJS_ERROR;
     }
 
-    array = njs_array_buffer_alloc(vm, size);
+    array = njs_array_buffer_alloc(vm, size, 1);
     if (njs_slow_path(array == NULL)) {
         return NJS_ERROR;
     }
diff -r e1dccc5a2f05 -r 27bb9caf186c src/njs_array_buffer.h
--- a/src/njs_array_buffer.h	Fri Sep 11 13:54:57 2020 +0000
+++ b/src/njs_array_buffer.h	Wed Sep 16 17:22:01 2020 +0300
@@ -12,7 +12,8 @@
     ((buffer)->size)
 
 
-njs_array_buffer_t *njs_array_buffer_alloc(njs_vm_t *vm, uint64_t size);
+njs_array_buffer_t *njs_array_buffer_alloc(njs_vm_t *vm, uint64_t size,
+    njs_bool_t zeroing);
 njs_int_t njs_array_buffer_writable(njs_vm_t *vm, njs_array_buffer_t *buffer);
 
 njs_inline njs_array_buffer_t *
@@ -29,7 +30,7 @@ njs_array_buffer_slice(njs_vm_t *vm, njs
 
     new_len = njs_max(final - first, 0);
 
-    new_buffer = njs_array_buffer_alloc(vm, new_len);
+    new_buffer = njs_array_buffer_alloc(vm, new_len, 1);
     if (new_buffer == NULL) {
         return NULL;
     }
diff -r e1dccc5a2f05 -r 27bb9caf186c src/njs_buffer.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/src/njs_buffer.c	Wed Sep 16 17:22:01 2020 +0300
@@ -0,0 +1,2944 @@
+
+/*
+ * Copyright (C) Alexander Borisov
+ * Copyright (C) Dmitry Volyntsev
+ * Copyright (C) NGINX, Inc.
+ */
+
+
+#include <njs_main.h>
+
+
+#define INT24_MAX  0x7FFFFF
+#define INT40_MAX  0x7FFFFFFFFFULL
+#define INT48_MAX  0x7FFFFFFFFFFFULL
+
+#define njs_buffer_magic(size, sign, little)                                 \
+    ((size << 2) | (sign << 1) | little)
+
+
+typedef njs_int_t (*njs_buffer_encode)(njs_vm_t *vm, njs_value_t *value,
+    const njs_str_t *src);
+typedef size_t (*njs_buffer_encode_length)(const njs_str_t *src,
+                                           size_t *out_size);
+
+typedef struct {
+    njs_str_t                 name;
+    njs_buffer_encode         encode;
+    njs_buffer_encode         decode;
+    njs_buffer_encode_length  decode_length;
+} njs_buffer_encoding_t;
+
+
+static njs_buffer_encoding_t  njs_buffer_encodings[] =
+{
+    {
+        njs_str("utf-8"),
+        njs_string_decode_utf8,
+        njs_string_decode_utf8,
+        njs_decode_utf8_length
+    },
+
+    {
+        njs_str("utf8"),
+        njs_string_decode_utf8,
+        njs_string_decode_utf8,
+        njs_decode_utf8_length
+    },
+
+    {
+        njs_str("hex"),
+        njs_string_hex,
+        njs_string_decode_hex,
+        njs_decode_hex_length
+    },
+
+    {
+        njs_str("base64"),
+        njs_string_base64,
+        njs_string_decode_base64,
+        njs_decode_base64_length
+    },
+
+    {
+        njs_str("base64url"),
+        njs_string_base64url,
+        njs_string_decode_base64url,
+        njs_decode_base64url_length
+    },
+
+    { njs_null_str, 0, 0, 0 }
+
+#define njs_buffer_utf8_encoding()  &njs_buffer_encodings[0]
+};
+
+
+static njs_int_t njs_buffer_from_object(njs_vm_t *vm, njs_value_t *value);
+static njs_int_t njs_buffer_from_array_buffer(njs_vm_t *vm, njs_value_t *value,
+    njs_value_t *length, njs_value_t *offset);
+static njs_int_t njs_buffer_from_typed_array(njs_vm_t *vm, njs_value_t *value);
+static njs_int_t njs_buffer_from_string(njs_vm_t *vm, njs_value_t *value,
+    const njs_buffer_encoding_t *encoding);
+static njs_int_t njs_buffer_write_string(njs_vm_t *vm, njs_value_t *value,
+    njs_typed_array_t *array, const njs_buffer_encoding_t *encoding,
+    uint64_t offset, uint64_t length);
+static njs_int_t njs_buffer_fill(njs_vm_t *vm, njs_typed_array_t *array,
+    njs_value_t *value, const njs_buffer_encoding_t *encoding, uint64_t offset,
+    uint64_t end);
+static njs_int_t njs_buffer_fill_string(njs_vm_t *vm, njs_value_t *value,
+    njs_typed_array_t *array, const njs_buffer_encoding_t *encoding,
+    uint8_t *start, uint8_t *end);
+static njs_int_t njs_buffer_fill_typed_array(njs_vm_t *vm, njs_value_t *value,
+    njs_typed_array_t *array, uint8_t *start, uint8_t *end);
+static const njs_buffer_encoding_t *njs_buffer_encoding(njs_vm_t *vm,
+    njs_value_t *value);
+static njs_int_t njs_buffer_decode_string(njs_vm_t *vm, njs_value_t *value,
+    njs_value_t *dst, const njs_buffer_encoding_t *encoding);
+static void njs_buffer_decode_destroy(njs_vm_t *vm, njs_value_t *source,
+    njs_value_t *target);
+
+
+njs_int_t
+njs_buffer_set(njs_vm_t *vm, njs_value_t *value, const u_char *start,
+    uint32_t size)
+{
+    njs_object_t        *proto;
+    njs_typed_array_t   *array;
+    njs_array_buffer_t  *buffer;
+
+    array = njs_mp_alloc(vm->mem_pool, sizeof(njs_typed_array_t)
+                                       + sizeof(njs_array_buffer_t));
+    if (njs_slow_path(array == NULL)) {
+        njs_memory_error(vm);
+        return NJS_ERROR;
+    }
+
+    buffer = (njs_array_buffer_t *) &array[1];
+
+    proto = &vm->prototypes[NJS_OBJ_TYPE_BUFFER].object;
+
+    njs_lvlhsh_init(&buffer->object.hash);
+    njs_lvlhsh_init(&buffer->object.shared_hash);
+    buffer->object.__proto__ = proto;
+    buffer->object.slots = NULL;
+    buffer->object.type = NJS_ARRAY_BUFFER;
+    buffer->object.shared = 1;
+    buffer->object.extensible = 1;
+    buffer->object.error_data = 0;
+    buffer->object.fast_array = 0;
+    buffer->u.data = (void *) start;
+    buffer->size = size;
+
+    array->type = NJS_OBJ_TYPE_UINT8_ARRAY;
+    njs_lvlhsh_init(&array->object.hash);
+    njs_lvlhsh_init(&array->object.shared_hash);
+    array->object.__proto__ = &vm->prototypes[array->type].object;
+    array->object.slots = NULL;
+    array->object.type = NJS_TYPED_ARRAY;
+    array->object.shared = 0;
+    array->object.extensible = 1;
+    array->object.error_data = 0;
+    array->object.fast_array = 1;
+    array->buffer = buffer;
+    array->offset = 0;
+    array->byte_length = size;
+
+    njs_set_typed_array(value, array);
+
+    return NJS_OK;
+}
+
+
+static njs_typed_array_t *
+njs_buffer_alloc_array(njs_vm_t *vm, size_t size, njs_bool_t zeroing)
+{
+    njs_value_t        value;
+    njs_typed_array_t  *array;
+
+    njs_set_number(&value, size);
+
+    array = njs_typed_array_alloc(vm, &value, 1, zeroing,
+                                  NJS_OBJ_TYPE_UINT8_ARRAY);
+    if (njs_slow_path(array == NULL)) {
+        return NULL;
+    }
+
+    array->object.__proto__ = &vm->prototypes[NJS_OBJ_TYPE_BUFFER].object;
+
+    return array;
+}
+
+
+static njs_int_t
+njs_buffer_constructor(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    njs_type_error(vm, "Buffer is not a constructor");
+
+    return NJS_ERROR;
+}
+
+
+static njs_int_t
+njs_buffer_alloc(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t safe)
+{
+    double                       size;
+    njs_int_t                    ret;
+    njs_typed_array_t            *array;
+    const njs_buffer_encoding_t  *encoding;
+
+    if (njs_slow_path(!njs_is_number(njs_arg(args, nargs, 1)))) {
+        njs_type_error(vm, "\"size\" argument must be of type number");
+        return NJS_ERROR;
+    }
+
+    size = njs_number(njs_argument(args, 1));
+    if (njs_slow_path(size < 0 || size > INT32_MAX)) {
+        njs_range_error(vm, "invalid size");
+        return NJS_ERROR;
+    }
+
+    array = njs_buffer_alloc_array(vm, size, safe || nargs <= 2);
+    if (njs_slow_path(array == NULL)) {
+        return NJS_ERROR;
+    }
+
+    if (safe && nargs > 2) {
+        encoding = njs_buffer_utf8_encoding();
+
+        if (nargs > 3 && njs_is_string(njs_argument(args, 2))) {
+            encoding = njs_buffer_encoding(vm, njs_argument(args, 3));
+            if (njs_slow_path(encoding == NULL)) {
+                return NJS_ERROR;
+            }
+        }
+
+        ret = njs_buffer_fill(vm, array, njs_argument(args, 2), encoding, 0,
+                              array->byte_length);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_ERROR;
+        }
+    }
+
+    njs_set_typed_array(&vm->retval, array);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_from(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    njs_int_t                    ret;
+    njs_value_t                  *value;
+    const njs_buffer_encoding_t  *encoding;
+
+    value = njs_arg(args, nargs, 1);
+
+    switch (value->type) {
+    case NJS_TYPED_ARRAY:
+        return njs_buffer_from_typed_array(vm, value);
+
+    case NJS_ARRAY_BUFFER:
+        return njs_buffer_from_array_buffer(vm, value, njs_arg(args, nargs, 2),
+                                            njs_arg(args, nargs, 3));
+
+    case NJS_STRING:
+        encoding = njs_buffer_utf8_encoding();
+
+        if (nargs > 2) {
+            encoding = njs_buffer_encoding(vm, njs_argument(args, 2));
+            if (njs_slow_path(encoding == NULL)) {
+                return NJS_ERROR;
+            }
+        }
+
+        return njs_buffer_from_string(vm, value, encoding);
+
+    default:
+        if (njs_is_object(value)) {
+            ret = njs_buffer_from_object(vm, value);
+            if (njs_slow_path(ret != NJS_DECLINED)) {
+                return ret;
+            }
+        }
+
+        njs_type_error(vm, "first argument %s is not a string "
+                       "or Buffer-like object", njs_type_string(value->type));
+    }
+
+    return NJS_ERROR;
+}
+
+
+static njs_int_t
+njs_buffer_from_object(njs_vm_t *vm, njs_value_t *value)
+{
+    double             num;
+    int64_t            len;
+    uint8_t            *p;
+    uint32_t           i;
+    njs_str_t          str;
+    njs_int_t          ret;
+    njs_array_t        *array;
+    njs_value_t        retval, length;
+    njs_typed_array_t  *buffer;
+
+    static const njs_value_t  string_length = njs_string("length");
+    static const njs_str_t  str_buffer = njs_str("Buffer");
+
+next:
+
+    ret = njs_value_property(vm, value, njs_value_arg(&string_length),
+                             &length);
+    if (njs_slow_path(ret == NJS_ERROR)) {
+        return ret;
+    }
+
+    if (ret == NJS_DECLINED) {
+        ret = njs_value_property(vm, value, njs_value_arg(&njs_string_type),
+                                 &retval);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_DECLINED;
+        }
+
+        ret = njs_value_to_string(vm, &retval, &retval);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_DECLINED;
+        }
+
+        njs_string_get(&retval, &str);
+
+        if (!njs_strstr_eq(&str, &str_buffer)) {
+            return NJS_DECLINED;
+        }
+
+        ret = njs_value_property(vm, value, njs_value_arg(&njs_string_data),
+                                 &retval);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_DECLINED;
+        }
+
+        if (njs_is_object(&retval)) {
+            value = &retval;
+            goto next;
+        }
+
+        return NJS_DECLINED;
+    }
+
+    ret = njs_value_to_length(vm, &length, &len);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    buffer = njs_buffer_alloc_array(vm, len, 0);
+    if (njs_slow_path(buffer == NULL)) {
+        return NJS_ERROR;
+    }
+
+    p = njs_typed_array_buffer(buffer)->u.u8;
+
+    if (njs_is_fast_array(value)) {
+        array = njs_array(value);
+
+        for (i = 0; i < array->length; i++) {
+            ret = njs_value_to_number(vm, &array->start[i], &num);
+            if (njs_slow_path(ret != NJS_OK)) {
+                return ret;
+            }
+
+            *p++ = njs_number_to_int32(num);
+        }
+
+        njs_set_typed_array(&vm->retval, buffer);
+
+        return NJS_OK;
+    }
+
+    for (i = 0; i < len; i++) {
+        ret = njs_value_property_i64(vm, value, i, &retval);
+        if (njs_slow_path(ret == NJS_ERROR)) {
+            return ret;
+        }
+
+        ret = njs_value_to_number(vm, &retval, &num);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
+        }
+
+        *p++ = njs_number_to_int32(num);
+    }
+
+    njs_set_typed_array(&vm->retval, buffer);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_from_array_buffer(njs_vm_t *vm, njs_value_t *value,
+    njs_value_t *offset, njs_value_t *length)
+{
+    int64_t             off, len;
+    njs_int_t           ret;
+    njs_value_t         arg;
+    njs_typed_array_t   *buffer;
+    njs_array_buffer_t  *array;
+
+    array = njs_array_buffer(value);
+
+    ret = njs_value_to_index(vm, offset, (uint64_t *) &off);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    if ((size_t) off > array->size) {
+        njs_range_error(vm, "\"offset\" is outside of buffer bounds");
+        return NJS_ERROR;
+    }
+
+    if (njs_is_defined(length)) {
+        ret = njs_value_to_length(vm, length, &len);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
+        }
+
+    } else {
+        len = array->size - off;
+    }
+
+    if ((size_t) (off + len) > array->size) {
+        njs_range_error(vm, "\"length\" is outside of buffer bounds");
+        return NJS_ERROR;
+    }
+
+    njs_set_array_buffer(&arg, array);
+
+    buffer = njs_typed_array_alloc(vm, &arg, 1, 0, NJS_OBJ_TYPE_UINT8_ARRAY);
+    if (njs_slow_path(buffer == NULL)) {
+        return NJS_ERROR;
+    }
+
+    buffer->object.__proto__ = &vm->prototypes[NJS_OBJ_TYPE_BUFFER].object;
+
+    buffer->offset = off;
+    buffer->byte_length = len;
+
+    njs_set_typed_array(&vm->retval, buffer);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_from_typed_array(njs_vm_t *vm, njs_value_t *value)
+{
+    uint8_t            *p;
+    uint32_t           i, length;
+    njs_typed_array_t  *buffer, *array;
+
+    array = njs_typed_array(value);
+    length = njs_typed_array_length(array);
+
+    buffer = njs_buffer_alloc_array(vm, length, 0);
+    if (njs_slow_path(buffer == NULL)) {
+        return NJS_ERROR;
+    }
+
+    p = njs_typed_array_buffer(buffer)->u.u8;
+
+    for (i = 0; i < length; i++) {
+        *p++ = njs_number_to_int32(njs_typed_array_prop(array, i));
+    }
+
+    njs_set_typed_array(&vm->retval, buffer);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_from_string(njs_vm_t *vm, njs_value_t *value,
+    const njs_buffer_encoding_t *encoding)
+{
+    njs_int_t          ret;
+    njs_str_t          str;
+    njs_value_t        dst;
+    njs_typed_array_t  *buffer;
+
+    ret = njs_buffer_decode_string(vm, value, &dst, encoding);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return NJS_ERROR;
+    }
+
+    njs_string_get(&dst, &str);
+
+    buffer = njs_buffer_alloc_array(vm, str.length, 0);
+    if (njs_slow_path(buffer == NULL)) {
+        return NJS_ERROR;
+    }
+
+    memcpy(njs_typed_array_buffer(buffer)->u.u8, str.start, str.length);
+
+    njs_buffer_decode_destroy(vm, value, &dst);
+
+    njs_set_typed_array(&vm->retval, buffer);
+
+    return NJS_OK;
+}
+
+
+static size_t
+njs_buffer_decode_string_length(njs_value_t *value,
+    const njs_buffer_encoding_t *encoding)
+{
+    size_t             size;
+    njs_str_t          str;
+    njs_string_prop_t  string;
+
+    (void) njs_string_prop(&string, value);
+
+    str.start = string.start;
+    str.length = string.size;
+    size = string.size;
+
+    if (encoding->decode == njs_string_decode_utf8 && string.length != 0) {
+        return size;
+    }
+
+    encoding->decode_length(&str, &size);
+
+    return size;
+}
+
+
+static njs_int_t
+njs_buffer_byte_length(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    size_t                       size;
+    njs_value_t                  *value, *enc;
+    const njs_buffer_encoding_t  *encoding;
+
+    value = njs_arg(args, nargs, 1);
+
+    switch (value->type) {
+    case NJS_TYPED_ARRAY:
+        njs_set_number(&vm->retval, njs_typed_array(value)->byte_length);
+        return NJS_OK;
+
+    case NJS_ARRAY_BUFFER:
+        njs_set_number(&vm->retval, njs_array_buffer(value)->size);
+        return NJS_OK;
+
+    case NJS_DATA_VIEW:
+        njs_set_number(&vm->retval, njs_data_view(value)->byte_length);
+        return NJS_OK;
+
+    case NJS_STRING:
+        enc = njs_arg(args, nargs, 2);
+        encoding = njs_buffer_utf8_encoding();
+
+        if (njs_is_defined(enc)) {
+            encoding = njs_buffer_encoding(vm, enc);
+            if (njs_slow_path(encoding == NULL)) {
+                return NJS_ERROR;
+            }
+        }
+
+        size = njs_buffer_decode_string_length(value, encoding);
+
+        njs_set_number(&vm->retval, size);
+
+        return NJS_OK;
+
+    default:
+        njs_type_error(vm, "first argument %s is not a string "
+                       "or Buffer-like object", njs_type_string(value->type));
+    }
+
+    return NJS_ERROR;
+}
+
+
+static njs_typed_array_t *
+njs_buffer_slot(njs_vm_t *vm, njs_value_t *value, const char *name)
+{
+    njs_typed_array_t  *array;
+
+    if (njs_slow_path(!njs_is_object(value))) {
+        goto failed;
+    }
+
+    array = njs_object_proto_lookup(njs_object(value), NJS_TYPED_ARRAY,
+                                    njs_typed_array_t);
+
+    if (njs_slow_path(array != NULL
+                      && array->type != NJS_OBJ_TYPE_UINT8_ARRAY))
+    {
+        goto failed;
+    }
+
+    return array;
+
+failed:
+
+    njs_type_error(vm, "\"%s\" argument must be an instance "
+                       "of Buffer or Uint8Array", name);
+    return NULL;
+}
+
+
+static njs_int_t
+njs_buffer_array_range(njs_vm_t *vm, njs_typed_array_t *array,
+    const njs_value_t *start, const njs_value_t *end, const char *name,
+    uint8_t **out_start, uint8_t **out_end)
+{
+    uint8_t    *u8;
+    uint64_t   num_start, num_end;
+    njs_int_t  ret;
+
+    num_start = 0;
+
+    if (njs_is_defined(start)) {
+        ret = njs_value_to_index(vm, njs_value_arg(start), &num_start);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
+        }
+    }
+
+    if (num_start > array->byte_length) {
+        njs_range_error(vm, "\"%sStart\" is out of range: %L", name, num_start);
+        return NJS_ERROR;
+    }
+
+    num_end = array->byte_length;
+
+    if (njs_is_defined(end)) {
+        ret = njs_value_to_index(vm, njs_value_arg(end), &num_end);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return ret;
+        }
+    }
+
+    if (num_end > array->byte_length) {
+        njs_range_error(vm, "\"%sEnd\" is out of range: %L", name, num_end);
+        return NJS_ERROR;
+    }
+
+    if (num_start > num_end) {
+        num_end = num_start;
+    }
+
+    u8 = njs_typed_array_buffer(array)->u.u8;
+    *out_start = &u8[array->offset + num_start];
+    *out_end = &u8[array->offset + num_end];
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_compare_array(njs_vm_t *vm, njs_value_t *val1, njs_value_t *val2,
+    const njs_value_t *target_start, const njs_value_t *target_end,
+    const njs_value_t *source_start, const njs_value_t *source_end)
+{
+    size_t             size, src_size, trg_size;
+    uint8_t            *src, *src_end, *trg, *trg_end;
+    njs_int_t          ret;
+    njs_typed_array_t  *source, *target;
+
+    source = njs_buffer_slot(vm , val1, "source");
+    if (njs_slow_path(source == NULL)) {
+        return NJS_ERROR;
+    }
+
+    target = njs_buffer_slot(vm , val2, "target");
+    if (njs_slow_path(target == NULL)) {
+        return NJS_ERROR;
+    }
+
+    ret = njs_buffer_array_range(vm, target, target_start, target_end, "target",
+                                 &trg, &trg_end);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    ret = njs_buffer_array_range(vm, source, source_start, source_end, "source",
+                                 &src, &src_end);
+    if (njs_slow_path(ret != NJS_OK)) {
+        return ret;
+    }
+
+    trg_size = trg_end - trg;
+    src_size = src_end - src;
+
+    size = njs_min(trg_size, src_size);
+
+    ret = memcmp(trg, src, size);
+
+    if (ret != 0) {
+        njs_set_number(&vm->retval, (ret < 0) ? 1 : -1);
+        return NJS_OK;
+    }
+
+    if (trg_size > src_size) {
+        ret = -1;
+
+    } else if (trg_size < src_size) {
+        ret = 1;
+    }
+
+    njs_set_number(&vm->retval, ret);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_compare(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    return njs_buffer_compare_array(vm, njs_arg(args, nargs, 1),
+                                    njs_arg(args, nargs, 2),
+                                    &njs_value_undefined, &njs_value_undefined,
+                                    &njs_value_undefined, &njs_value_undefined);
+}
+
+
+static njs_int_t
+njs_buffer_concat(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    u_char             *p;
+    size_t             n;
+    int64_t            i, len, list_len;
+    njs_int_t          ret;
+    njs_value_t        *list, *value, *length, retval;
+    njs_array_t        *array;
+    njs_typed_array_t  *buffer, *arr;
+
+    list = njs_arg(args, nargs, 1);
+
+    if (njs_slow_path(!njs_is_array(list))) {
+        njs_type_error(vm, "\"list\" argument must be an instance of Array");
+        return NJS_ERROR;
+    }
+
+    len = 0;
+    ret = njs_object_length(vm, list, &list_len);
+    if (njs_slow_path(ret == NJS_ERROR)) {
+        return ret;
+    }
+
+    if (njs_is_fast_array(list)) {
+        array = njs_array(list);
+        for (i = 0; i < list_len; i++) {
+            value = &array->start[i];
+
+            if (njs_slow_path(!njs_is_typed_array_uint8(value))) {
+                njs_type_error(vm, "\"list[%L]\" argument must be an "
+                                   "instance of Buffer or Uint8Array", i);
+                return NJS_ERROR;
+            }
+
+            arr = njs_typed_array(value);
+
+            if (njs_slow_path((SIZE_MAX - len) < arr->byte_length)) {
+                njs_type_error(vm, "Invalid length");
+                return NJS_ERROR;
+            }
+
+            len += arr->byte_length;
+        }
+
+    } else {
+
+        for (i = 0; i < list_len; i++) {
+            ret = njs_value_property_i64(vm, list, i, &retval);
+            if (njs_slow_path(ret == NJS_ERROR)) {
+                return ret;
+            }
+
+            if (njs_slow_path(!njs_is_typed_array(&retval))) {
+                njs_type_error(vm, "\"list[%L]\" argument must be an "
+                                   "instance of Buffer or Uint8Array", i);
+                return NJS_ERROR;
+            }
+
+            arr = njs_typed_array(&retval);
+
+            if (njs_slow_path((SIZE_MAX - len) < arr->byte_length)) {
+                njs_type_error(vm, "Invalid length");
+                return NJS_ERROR;
+            }
+
+            len += arr->byte_length;
+        }
+    }
+
+    length = njs_arg(args, nargs, 2);
+    if (njs_is_defined(length)) {
+        if (njs_slow_path(!njs_is_number(length))) {
+            njs_type_error(vm, "\"length\" argument must be of type number");
+            return NJS_ERROR;
+        }
+
+        len = njs_number(length);
+        if (njs_slow_path(len < 0)) {
+            njs_range_error(vm, "\"length\" is out of range");
+            return NJS_ERROR;
+        }
+    }
+
+    buffer = njs_buffer_alloc_array(vm, len, 0);
+    if (njs_slow_path(buffer == NULL)) {
+        return NJS_ERROR;
+    }
+
+    p = njs_typed_array_buffer(buffer)->u.u8;
+
+    if (njs_is_fast_array(list)) {
+        array = njs_array(list);
+
+        for (i = 0; len != 0 && i < list_len; i++) {
+            arr = njs_typed_array(&array->start[i]);
+            n = njs_min((size_t) len, arr->byte_length);
+
+            p = njs_cpymem(p, njs_typed_array_buffer(arr)->u.u8, n);
+
+            len -= n;
+        }
+
+    } else {
+        for (i = 0; len != 0 && i < list_len; i++) {
+            ret = njs_value_property_i64(vm, list, i, &retval);
+            if (njs_slow_path(ret == NJS_ERROR)) {
+                return ret;
+            }
+
+            arr = njs_typed_array(&retval);
+            n = njs_min((size_t) len, arr->byte_length);
+
+            p = njs_cpymem(p, njs_typed_array_buffer(arr)->u.u8, n);
+
+            len -= n;
+        }
+    }
+
+    if (len != 0) {
+        njs_memzero(p, len);
+    }
+
+    njs_set_typed_array(&vm->retval, buffer);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_is_buffer(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    njs_bool_t         is;
+    njs_typed_array_t  *array;
+
+    is = 0;
+
+    array = njs_buffer_slot(vm , njs_arg(args, nargs, 1), "source");
+
+    if (njs_fast_path(array != NULL && array->object.__proto__
+                      == &vm->prototypes[NJS_OBJ_TYPE_BUFFER].object))
+    {
+        is = 1;
+    }
+
+    njs_set_boolean(&vm->retval, is);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_is_encoding(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    njs_set_boolean(&vm->retval,
+                    njs_buffer_encoding(vm, njs_arg(args, nargs, 1)) != NULL);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_prototype_length(njs_vm_t *vm, njs_object_prop_t *prop,
+    njs_value_t *value, njs_value_t *setval, njs_value_t *retval)
+{
+    njs_typed_array_t  *array;
+
+    array = njs_buffer_slot(vm, value, "this");
+    if (njs_slow_path(array == NULL)) {
+        njs_set_undefined(retval);
+        return NJS_DECLINED;
+    }
+
+    njs_set_number(retval, array->byte_length);
+
+    return NJS_OK;
+}
+
+
+static njs_int_t
+njs_buffer_prototype_read_int(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t magic)
+{
+    double              v;
+    uint32_t            u32;
+    uint64_t            u64, index, size;
+    njs_int_t           ret;
+    njs_bool_t          little, swap, sign;
+    njs_value_t         *this, *value;
+    const uint8_t       *u8;

From ct at flyingcircus.io  Wed Sep 16 15:28:57 2020
From: ct at flyingcircus.io (Christian Theune)
Date: Wed, 16 Sep 2020 17:28:57 +0200
Subject: (draft) Privacy by design - offer more convenient way to anonymize
 IPs in access log by default 
Message-ID: <117E0BAF-302D-41DC-97FE-C6346E018833@flyingcircus.io>

Hi!

first time posting here. I started working on a builtin-way for nginx to provide GDPR conformaing access logs reliably by default:

https://github.com/nginx/nginx/compare/branches/stable-1.18...flyingcircusio:ctheune-anonymize-by-default

This isn?t ready to be merged at this point as it suffers at least from two things:

a) I?m not a C programmer by nature and might be making stupid mistakes. This is ?monkey see monkey do code?.

b) You likely do not want to have this on by default for everyone, so I?m expecting that this requires a config option (runtime or compile, with a slight preference to runtime from me).
   The new built-in variable remote_addr_anon should be available by default in any case, though.

Some background: we tried implementing this purely by using a mapping approach, but this doesn?t make it a proper default as everybody defining an access log has to mention the proper format or accidentally IPs will leak. This happens over and over and we?d prefer a ?privacy by design? approach very much.

There is a trac ticket (https://trac.nginx.org/nginx/ticket/868) that already discusses this, however the .1 is IMHO not a good approach to recommend as it suggests a real IP whereas our impression is that nulling the last byte in IPv4 and nulling the last 8 bytes in IPv6 is the proper approach and it?s directly visible that this is an anonymized IP.

Let me know what you think about this and - if you?d like to see this in the mainstream code - I?d appreciate if you give me the necessary hints to bring this code to the level of quality that is proper for nginx.

Kind regards,
Christian

--
Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
Flying Circus Internet Operations GmbH ? http://flyingcircus.io
Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: Message signed with OpenPGP
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20200916/e67bed5c/attachment-0001.bin>

From ct at flyingcircus.io  Wed Sep 16 19:20:24 2020
From: ct at flyingcircus.io (Christian Theune)
Date: Wed, 16 Sep 2020 21:20:24 +0200
Subject: (draft) Privacy by design - offer more convenient way to
 anonymize IPs in access log by default 
In-Reply-To: <117E0BAF-302D-41DC-97FE-C6346E018833@flyingcircus.io>
References: <117E0BAF-302D-41DC-97FE-C6346E018833@flyingcircus.io>
Message-ID: <97F30C1F-9577-408B-9EFF-CC10B066EDBF@flyingcircus.io>

Replying to myself for now ;)

One thing I can imagine being desirable would be to make the number of bits (v4/v6) that are masked configurable.

The current choice of 8 bits for IPv4 and 64-bit on IPv6 is based on our tradition that was considered a reasonable practice with respect to GDPR by courts in Germany. Unfortunately I can?t find a reliable source at the moment, but would take the time to dig one up if needed for the discussion.

Cheers,
Christian

> On 16. Sep 2020, at 17:28, Christian Theune <ct at flyingcircus.io> wrote:
> 
> Signed PGP part
> Hi!
> 
> first time posting here. I started working on a builtin-way for nginx to provide GDPR conformaing access logs reliably by default:
> 
> https://github.com/nginx/nginx/compare/branches/stable-1.18...flyingcircusio:ctheune-anonymize-by-default
> 
> This isn?t ready to be merged at this point as it suffers at least from two things:
> 
> a) I?m not a C programmer by nature and might be making stupid mistakes. This is ?monkey see monkey do code?.
> 
> b) You likely do not want to have this on by default for everyone, so I?m expecting that this requires a config option (runtime or compile, with a slight preference to runtime from me).
>   The new built-in variable remote_addr_anon should be available by default in any case, though.
> 
> Some background: we tried implementing this purely by using a mapping approach, but this doesn?t make it a proper default as everybody defining an access log has to mention the proper format or accidentally IPs will leak. This happens over and over and we?d prefer a ?privacy by design? approach very much.
> 
> There is a trac ticket (https://trac.nginx.org/nginx/ticket/868) that already discusses this, however the .1 is IMHO not a good approach to recommend as it suggests a real IP whereas our impression is that nulling the last byte in IPv4 and nulling the last 8 bytes in IPv6 is the proper approach and it?s directly visible that this is an anonymized IP.
> 
> Let me know what you think about this and - if you?d like to see this in the mainstream code - I?d appreciate if you give me the necessary hints to bring this code to the level of quality that is proper for nginx.
> 
> Kind regards,
> Christian
> 
> --
> Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
> Flying Circus Internet Operations GmbH ? http://flyingcircus.io
> Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
> HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick

--
Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
Flying Circus Internet Operations GmbH ? http://flyingcircus.io
Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: Message signed with OpenPGP
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20200916/1fb5fbd5/attachment.bin>

From mdounin at mdounin.ru  Wed Sep 16 21:41:27 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 16 Sep 2020 21:41:27 +0000
Subject: [nginx] SSL: workaround for incorrect SSL_write() errors in OpenSSL
 1.1.1.
Message-ID: <hg.61011bfcdb49.1600292487.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/61011bfcdb49
branches:  
changeset: 7706:61011bfcdb49
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Sep 16 18:26:22 2020 +0300
description:
SSL: workaround for incorrect SSL_write() errors in OpenSSL 1.1.1.

OpenSSL 1.1.1 fails to return SSL_ERROR_SYSCALL if an error happens
during SSL_write() after close_notify alert from the peer, and returns
SSL_ERROR_ZERO_RETURN instead.  Broken by this commit, which removes
the "i == 0" check around the SSL_RECEIVED_SHUTDOWN one:

https://git.openssl.org/?p=openssl.git;a=commitdiff;h=8051ab2

In particular, if a client closed the connection without reading
the response but with properly sent close_notify alert, this resulted in
unexpected "SSL_write() failed while ..." critical log message instead
of correct "SSL_write() failed (32: Broken pipe)" at the info level.

Since SSL_ERROR_ZERO_RETURN cannot be legitimately returned after
SSL_write(), the fix is to convert all SSL_ERROR_ZERO_RETURN errors
after SSL_write() to SSL_ERROR_SYSCALL.

diffstat:

 src/event/ngx_event_openssl.c |  12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diffs (22 lines):

diff -r 3781de64e747 -r 61011bfcdb49 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Wed Sep 09 19:26:27 2020 +0300
+++ b/src/event/ngx_event_openssl.c	Wed Sep 16 18:26:22 2020 +0300
@@ -2573,6 +2573,18 @@ ngx_ssl_write(ngx_connection_t *c, u_cha
 
     sslerr = SSL_get_error(c->ssl->connection, n);
 
+    if (sslerr == SSL_ERROR_ZERO_RETURN) {
+
+        /*
+         * OpenSSL 1.1.1 fails to return SSL_ERROR_SYSCALL if an error
+         * happens during SSL_write() after close_notify alert from the
+         * peer, and returns SSL_ERROR_ZERO_RETURN instead,
+         * https://git.openssl.org/?p=openssl.git;a=commitdiff;h=8051ab2
+         */
+
+        sslerr = SSL_ERROR_SYSCALL;
+    }
+
     err = (sslerr == SSL_ERROR_SYSCALL) ? ngx_errno : 0;
 
     ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_get_error: %d", sslerr);

From mdounin at mdounin.ru  Wed Sep 16 21:41:30 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 16 Sep 2020 21:41:30 +0000
Subject: [nginx] SSL: fixed event handling during shutdown.
Message-ID: <hg.adaec579a967.1600292490.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/adaec579a967
branches:  
changeset: 7707:adaec579a967
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Sep 16 18:26:23 2020 +0300
description:
SSL: fixed event handling during shutdown.

The c->read->ready and c->write->ready flags need to be cleared to ensure
that appropriate read or write events will be reported by kernel.  Without
this, SSL shutdown might wait till the timeout after blocking on writing
or reading even if there is a socket activity.

diffstat:

 src/event/ngx_event_openssl.c |  7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diffs (17 lines):

diff -r 61011bfcdb49 -r adaec579a967 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Wed Sep 16 18:26:22 2020 +0300
+++ b/src/event/ngx_event_openssl.c	Wed Sep 16 18:26:23 2020 +0300
@@ -2865,6 +2865,13 @@ ngx_ssl_shutdown(ngx_connection_t *c)
             c->read->handler = ngx_ssl_shutdown_handler;
             c->write->handler = ngx_ssl_shutdown_handler;
 
+            if (sslerr == SSL_ERROR_WANT_READ) {
+                c->read->ready = 0;
+
+            } else {
+                c->write->ready = 0;
+            }
+
             if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
                 return NGX_ERROR;
             }

From mdounin at mdounin.ru  Wed Sep 16 21:41:33 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 16 Sep 2020 21:41:33 +0000
Subject: [nginx] SSL: disabled shutdown after connection errors.
Message-ID: <hg.f3c87533f92c.1600292493.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/f3c87533f92c
branches:  
changeset: 7708:f3c87533f92c
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Sep 16 18:26:24 2020 +0300
description:
SSL: disabled shutdown after connection errors.

This fixes "SSL_shutdown() failed (SSL: ... bad write retry)" errors
as observed on the second SSL_shutdown() call after SSL shutdown fixes in
09fb2135a589 (1.19.2), notably when sending fails in ngx_http_test_expect(),
similarly to ticket #1194.

Note that there are some places where c->error is misused to prevent
further output, such as ngx_http_v2_finalize_connection() if there
are pending streams, or in filter finalization.  These places seem
to be extreme enough to don't care about missing shutdown though.
For example, filter finalization currently prevents keepalive from
being used.

diffstat:

 src/event/ngx_event_openssl.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r adaec579a967 -r f3c87533f92c src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Wed Sep 16 18:26:23 2020 +0300
+++ b/src/event/ngx_event_openssl.c	Wed Sep 16 18:26:24 2020 +0300
@@ -2805,7 +2805,7 @@ ngx_ssl_shutdown(ngx_connection_t *c)
         return NGX_OK;
     }
 
-    if (c->timedout) {
+    if (c->timedout || c->error) {
         mode = SSL_RECEIVED_SHUTDOWN|SSL_SENT_SHUTDOWN;
         SSL_set_quiet_shutdown(c->ssl->connection, 1);
 

From mdounin at mdounin.ru  Wed Sep 16 21:41:35 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 16 Sep 2020 21:41:35 +0000
Subject: [nginx] SSL: disabled shutdown when there are buffered data.
Message-ID: <hg.052ecc68d350.1600292495.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/052ecc68d350
branches:  
changeset: 7709:052ecc68d350
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Sep 16 18:26:25 2020 +0300
description:
SSL: disabled shutdown when there are buffered data.

This fixes "SSL_shutdown() failed (SSL: ... bad write retry)" errors
as observed on the second SSL_shutdown() call after SSL shutdown fixes in
09fb2135a589 (1.19.2), notably when HTTP/2 connections are closed due
to read timeouts while there are incomplete writes.

diffstat:

 src/event/ngx_event_openssl.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r f3c87533f92c -r 052ecc68d350 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Wed Sep 16 18:26:24 2020 +0300
+++ b/src/event/ngx_event_openssl.c	Wed Sep 16 18:26:25 2020 +0300
@@ -2805,7 +2805,7 @@ ngx_ssl_shutdown(ngx_connection_t *c)
         return NGX_OK;
     }
 
-    if (c->timedout || c->error) {
+    if (c->timedout || c->error || c->buffered) {
         mode = SSL_RECEIVED_SHUTDOWN|SSL_SENT_SHUTDOWN;
         SSL_set_quiet_shutdown(c->ssl->connection, 1);
 

From corentin.regal at gmail.com  Thu Sep 17 08:39:45 2020
From: corentin.regal at gmail.com (Corentin REGAL)
Date: Thu, 17 Sep 2020 10:39:45 +0200
Subject: Fix regex capture when using map directive and rewrite
Message-ID: <CALQyoc8yVwU0taJH3nPrNQsW7tNoV1Ks5w6jaejA9ntOscL_Pw@mail.gmail.com>

Hello,

I fixed a bug when using map directive and rewrite.
Captured variables in the map directive override captures of the rewrite.

*nginx.conf*
daemon off;
error_log  /var/log/nginx/error.log debug;
worker_processes  1;
events {
    worker_connections  1024;
}

http {
    include            mime.types;
    default_type       application/octet-stream;
    sendfile           on;
    keepalive_timeout  0;

    map $http_content_type $input_type {
        default  json;
        ~image/  image;
    }

    server {
        server_name localhost;
        listen 80;
        location / {
            rewrite "/(.+)" /$1_${input_type} redirect;
        }
    }
}


*before patch (/_image)*
curl -v -H'Content-Type: image/png' localhost:80/predict
...
< Location: http://localhost/_image
...

*after patch (/predict_image)*
curl -v -H'Content-Type: image/png' localhost:80/predict
...
< Location: http://localhost/predict_image
...

*patch*
# HG changeset patch
# User Corentin Regal <corentin.regal at gmail.com>
# Date 1600329166 0
#      Thu Sep 17 07:52:46 2020 +0000
# Node ID a065d5f865e90a7426d37b30a9faa72e0966756f
# Parent  052ecc68d35038b1b4adde12efe6249a92055f09
Fix ngx_http_map_find to not erase rewrite captures

diff -r 052ecc68d350 -r a065d5f865e9 src/http/ngx_http_variables.c
--- a/src/http/ngx_http_variables.c     Wed Sep 16 18:26:25 2020 +0300
+++ b/src/http/ngx_http_variables.c     Thu Sep 17 07:52:46 2020 +0000
@@ -2410,7 +2410,7 @@

         for (i = 0; i < map->nregex; i++) {

-            n = ngx_http_regex_exec(r, reg[i].regex, match);
+            n = ngx_regex_exec(reg[i].regex->regex, match, NULL, 0);

             if (n == NGX_OK) {
                 return reg[i].value;

It's my first time using Mercurial or a mailing list, I hope I did it right
:)

Regards,

Corentin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20200917/e2f5766e/attachment.htm>

From alexander.borisov at nginx.com  Thu Sep 17 10:11:41 2020
From: alexander.borisov at nginx.com (Alexander Borisov)
Date: Thu, 17 Sep 2020 10:11:41 +0000
Subject: [njs] Added forgotten 'break' statement missed in 27bb9caf186c.
Message-ID: <hg.0cfc928cc1e4.1600337501.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/0cfc928cc1e4
branches:  
changeset: 1524:0cfc928cc1e4
user:      Alexander Borisov <alexander.borisov at nginx.com>
date:      Thu Sep 17 13:10:51 2020 +0300
description:
Added forgotten 'break' statement missed in 27bb9caf186c.

Found by Coverity (CID 1466731).

diffstat:

 src/njs_buffer.c |  2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diffs (12 lines):

diff -r 27bb9caf186c -r 0cfc928cc1e4 src/njs_buffer.c
--- a/src/njs_buffer.c	Wed Sep 16 17:22:01 2020 +0300
+++ b/src/njs_buffer.c	Thu Sep 17 13:10:51 2020 +0300
@@ -2133,6 +2133,8 @@ njs_buffer_prototype_swap(njs_vm_t *vm, 
             *((uint32_t *) p) = njs_bswap_u32(*((uint32_t *) p));
         }
 
+        break;
+
     case 8:
     default:
         for (; p < end; p += 8) {

From mikhail.isachenkov at nginx.com  Mon Sep 21 11:29:06 2020
From: mikhail.isachenkov at nginx.com (Mikhail Isachenkov)
Date: Mon, 21 Sep 2020 14:29:06 +0300
Subject: [PATCH] Use BPF to distribute packet to different work thread.
In-Reply-To: <SN6PR11MB3517E97F77028B2A11C3B9EF87200@SN6PR11MB3517.namprd11.prod.outlook.com>
References: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200910101132.GA38727@vl.krasnogorsk.ru>
 <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200913233936.GV18881@mdounin.ru>
 <SN6PR11MB3517F8A7E457669E49D3483C87230@SN6PR11MB3517.namprd11.prod.outlook.com>
 <SN6PR11MB3517E97F77028B2A11C3B9EF87200@SN6PR11MB3517.namprd11.prod.outlook.com>
Message-ID: <968ffc6d-aabb-1da1-338c-ec22ae745f48@nginx.com>

Hi Liu Qiao,

We've testing early version patch with same BPF code (on AWS cloud, 
without ADQ-capable cards) on relatively small payloads and found no 
significant difference. We'd like to retest it with large payload size; 
could you please elaborate a bit more how you did perform the test?
I mean 'nginx -T' output, number of CPU cores and CPU model, test 
scripts, 1-megabyte file and any system tuning parameters. One of 
caveats was that wrk may produce significant client load and most 
effective way to distribute client load between CPU cores is running wrk 
in one thread via taskset.

Another big caveat that I've found during my test is the strange 
behavior of this BPF code: when client and server runs on the same 
server, all requests was served by one nginx worker process. Did you try 
to run wrk and nginx locally in your test?

Thanks in advance!

15.09.2020 05:08, Liu, Qiao ?????:
> Below is 5 times test result compare, 112 threads, 10000 connections, 1M object http request. Seems P99 have great improvement, and Max is also reduced
> 
> 
> 
>                                     AVG          Stdev            Max        P99
>                    test 1      1.32s        447.09ms     5.48s      2.82s
> BPF           test 2      1.39s        513.8ms       9.42s      3.1s
>                    test 3      1.4s          341.38ms     5.63s      2.55s
>                    test 4      1.41s        407.45ms     6.96s      2.77s
>                    test 5      1.29s        644.81ms     9.45s      3.74s
>                   Average  1.362s      470.906ms   7.388s    2.996s
> 
> NonBPF   test 1      1.48s         916.88ms     9.44s       5.08s
>                   test 2      1.43s         658.48ms     9.54s       3.92s
>                   test 3      1.41s         650.38ms     8.63s       3.59s
>                   test 4      1.29s         1010ms        10s           5.21s
>                   test 5      1.31s         875.01ms     9.53s       4.39s
>               Average     1.384s        822.15ms    9.428s    4.438s
> 
> 
> Thanks
> LQ
> -----Original Message-----
> From: nginx-devel <nginx-devel-bounces at nginx.org> On Behalf Of Liu, Qiao
> Sent: Monday, September 14, 2020 9:18 AM
> To: nginx-devel at nginx.org
> Subject: RE: [PATCH] Use BPF to distribute packet to different work thread.
> 
> Hi, Maxim Dounin:
> Thanks for your reply, this server is random selected, we just do BPF and no-BPF test, I think the latency based on server configuration, not related with BPF patch, also the NIC of the server is Mellanox, not ADQ capable hardware , we will do more test Thanks LQ
> 
> -----Original Message-----
> From: nginx-devel <nginx-devel-bounces at nginx.org> On Behalf Of Maxim Dounin
> Sent: Monday, September 14, 2020 7:40 AM
> To: nginx-devel at nginx.org
> Subject: Re: [PATCH] Use BPF to distribute packet to different work thread.
> 
> Hello!
> 
> On Fri, Sep 11, 2020 at 05:41:47AM +0000, Liu, Qiao wrote:
> 
>> Hi, Vladimir Homutov:
>> The below is our WRK test result output with BPF enable
>>
>>    112 threads and 10000 connections
>>    Thread Stats   Avg      Stdev     Max   +/- Stdev
>>      Latency   608.23ms  820.71ms  10.00s    87.48%
>>      Connect    16.52ms   54.53ms   1.99s    94.73%
>>      Delay     153.13ms  182.17ms   2.00s    90.74%
>>      Req/Sec   244.79    142.32     1.99k    68.40%
>>    Latency Distribution
>>    50.00%  293.50ms
>>    75.00%  778.33ms
>>    90.00%    1.61s
>>    99.00%    3.71s
>>    99.90%    7.03s
>>    99.99%    8.94s
>>    Connect Distribution
>>    50.00%    1.93ms
>>    75.00%    2.85ms
>>    90.00%   55.76ms
>>    99.00%  229.19ms
>>    99.90%  656.79ms
>>    99.99%    1.43s
>>    Delay Distribution
>>    50.00%  110.96ms
>>    75.00%  193.67ms
>>    90.00%  321.77ms
>>    99.00%  959.27ms
>>    99.90%    1.57s
>>    99.99%    1.91s
>> Compared with no BPF but enable reuseport as below
>>
>> 112 threads and 10000 connections
>>    Thread Stats   Avg      Stdev     Max   +/- Stdev
>>      Latency   680.50ms  943.69ms  10.00s    87.18%
>>      Connect    58.44ms  238.08ms   2.00s    94.58%
>>      Delay     158.84ms  256.28ms   2.00s    90.92%
>>      Req/Sec   244.51    151.00     1.41k    69.67%
>>    Latency Distribution
>>    50.00%  317.61ms
>>    75.00%  913.52ms
>>    90.00%    1.90s
>>    99.00%    4.30s
>>    99.90%    6.52s
>>    99.99%    8.80s
>>    Connect Distribution
>>    50.00%    1.88ms
>>    75.00%    2.21ms
>>    90.00%   55.94ms
>>    99.00%    1.45s
>>    99.90%    1.95s
>>    99.99%    2.00s
>>    Delay Distribution
>>    50.00%   73.01ms
>>    75.00%  190.40ms
>>    90.00%  387.01ms
>>    99.00%    1.34s
>>    99.90%    1.86s
>>    99.99%    1.99s
>>
>>
>>  From the above results, there shows almost 20% percent latency
>> reduction. P99 latency of BPF is 3.71s , but without BPF is 4.3s.
> 
> Thank you for the results.
> 
> Given that latency stdev is way higher than the average latency, I don't think the "20% percent latency reduction" observed is statistically significant.  Please try running several tests and use ministat(1) to check the results.
> 
> Also, the latency values look very high, and request rate very low.  What's on the server side?
> 
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
> 

-- 
Best regards,
Mikhail Isachenkov
NGINX Professional Services

From xeioex at nginx.com  Mon Sep 21 12:11:19 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 21 Sep 2020 12:11:19 +0000
Subject: [njs] Added support for detached-buffers.
Message-ID: <hg.f7f7994b69b5.1600690279.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/f7f7994b69b5
branches:  
changeset: 1525:f7f7994b69b5
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Fri Sep 18 18:00:32 2020 +0000
description:
Added support for detached-buffers.

diffstat:

 auto/cc                  |    4 +
 auto/help                |    6 +-
 auto/options             |    4 +-
 src/njs_array_buffer.c   |    4 +
 src/njs_buffer.c         |   90 ++++++++++++++++---
 src/njs_builtin.c        |   69 +++++++++++++++
 src/njs_object_hash.h    |    8 +
 src/njs_typed_array.c    |  209 ++++++++++++++++++++++++++++++++++++++--------
 src/njs_value.c          |   10 ++
 src/njs_value.h          |    4 +
 src/njs_vm.h             |    5 +-
 src/test/njs_unit_test.c |    4 +
 12 files changed, 360 insertions(+), 57 deletions(-)

diffs (905 lines):

diff -r 0cfc928cc1e4 -r f7f7994b69b5 auto/cc
--- a/auto/cc	Thu Sep 17 13:10:51 2020 +0300
+++ b/auto/cc	Fri Sep 18 18:00:32 2020 +0000
@@ -177,5 +177,9 @@ if [ "$NJS_DEBUG_MEMORY" = "YES" ]; then
         njs_define=NJS_DEBUG_MEMORY . auto/define
 fi
 
+if [ "$NJS_TEST262" = "YES" ]; then
+        njs_define=NJS_TEST262 . auto/define
+fi
+
 # Stop on error exit status again.
 set -e
diff -r 0cfc928cc1e4 -r f7f7994b69b5 auto/help
--- a/auto/help	Thu Sep 17 13:10:51 2020 +0300
+++ b/auto/help	Fri Sep 18 18:00:32 2020 +0000
@@ -14,10 +14,12 @@ default: "$NJS_CC_OPT"
 default: "$NJS_LD_OPT"
   --ar=FILE                 set static linking program, default: "$AR"
 
+  --address-sanitizer=YES   enables build with address sanitizer, \
+default: "$NJS_ADDRESS_SANITIZER"
   --debug=YES               enables additional runtime checks, \
 default: "$NJS_DEBUG"
   --debug-memory=YES        enables memory alloc debug, \
 default: "$NJS_DEBUG_MEMORY"
-  --address-sanitizer=YES   enables build with address sanitizer, \
-default: "$NJS_ADDRESS_SANITIZER"
+  --test262=YES             enables test262 extentions, \
+default: "$NJS_TEST262"
 END
diff -r 0cfc928cc1e4 -r f7f7994b69b5 auto/options
--- a/auto/options	Thu Sep 17 13:10:51 2020 +0300
+++ b/auto/options	Fri Sep 18 18:00:32 2020 +0000
@@ -9,6 +9,7 @@ NJS_LD_OPT=${NJS_CC_OPT:--O}
 NJS_DEBUG=NO
 NJS_DEBUG_MEMORY=NO
 NJS_ADDRESS_SANITIZER=NO
+NJS_TEST262=YES
 
 NJS_CONFIGURE_OPTIONS=
 
@@ -25,9 +26,10 @@ do
         --ld-opt=*)                      NJS_LD_OPT="$value"                 ;;
         --ar=*)                          AR="$value"                         ;;
 
+        --address-sanitizer=*)           NJS_ADDRESS_SANITIZER="$value"      ;;
         --debug=*)                       NJS_DEBUG="$value"                  ;;
         --debug-memory=*)                NJS_DEBUG_MEMORY="$value"           ;;
-        --address-sanitizer=*)           NJS_ADDRESS_SANITIZER="$value"      ;;
+        --test262=*)                     NJS_TEST262="$value"                ;;
 
         --help)
             . auto/help
diff -r 0cfc928cc1e4 -r f7f7994b69b5 src/njs_array_buffer.c
--- a/src/njs_array_buffer.c	Thu Sep 17 13:10:51 2020 +0300
+++ b/src/njs_array_buffer.c	Fri Sep 18 18:00:32 2020 +0000
@@ -204,6 +204,10 @@ njs_array_buffer_prototype_byte_length(n
     }
 
     array = njs_array_buffer(value);
+    if (njs_slow_path(njs_is_detached_buffer(array))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
 
     njs_set_number(&vm->retval, array->size);
 
diff -r 0cfc928cc1e4 -r f7f7994b69b5 src/njs_buffer.c
--- a/src/njs_buffer.c	Thu Sep 17 13:10:51 2020 +0300
+++ b/src/njs_buffer.c	Fri Sep 18 18:00:32 2020 +0000
@@ -441,6 +441,11 @@ njs_buffer_from_typed_array(njs_vm_t *vm
     njs_typed_array_t  *buffer, *array;
 
     array = njs_typed_array(value);
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     length = njs_typed_array_length(array);
 
     buffer = njs_buffer_alloc_array(vm, length, 0);
@@ -597,9 +602,9 @@ njs_buffer_array_range(njs_vm_t *vm, njs
     const njs_value_t *start, const njs_value_t *end, const char *name,
     uint8_t **out_start, uint8_t **out_end)
 {
-    uint8_t    *u8;
-    uint64_t   num_start, num_end;
-    njs_int_t  ret;
+    uint64_t            num_start, num_end;
+    njs_int_t           ret;
+    njs_array_buffer_t  *buffer;
 
     num_start = 0;
 
@@ -633,9 +638,14 @@ njs_buffer_array_range(njs_vm_t *vm, njs
         num_end = num_start;
     }
 
-    u8 = njs_typed_array_buffer(array)->u.u8;
-    *out_start = &u8[array->offset + num_start];
-    *out_end = &u8[array->offset + num_end];
+    buffer = njs_typed_array_buffer(array);
+    if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
+    *out_start = &buffer->u.u8[array->offset + num_start];
+    *out_end = &buffer->u.u8[array->offset + num_end];
 
     return NJS_OK;
 }
@@ -746,6 +756,10 @@ njs_buffer_concat(njs_vm_t *vm, njs_valu
             }
 
             arr = njs_typed_array(value);
+            if (njs_slow_path(njs_is_detached_buffer(arr->buffer))) {
+                njs_type_error(vm, "detached buffer");
+                return NJS_ERROR;
+            }
 
             if (njs_slow_path((SIZE_MAX - len) < arr->byte_length)) {
                 njs_type_error(vm, "Invalid length");
@@ -770,6 +784,10 @@ njs_buffer_concat(njs_vm_t *vm, njs_valu
             }
 
             arr = njs_typed_array(&retval);
+            if (njs_slow_path(njs_is_detached_buffer(arr->buffer))) {
+                njs_type_error(vm, "detached buffer");
+                return NJS_ERROR;
+            }
 
             if (njs_slow_path((SIZE_MAX - len) < arr->byte_length)) {
                 njs_type_error(vm, "Invalid length");
@@ -947,6 +965,11 @@ njs_buffer_prototype_read_int(njs_vm_t *
 #endif
 
     buffer = array->buffer;
+    if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     u8 = &buffer->u.u8[index + array->offset];
 
     switch (size) {
@@ -1127,6 +1150,11 @@ njs_buffer_prototype_read_float(njs_vm_t
 #endif
 
     buffer = array->buffer;
+    if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     u8 = &buffer->u.u8[index + array->offset];
 
     switch (size) {
@@ -1657,6 +1685,11 @@ njs_buffer_fill(njs_vm_t *vm, njs_typed_
             return ret;
         }
 
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
         memset(start, njs_number_to_uint32(num) & 0xff, end - offset);
     }
 
@@ -1783,6 +1816,11 @@ njs_buffer_prototype_to_string(njs_vm_t 
         end = njs_min(end, array->byte_length);
     }
 
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     str.start = &njs_typed_array_buffer(array)->u.u8[array->offset + start];
     str.length = end - start;
 
@@ -1890,12 +1928,14 @@ static njs_int_t
 njs_buffer_prototype_index_of(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t last)
 {
-    uint8_t                      *u8, byte;
+    uint8_t                      byte;
     int64_t                      from, to, increment, length, offset, index, i;
     njs_int_t                    ret;
     njs_str_t                    str;
     njs_value_t                  *this, *value, *value_from, *enc, dst;
+    const uint8_t                *u8;
     njs_typed_array_t            *array, *src;
+    njs_array_buffer_t           *buffer;
     const njs_buffer_encoding_t  *encoding;
 
     encoding = njs_buffer_utf8_encoding();
@@ -1977,7 +2017,13 @@ njs_buffer_prototype_index_of(njs_vm_t *
         }
     }
 
-    u8 = &njs_typed_array_buffer(array)->u.u8[0];
+    buffer = njs_typed_array_buffer(array);
+    if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
+    u8 = &buffer->u.u8[0];
     offset = array->offset;
 
     switch (value->type) {
@@ -1997,6 +2043,11 @@ njs_buffer_prototype_index_of(njs_vm_t *
                 goto fail;
             }
 
+            if (njs_slow_path(njs_is_detached_buffer(src->buffer))) {
+                njs_type_error(vm, "detached buffer");
+                return NJS_ERROR;
+            }
+
             str.start = &src->buffer->u.u8[src->offset];
             str.length = src->byte_length;
         }
@@ -2152,13 +2203,14 @@ static njs_int_t
 njs_buffer_prototype_to_json(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t unused)
 {
-    u_char             *p, *end;
-    njs_int_t          ret;
-    njs_value_t        *value;
-    njs_value_t        object, array;
-    njs_array_t        *arr;
-    njs_object_t       *obj;
-    njs_typed_array_t  *ta;
+    u_char              *p, *end;
+    njs_int_t           ret;
+    njs_value_t         *value;
+    njs_value_t         object, array;
+    njs_array_t         *arr;
+    njs_object_t        *obj;
+    njs_typed_array_t   *ta;
+    njs_array_buffer_t  *buffer;
 
     static const njs_value_t  string_buffer = njs_string("Buffer");
 
@@ -2185,7 +2237,13 @@ njs_buffer_prototype_to_json(njs_vm_t *v
         return NJS_ERROR;
     }
 
-    p = &njs_typed_array_buffer(ta)->u.u8[ta->offset];
+    buffer = njs_typed_array_buffer(ta);
+    if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
+    p = &buffer->u.u8[ta->offset];
     end = p + ta->byte_length;
     value = arr->start;
 
diff -r 0cfc928cc1e4 -r f7f7994b69b5 src/njs_builtin.c
--- a/src/njs_builtin.c	Thu Sep 17 13:10:51 2020 +0300
+++ b/src/njs_builtin.c	Fri Sep 18 18:00:32 2020 +0000
@@ -35,6 +35,9 @@ static njs_int_t njs_env_hash_init(njs_v
 static const njs_object_init_t  njs_global_this_init;
 static const njs_object_init_t  njs_njs_object_init;
 static const njs_object_init_t  njs_process_object_init;
+#ifdef NJS_TEST262
+static const njs_object_init_t  njs_262_object_init;
+#endif
 
 
 static const njs_object_init_t  *njs_object_init[] = {
@@ -43,6 +46,9 @@ static const njs_object_init_t  *njs_obj
     &njs_process_object_init,
     &njs_math_object_init,
     &njs_json_object_init,
+#ifdef NJS_TEST262
+    &njs_262_object_init,
+#endif
     NULL
 };
 
@@ -1259,6 +1265,18 @@ static const njs_object_prop_t  njs_glob
         .configurable = 1,
     },
 
+#ifdef NJS_TEST262
+    {
+        .type = NJS_PROPERTY_HANDLER,
+        .name = njs_string("$262"),
+        .value = njs_prop_handler2(njs_top_level_object, NJS_OBJECT_262,
+                                  NJS_262_HASH),
+        .writable = 1,
+        .enumerable = 1,
+        .configurable = 1,
+    },
+#endif
+
     /* Global constructors. */
 
     {
@@ -1831,3 +1849,54 @@ static const njs_object_init_t  njs_proc
     njs_process_object_properties,
     njs_nitems(njs_process_object_properties),
 };
+
+
+#if (NJS_TEST262)
+
+static njs_int_t
+njs_262_detach_array_buffer(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+    njs_index_t unused)
+{
+    njs_value_t         *value;
+    njs_array_buffer_t  *buffer;
+
+    value = njs_arg(args, nargs, 1);
+    if (njs_slow_path(!njs_is_array_buffer(value))) {
+        njs_type_error(vm, "\"this\" is not an ArrayBuffer");
+        return NJS_ERROR;
+    }
+
+    buffer = njs_array_buffer(value);
+    buffer->u.data = NULL;
+    buffer->size = 0;
+
+    njs_set_null(&vm->retval);
+
+    return NJS_OK;
+}
+
+static const njs_object_prop_t  njs_262_object_properties[] =
+{
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_wellknown_symbol(NJS_SYMBOL_TO_STRING_TAG),
+        .value = njs_string("$262"),
+        .configurable = 1,
+    },
+
+    {
+        .type = NJS_PROPERTY,
+        .name = njs_long_string("detachArrayBuffer"),
+        .value = njs_native_function(njs_262_detach_array_buffer, 2),
+        .writable = 1,
+        .configurable = 1,
+    },
+};
+
+
+static const njs_object_init_t  njs_262_object_init = {
+    njs_262_object_properties,
+    njs_nitems(njs_262_object_properties),
+};
+
+#endif
diff -r 0cfc928cc1e4 -r f7f7994b69b5 src/njs_object_hash.h
--- a/src/njs_object_hash.h	Thu Sep 17 13:10:51 2020 +0300
+++ b/src/njs_object_hash.h	Fri Sep 18 18:00:32 2020 +0000
@@ -301,6 +301,14 @@
         'n'), 'j'), 's')
 
 
+#define NJS_262_HASH                                                          \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(                                                         \
+    njs_djb_hash_add(NJS_DJB_HASH_INIT,                                       \
+        '$'), '2'), '6'), '2')
+
+
 #define NJS_NUMBER_HASH                                                       \
     njs_djb_hash_add(                                                         \
     njs_djb_hash_add(                                                         \
diff -r 0cfc928cc1e4 -r f7f7994b69b5 src/njs_typed_array.c
--- a/src/njs_typed_array.c	Thu Sep 17 13:10:51 2020 +0300
+++ b/src/njs_typed_array.c	Fri Sep 18 18:00:32 2020 +0000
@@ -64,7 +64,19 @@ njs_typed_array_alloc(njs_vm_t *vm, njs_
             return NULL;
         }
 
-        if (!njs_is_undefined(njs_arg(args, nargs, 2))) {
+        if (njs_is_defined(njs_arg(args, nargs, 2))) {
+            ret = njs_value_to_index(vm, njs_argument(args, 2), &size);
+            if (njs_slow_path(ret != NJS_OK)) {
+                return NULL;
+            }
+        }
+
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NULL;
+        }
+
+        if (njs_is_defined(njs_arg(args, nargs, 2))) {
             ret = njs_value_to_index(vm, njs_argument(args, 2), &size);
             if (njs_slow_path(ret != NJS_OK)) {
                 return NULL;
@@ -95,6 +107,11 @@ njs_typed_array_alloc(njs_vm_t *vm, njs_
 
     } else if (njs_is_typed_array(value)) {
         src_tarray = njs_typed_array(value);
+        if (njs_slow_path(njs_is_detached_buffer(src_tarray->buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NULL;
+        }
+
         size = (uint64_t) njs_typed_array_length(src_tarray) * element_size;
 
     } else if (njs_is_object(value)) {
@@ -224,9 +241,10 @@ static njs_int_t
 njs_typed_array_create(njs_vm_t *vm, njs_value_t *constructor,
     njs_value_t *args, njs_uint_t nargs, njs_value_t *retval)
 {
-    njs_int_t     ret;
-    njs_value_t   this;
-    njs_object_t  *object;
+    njs_int_t          ret;
+    njs_value_t        this;
+    njs_object_t       *object;
+    njs_typed_array_t  *array;
 
     object = njs_function_new_object(vm, constructor);
     if (njs_slow_path(object == NULL)) {
@@ -247,8 +265,14 @@ njs_typed_array_create(njs_vm_t *vm, njs
         return NJS_ERROR;
     }
 
+    array = njs_typed_array(retval);
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     if (njs_slow_path(nargs == 1 && njs_is_number(&args[0])
-                      && njs_typed_array_length(njs_typed_array(retval))
+                      && njs_typed_array_length(array)
                          < njs_number(&args[0])))
     {
         njs_type_error(vm, "Derived TypedArray constructor "
@@ -425,6 +449,10 @@ njs_typed_array_writable(njs_vm_t *vm, n
     njs_array_buffer_t  *buffer;
 
     buffer = array->buffer;
+    if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NULL;
+    }
 
     ret = njs_array_buffer_writable(vm, buffer);
     if (njs_slow_path(ret != NJS_OK)) {
@@ -485,6 +513,7 @@ static njs_int_t
 njs_typed_array_prototype_length(njs_vm_t *vm, njs_value_t *args,
     njs_uint_t nargs, njs_index_t unused)
 {
+    uint32_t           length;
     njs_value_t        *this;
     njs_typed_array_t  *array;
 
@@ -496,8 +525,13 @@ njs_typed_array_prototype_length(njs_vm_
     }
 
     array = njs_typed_array(this);
-
-    njs_set_number(&vm->retval, njs_typed_array_length(array));
+    length = njs_typed_array_length(array);
+
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        length = 0;
+    }
+
+    njs_set_number(&vm->retval, length);
 
     return NJS_OK;
 }
@@ -529,6 +563,7 @@ static njs_int_t
 njs_typed_array_prototype_byte_length(njs_vm_t *vm, njs_value_t *args,
     njs_uint_t nargs, njs_index_t unused)
 {
+    size_t             byte_length;
     njs_value_t        *this;
     njs_typed_array_t  *array;
 
@@ -540,8 +575,18 @@ njs_typed_array_prototype_byte_length(nj
     }
 
     array = njs_typed_array(this);
-
-    njs_set_number(&vm->retval, array->byte_length);
+    byte_length = array->byte_length;
+
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        if (njs_is_data_view(this)) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
+        byte_length = 0;
+    }
+
+    njs_set_number(&vm->retval, byte_length);
 
     return NJS_OK;
 }
@@ -551,6 +596,7 @@ static njs_int_t
 njs_typed_array_prototype_byte_offset(njs_vm_t *vm, njs_value_t *args,
     njs_uint_t nargs, njs_index_t unused)
 {
+    size_t             byte_offset;
     njs_value_t        *this;
     njs_typed_array_t  *array;
 
@@ -562,8 +608,18 @@ njs_typed_array_prototype_byte_offset(nj
     }
 
     array = njs_typed_array(this);
-
-    njs_set_number(&vm->retval, njs_typed_array_offset(array));
+    byte_offset = njs_typed_array_offset(array);
+
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        if (njs_is_data_view(this)) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
+        byte_offset = 0;
+    }
+
+    njs_set_number(&vm->retval, byte_offset);
 
     return NJS_OK;
 }
@@ -693,6 +749,11 @@ njs_typed_array_prototype_set(njs_vm_t *
 
     if (njs_is_typed_array(src)) {
         src_tarray = njs_typed_array(src);
+        if (njs_slow_path(njs_is_detached_buffer(src_tarray->buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
         src_length = njs_typed_array_length(src_tarray);
 
         if (njs_slow_path((src_length > length)
@@ -767,6 +828,11 @@ njs_typed_array_prototype_set(njs_vm_t *
                 }
             }
 
+            if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+                njs_type_error(vm, "detached buffer");
+                return NJS_ERROR;
+            }
+
             njs_typed_array_prop_set(vm, self, offset + i, num);
         }
     }
@@ -803,6 +869,11 @@ njs_typed_array_prototype_fill(njs_vm_t 
     }
 
     array = njs_typed_array(this);
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     length = njs_typed_array_length(array);
 
     setval = njs_lvalue_arg(&lvalue, args, nargs, 1);
@@ -929,6 +1000,10 @@ njs_typed_array_prototype_slice(njs_vm_t
     array = njs_typed_array(this);
     length = njs_typed_array_length(array);
     buffer = njs_typed_array_buffer(array);
+    if (njs_slow_path(copy && njs_is_detached_buffer(buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
 
     ret = njs_value_to_integer(vm, njs_arg(args, nargs, 1), &start);
     if (njs_slow_path(ret != NJS_OK)) {
@@ -966,13 +1041,16 @@ njs_typed_array_prototype_slice(njs_vm_t
             return ret;
         }
 
-        new_array = njs_typed_array(&vm->retval);
-        if (njs_typed_array_length(new_array) < count) {
-            njs_type_error(vm, "Derived TypedArray constructor is "
-                           "too small array");
+        if (count == 0) {
+            return NJS_OK;
+        }
+
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
             return NJS_ERROR;
         }
 
+        new_array = njs_typed_array(&vm->retval);
         new_buffer = njs_typed_array_buffer(new_array);
         element_size = njs_typed_array_element_size(array->type);
 
@@ -1022,6 +1100,11 @@ njs_typed_array_prototype_copy_within(nj
     }
 
     array = njs_typed_array(this);
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     length = njs_typed_array_length(array);
 
     value = njs_arg(args, nargs, 1);
@@ -1044,10 +1127,9 @@ njs_typed_array_prototype_copy_within(nj
 
     value = njs_arg(args, nargs, 3);
 
-    if (njs_is_undefined(value)) {
-        final = length;
-
-    } else {
+    final = length;
+
+    if (njs_is_defined(value)) {
         ret = njs_value_to_integer(vm, value, &final);
         if (njs_slow_path(ret != NJS_OK)) {
             return NJS_ERROR;
@@ -1085,14 +1167,15 @@ static njs_int_t
 njs_typed_array_prototype_iterator(njs_vm_t *vm, njs_value_t *args,
     njs_uint_t nargs, njs_index_t type)
 {
-    double             val;
-    int64_t            i, length;
-    njs_int_t          ret;
-    njs_arr_t          results;
-    njs_value_t        *this, *this_arg, *r;
-    njs_value_t        arguments[4], retval;
-    njs_function_t     *function;
-    njs_typed_array_t  *array, *dst;
+    double              val;
+    int64_t             i, length;
+    njs_int_t           ret;
+    njs_arr_t           results;
+    njs_value_t         *this, *this_arg, *r;
+    njs_value_t         arguments[4], retval;
+    njs_function_t      *function;
+    njs_typed_array_t   *array, *dst;
+    njs_array_buffer_t  *buffer;
 
     this = njs_argument(args, 0);
     if (njs_slow_path(!njs_is_typed_array(this))) {
@@ -1112,6 +1195,7 @@ njs_typed_array_prototype_iterator(njs_v
     function = njs_function(njs_argument(args, 1));
     this_arg = njs_arg(args, nargs, 2);
 
+    buffer = array->buffer;
     results.separate = 0;
     results.pointer = 0;
 
@@ -1136,6 +1220,11 @@ njs_typed_array_prototype_iterator(njs_v
     }
 
     for (i = 0; i < length; i++) {
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
         val = njs_typed_array_prop(array, i);
 
         arguments[0] = *this_arg;
@@ -1349,6 +1438,11 @@ njs_typed_array_prototype_index_of(njs_v
         }
     }
 
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     v = njs_number(njs_argument(args, 1));
 
     i64 = v;
@@ -1491,12 +1585,13 @@ static njs_int_t
 njs_typed_array_prototype_reduce(njs_vm_t *vm, njs_value_t *args,
     njs_uint_t nargs, njs_index_t right)
 {
-    int64_t            i, from, to, increment, length;
-    njs_int_t          ret;
-    njs_value_t        *this, accumulator;
-    njs_value_t        arguments[5];
-    njs_function_t     *function;
-    njs_typed_array_t  *array;
+    int64_t             i, from, to, increment, length;
+    njs_int_t           ret;
+    njs_value_t         *this, accumulator;
+    njs_value_t         arguments[5];
+    njs_function_t      *function;
+    njs_typed_array_t   *array;
+    njs_array_buffer_t  *buffer;
 
     this = njs_argument(args, 0);
     if (njs_slow_path(!njs_is_typed_array(this))) {
@@ -1530,15 +1625,27 @@ njs_typed_array_prototype_reduce(njs_vm_
         increment = 1;
     }
 
+    buffer = array->buffer;
+
     if (nargs > 2) {
         accumulator = *njs_argument(args, 2);
 
     } else {
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
         njs_set_number(&accumulator, njs_typed_array_prop(array, from));
         from += increment;
     }
 
     for (i = from; i != to; i += increment) {
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
         njs_set_undefined(&arguments[0]);
         arguments[1] = accumulator;
         njs_set_number(&arguments[2], njs_typed_array_prop(array, i));
@@ -1785,6 +1892,7 @@ njs_typed_array_get_f64(const void *a)
 
 typedef struct {
     njs_vm_t               *vm;
+    njs_array_buffer_t     *buffer;
     njs_function_t         *function;
     njs_bool_t             exception;
     double                 (*get)(const void *v);
@@ -1821,6 +1929,11 @@ njs_typed_array_generic_compare(const vo
         goto exception;
     }
 
+    if (njs_slow_path(njs_is_detached_buffer(ctx->buffer))) {
+        njs_type_error(ctx->vm, "detached buffer");
+        goto exception;
+    }
+
     if (njs_slow_path(isnan(num))) {
         return 0;
     }
@@ -1858,7 +1971,14 @@ njs_typed_array_prototype_sort(njs_vm_t 
         return NJS_ERROR;
     }
 
+    array = njs_typed_array(this);
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     ctx.vm = vm;
+    ctx.buffer = array->buffer;
     ctx.exception = 0;
 
     comparefn = njs_arg(args, nargs, 1);
@@ -1875,8 +1995,6 @@ njs_typed_array_prototype_sort(njs_vm_t 
         ctx.function = NULL;
     }
 
-    array = njs_typed_array(this);
-
     switch (array->type) {
     case NJS_OBJ_TYPE_UINT8_ARRAY:
     case NJS_OBJ_TYPE_UINT8_CLAMPED_ARRAY:
@@ -2022,6 +2140,11 @@ njs_typed_array_prototype_join(njs_vm_t 
     }
 
     array = njs_typed_array(this);
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     arr_length = njs_typed_array_length(array);
 
     separator = njs_arg(args, nargs, 1);
@@ -2406,13 +2529,17 @@ njs_data_view_constructor(njs_vm_t *vm, 
     size = 0;
     offset = 0;
 
-    buffer = njs_array_buffer(njs_argument(args, 1));
-
     ret = njs_value_to_index(vm, njs_arg(args, nargs, 2), &offset);
     if (njs_slow_path(ret != NJS_OK)) {
         return NJS_ERROR;
     }
 
+    buffer = njs_array_buffer(njs_argument(args, 1));
+    if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     if (!njs_is_undefined(njs_arg(args, nargs, 3))) {
         ret = njs_value_to_index(vm, njs_argument(args, 3), &size);
         if (njs_slow_path(ret != NJS_OK)) {
@@ -2526,6 +2653,10 @@ njs_data_view_prototype_get(njs_vm_t *vm
 #endif
 
     view = njs_data_view(this);
+    if (njs_slow_path(njs_is_detached_buffer(view->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
 
     if (njs_typed_array_element_size(type) + index > view->byte_length) {
         njs_range_error(vm, "index %uL is outside the bound of the buffer",
@@ -2647,6 +2778,10 @@ njs_data_view_prototype_set(njs_vm_t *vm
 #endif
 
     view = njs_data_view(this);
+    if (njs_slow_path(njs_is_detached_buffer(view->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
 
     if (njs_typed_array_element_size(type) + index > view->byte_length) {
         njs_range_error(vm, "index %uL is outside the bound of the buffer",
diff -r 0cfc928cc1e4 -r f7f7994b69b5 src/njs_value.c
--- a/src/njs_value.c	Thu Sep 17 13:10:51 2020 +0300
+++ b/src/njs_value.c	Fri Sep 18 18:00:32 2020 +0000
@@ -807,6 +807,11 @@ njs_typed_array_property_query(njs_vm_t 
 {
     njs_object_prop_t  *prop;
 
+    if (njs_slow_path(njs_is_detached_buffer(array->buffer))) {
+        njs_type_error(vm, "detached buffer");
+        return NJS_ERROR;
+    }
+
     if (index >= njs_typed_array_length(array)) {
         return NJS_DECLINED;
     }
@@ -959,6 +964,11 @@ njs_value_property(njs_vm_t *vm, njs_val
         if (njs_is_typed_array(value)) {
             tarray = njs_typed_array(value);
 
+            if (njs_slow_path(njs_is_detached_buffer(tarray->buffer))) {
+                njs_type_error(vm, "detached buffer");
+                return NJS_ERROR;
+            }
+
             if (njs_slow_path(index >= njs_typed_array_length(tarray))) {
                 goto slow_path;
             }
diff -r 0cfc928cc1e4 -r f7f7994b69b5 src/njs_value.h
--- a/src/njs_value.h	Thu Sep 17 13:10:51 2020 +0300
+++ b/src/njs_value.h	Fri Sep 18 18:00:32 2020 +0000
@@ -659,6 +659,10 @@ typedef struct {
     ((value)->type == NJS_TYPED_ARRAY)
 
 
+#define njs_is_detached_buffer(buffer)                                        \
+    ((buffer)->u.data == NULL)
+
+
 #define njs_is_data_view(value)                                               \
     ((value)->type == NJS_DATA_VIEW)
 
diff -r 0cfc928cc1e4 -r f7f7994b69b5 src/njs_vm.h
--- a/src/njs_vm.h	Thu Sep 17 13:10:51 2020 +0300
+++ b/src/njs_vm.h	Fri Sep 18 18:00:32 2020 +0000
@@ -140,7 +140,10 @@ enum njs_object_e {
     NJS_OBJECT_PROCESS,
     NJS_OBJECT_MATH,
     NJS_OBJECT_JSON,
-#define NJS_OBJECT_MAX         (NJS_OBJECT_JSON + 1)
+#ifdef NJS_TEST262
+    NJS_OBJECT_262,
+#endif
+    NJS_OBJECT_MAX
 };
 
 
diff -r 0cfc928cc1e4 -r f7f7994b69b5 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Thu Sep 17 13:10:51 2020 +0300
+++ b/src/test/njs_unit_test.c	Fri Sep 18 18:00:32 2020 +0000
@@ -19578,7 +19578,11 @@ static njs_unit_test_t  njs_externals_te
       njs_str("true") },
 
     { njs_str("Object.keys(this).sort()"),
+#if (NJS_TEST262)
+      njs_str("$262,$r,$r2,$r3,$shared,global,njs,process") },
+#else
       njs_str("$r,$r2,$r3,$shared,global,njs,process") },
+#endif
 
     { njs_str("Object.getOwnPropertySymbols($r2)[0] == Symbol.toStringTag"),
       njs_str("true") },

From mdounin at mdounin.ru  Mon Sep 21 13:17:23 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 21 Sep 2020 16:17:23 +0300
Subject: Queries on Nginx 1.14.2 support
In-Reply-To: <CH2PR10MB4296DE941ACA9DC25A63DD05873A0@CH2PR10MB4296.namprd10.prod.outlook.com>
References: <CH2PR10MB4296DE941ACA9DC25A63DD05873A0@CH2PR10MB4296.namprd10.prod.outlook.com>
Message-ID: <20200921131723.GB1136@mdounin.ru>

Hello!

On Mon, Sep 21, 2020 at 04:50:23AM +0000, Suryadevara, Revanth wrote:

> We noticed that on Debian 10 the default Nginx version is 
> 1.14.2-2+deb10u3. However on your website 
> (http://nginx.org/en/download.html) Nginx 1.14.2 is deemed as a 
> "Legacy Version". So does that mean 1.14.2 is End of Support ?
> Will there be any patches provided to this version if any major 
> issues are found ?
> In general do you suggest installing any version of Nginx on 
> Debian 10 which is not part of Debian 10 repository ? If yes, do 
> we need to be aware of any issues ?

These questions does not look to be appropriate for nginx-devel@ 
mailing list, not to mention nginx-announce at .  Also, please avoid 
cross-posting, and keep your questions in the most relevant 
mailing list instead.

Thank you.

-- 
Maxim Dounin
http://mdounin.ru/

From xeioex at nginx.com  Mon Sep 21 17:16:15 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 21 Sep 2020 17:16:15 +0000
Subject: [njs] Fixed unit test on big-endian platforms.
Message-ID: <hg.00fcf5b00ce3.1600708575.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/00fcf5b00ce3
branches:  
changeset: 1526:00fcf5b00ce3
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Mon Sep 21 17:15:10 2020 +0000
description:
Fixed unit test on big-endian platforms.

diffstat:

 src/test/njs_unit_test.c |  43 +++++++++++++++++--------------------------
 1 files changed, 17 insertions(+), 26 deletions(-)

diffs (99 lines):

diff -r f7f7994b69b5 -r 00fcf5b00ce3 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Fri Sep 18 18:00:32 2020 +0000
+++ b/src/test/njs_unit_test.c	Mon Sep 21 17:15:10 2020 +0000
@@ -12,9 +12,9 @@
 #define NJS_HAVE_LARGE_STACK (!NJS_HAVE_ADDRESS_SANITIZER && !NJS_HAVE_MEMORY_SANITIZER)
 
 #ifdef NJS_HAVE_LITTLE_ENDIAN
-#define njs_evar(little, big) (little)
+#define njs_evar(little, big) little
 #else
-#define njs_evar(little, big) (big)
+#define njs_evar(little, big) big
 #endif
 
 
@@ -18345,14 +18345,14 @@ static njs_unit_test_t  njs_test[] =
     { njs_str("var de = new TextDecoder();"
               "var buf = new Uint32Array([1,2,3]).buffer;"
               "var en = new TextEncoder();"
-              "njs.dump(en.encode(de.decode(buf)))"),
-      njs_str("Uint8Array [1,0,0,0,2,0,0,0,3,0,0,0]") },
+              "njs.dump(new Uint32Array(en.encode(de.decode(buf)).buffer))"),
+      njs_str("Uint32Array [1,2,3]") },
 
     { njs_str("var de = new TextDecoder();"
               "var buf = new Uint32Array([1,2,3]).subarray(1,2);"
               "var en = new TextEncoder();"
-              "njs.dump(en.encode(de.decode(buf)))"),
-      njs_str("Uint8Array [2,0,0,0]") },
+              "njs.dump(new Uint32Array(en.encode(de.decode(buf)).buffer))"),
+      njs_str("Uint32Array [2]") },
 
     /* Buffer */
 
@@ -18408,21 +18408,12 @@ static njs_unit_test_t  njs_test[] =
               "var buf = Buffer.alloc(8, foo); buf"),
       njs_str("XXXXXXXX") },
 
-    { njs_str("var foo = new Uint16Array(10).fill(0xB1CE);"
-              "var buf = Buffer.alloc(10, foo); buf"),
-      njs_str("?????") },
-
-    { njs_str("var foo = new Uint16Array(20).fill(0xB1CE);"
-              "var buf = Buffer.alloc(10, foo); buf"),
-      njs_str("?????") },
-
-    { njs_str("var foo = new Uint16Array(2).fill(0xB1CE);"
-              "var buf = Buffer.alloc(10, foo); buf"),
-      njs_str("?????") },
-
-    { njs_str("var foo = new Uint16Array(2).fill(0xB1CE);"
-              "var buf = Buffer.alloc(10, foo); buf"),
-      njs_str("?????") },
+    { njs_str("[1,2,10,20].every(v => {"
+              "     var src = new Uint16Array(v).fill(0xB1CE);"
+              "     var buf = Buffer.alloc(10, src);"
+              "     return buf.toString() === " njs_evar("'?????'", "'??????'")
+              "})"),
+      njs_str("true") },
 
     { njs_str("var foo = Buffer.alloc(10, '?');"
               "var buf = Buffer.alloc(4, foo); buf"),
@@ -18482,11 +18473,11 @@ static njs_unit_test_t  njs_test[] =
               "var buf = Buffer.from(foo.buffer);"
               "foo[1] = 6000;"
               "njs.dump(buf)"),
-      njs_str("Buffer [136,19,112,23]") },
+      njs_str("Buffer [" njs_evar("136,19,112,23", "19,136,23,112") "]") },
 
     { njs_str("var foo = new Uint16Array(2).fill(950);"
               "var buf = Buffer.from(foo.buffer, 1); njs.dump(buf)"),
-      njs_str("Buffer [3,182,3]") },
+      njs_str("Buffer [" njs_evar("3,182,3", "182,3,182") "]") },
 
     { njs_str("var foo = new Uint16Array(2).fill(950);"
               "var buf = Buffer.from(foo.buffer, -1); njs.dump(buf)"),
@@ -18498,7 +18489,7 @@ static njs_unit_test_t  njs_test[] =
 
     { njs_str("var foo = new Uint16Array(2).fill(950);"
               "var buf = Buffer.from(foo.buffer, 2, 1); njs.dump(buf)"),
-      njs_str("Buffer [182]") },
+      njs_str("Buffer [" njs_evar("182", "3") "]") },
 
     { njs_str("var foo = new Uint16Array(2).fill(950);"
               "var buf = Buffer.from(foo.buffer, 2, -1); njs.dump(buf)"),
@@ -18514,11 +18505,11 @@ static njs_unit_test_t  njs_test[] =
 
     { njs_str("var foo = new Uint16Array(2).fill(950);"
               "var buf = Buffer.from(foo.buffer, 2, 2); njs.dump(buf)"),
-      njs_str("Buffer [182,3]") },
+      njs_str("Buffer [" njs_evar("182,3", "3,182") "]") },
 
     { njs_str("var foo = new Uint16Array(2).fill(950);"
               "var buf = Buffer.from(foo.buffer, '2', '2'); njs.dump(buf)"),
-      njs_str("Buffer [182,3]") },
+      njs_str("Buffer [" njs_evar("182,3", "3,182") "]") },
 
     { njs_str("var foo = new Uint32Array(1).fill(0xF1F2F3F4);"
               "var buf = Buffer.from(foo); njs.dump(buf)"),

From jan.prachar at gmail.com  Mon Sep 21 18:09:29 2020
From: jan.prachar at gmail.com (Jan =?UTF-8?Q?Pracha=C5=99?=)
Date: Mon, 21 Sep 2020 20:09:29 +0200
Subject: [PATCH] Proxy: set u->keepalive also if the whole body has already
 been read.
Message-ID: <772fee1c477f3e1415d87b0b938cdd0113807b6a.camel@gmail.com>

# HG changeset patch
# User Jan Pracha? <jan.prachar at gmail.com>
# Date 1600710589 -7200
#      Mon Sep 21 19:49:49 2020 +0200
# Node ID f211684e1acee34eabfdd9dd39283bcff8dc7087
# Parent  052ecc68d35038b1b4adde12efe6249a92055f09
Proxy: set u->keepalive also if the whole body has already been read.

HTTP redirection responses often contain a small few hundred bytes
body. This
allows to left keep alive connection open, if the body fits into the
upstream
buffer.

diff -r 052ecc68d350 -r f211684e1ace
src/http/modules/ngx_http_proxy_module.c
--- a/src/http/modules/ngx_http_proxy_module.c	Wed Sep 16 18:26:25
2020 +0300
+++ b/src/http/modules/ngx_http_proxy_module.c	Mon Sep 21 19:49:49
2020 +0200
@@ -1905,7 +1905,8 @@
             }
 
             /*
-             * set u->keepalive if response has no body; this allows
to keep
+             * set u->keepalive if response has no body or if the
whole body
+             * has been already read to u->buffer; this allows to keep
              * connections alive in case of r->header_only or X-Accel-
Redirect
              */
 
@@ -1915,7 +1916,7 @@
                 || u->headers_in.status_n == NGX_HTTP_NOT_MODIFIED
                 || ctx->head
                 || (!u->headers_in.chunked
-                    && u->headers_in.content_length_n == 0))
+                    && u->headers_in.content_length_n <= u-
>buffer.last-u->buffer.pos))
             {
                 u->keepalive = !u->headers_in.connection_close;
             }


From mdounin at mdounin.ru  Mon Sep 21 19:07:00 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 21 Sep 2020 22:07:00 +0300
Subject: [PATCH] Proxy: set u->keepalive also if the whole body has
 already been read.
In-Reply-To: <772fee1c477f3e1415d87b0b938cdd0113807b6a.camel@gmail.com>
References: <772fee1c477f3e1415d87b0b938cdd0113807b6a.camel@gmail.com>
Message-ID: <20200921190700.GD1136@mdounin.ru>

Hello!

On Mon, Sep 21, 2020 at 08:09:29PM +0200, Jan Pracha? wrote:

> # HG changeset patch
> # User Jan Pracha? <jan.prachar at gmail.com>
> # Date 1600710589 -7200
> #      Mon Sep 21 19:49:49 2020 +0200
> # Node ID f211684e1acee34eabfdd9dd39283bcff8dc7087
> # Parent  052ecc68d35038b1b4adde12efe6249a92055f09
> Proxy: set u->keepalive also if the whole body has already been read.
> 
> HTTP redirection responses often contain a small few hundred bytes
> body. This
> allows to left keep alive connection open, if the body fits into the
> upstream
> buffer.

HTTP redirection responses are expected to be read from the 
connection and returned to the client.  The code to do it will 
properly read the response body, check it, and will set 
u->keepalive as appropriate.

Most likely, you are trying to cover a specific case when 
redirections responses are intercepted using 
proxy_intercept_errors and error_page, as recently reported in 
ticket #2033 (https://trac.nginx.org/nginx/ticket/2033).

> diff -r 052ecc68d350 -r f211684e1ace
> src/http/modules/ngx_http_proxy_module.c
> --- a/src/http/modules/ngx_http_proxy_module.c	Wed Sep 16 18:26:25
> 2020 +0300
> +++ b/src/http/modules/ngx_http_proxy_module.c	Mon Sep 21 19:49:49
> 2020 +0200
> @@ -1905,7 +1905,8 @@
>              }
>  
>              /*
> -             * set u->keepalive if response has no body; this allows
> to keep
> +             * set u->keepalive if response has no body or if the
> whole body
> +             * has been already read to u->buffer; this allows to keep
>               * connections alive in case of r->header_only or X-Accel-
> Redirect
>               */
>  
> @@ -1915,7 +1916,7 @@
>                  || u->headers_in.status_n == NGX_HTTP_NOT_MODIFIED
>                  || ctx->head
>                  || (!u->headers_in.chunked
> -                    && u->headers_in.content_length_n == 0))
> +                    && u->headers_in.content_length_n <= u-
> >buffer.last-u->buffer.pos))
>              {
>                  u->keepalive = !u->headers_in.connection_close;
>              }

I can't say I like the idea of connection cacheability to depend 
on timing and buffer size factors.

Further, the "<=" comparison looks wrong: we shouldn't cache 
connections if there are more data than expected.

-- 
Maxim Dounin
http://mdounin.ru/

From jan.prachar at gmail.com  Mon Sep 21 19:25:36 2020
From: jan.prachar at gmail.com (Jan =?UTF-8?Q?Pracha=C5=99?=)
Date: Mon, 21 Sep 2020 21:25:36 +0200
Subject: [PATCH] Proxy: set u->keepalive also if the whole body has
 already been read.
In-Reply-To: <20200921190700.GD1136@mdounin.ru>
References: <772fee1c477f3e1415d87b0b938cdd0113807b6a.camel@gmail.com>
 <20200921190700.GD1136@mdounin.ru>
Message-ID: <5d7bb247c089357a69cd826942c32f24fd5d57d1.camel@gmail.com>

On Po, 2020-09-21 at 22:07 +0300, Maxim Dounin wrote:
> Hello!
> 
> On Mon, Sep 21, 2020 at 08:09:29PM +0200, Jan Pracha? wrote:
> 
> > # HG changeset patch
> > # User Jan Pracha? <jan.prachar at gmail.com>
> > # Date 1600710589 -7200
> > #      Mon Sep 21 19:49:49 2020 +0200
> > # Node ID f211684e1acee34eabfdd9dd39283bcff8dc7087
> > # Parent  052ecc68d35038b1b4adde12efe6249a92055f09
> > Proxy: set u->keepalive also if the whole body has already been
> > read.
> > 
> > HTTP redirection responses often contain a small few hundred bytes
> > body. This
> > allows to left keep alive connection open, if the body fits into
> > the
> > upstream
> > buffer.
> 
> HTTP redirection responses are expected to be read from the 
> connection and returned to the client.  The code to do it will 
> properly read the response body, check it, and will set 
> u->keepalive as appropriate.
> 
> Most likely, you are trying to cover a specific case when 
> redirections responses are intercepted using 
> proxy_intercept_errors and error_page, as recently reported in 
> ticket #2033 (https://trac.nginx.org/nginx/ticket/2033).

Hello, 

Yes. This is one of the use-case. I am also trying to cover X-Accel-
Redirect header in a response from upstream.


> > diff -r 052ecc68d350 -r f211684e1ace
> > src/http/modules/ngx_http_proxy_module.c
> > --- a/src/http/modules/ngx_http_proxy_module.c	Wed Sep 16
> > 18:26:25
> > 2020 +0300
> > +++ b/src/http/modules/ngx_http_proxy_module.c	Mon Sep 21
> > 19:49:49
> > 2020 +0200
> > @@ -1905,7 +1905,8 @@
> >              }
> >  
> >              /*
> > -             * set u->keepalive if response has no body; this
> > allows
> > to keep
> > +             * set u->keepalive if response has no body or if the
> > whole body
> > +             * has been already read to u->buffer; this allows to
> > keep
> >               * connections alive in case of r->header_only or X-
> > Accel-
> > Redirect
> >               */
> >  
> > @@ -1915,7 +1916,7 @@
> >                  || u->headers_in.status_n == NGX_HTTP_NOT_MODIFIED
> >                  || ctx->head
> >                  || (!u->headers_in.chunked
> > -                    && u->headers_in.content_length_n == 0))
> > +                    && u->headers_in.content_length_n <= u-
> > > buffer.last-u->buffer.pos))
> >              {
> >                  u->keepalive = !u->headers_in.connection_close;
> >              }
> 
> I can't say I like the idea of connection cacheability to depend 
> on timing and buffer size factors.
> 
> Further, the "<=" comparison looks wrong: we shouldn't cache 
> connections if there are more data than expected.

Thank you, that's a good point.


Best, Jan Pracha?



From gaoyan09 at baidu.com  Tue Sep 22 04:34:52 2020
From: gaoyan09 at baidu.com (Gao,Yan(ACG VCP))
Date: Tue, 22 Sep 2020 04:34:52 +0000
Subject: udp reuseport when reload/restart 
Message-ID: <4FF90486-41EC-413C-87E5-ACC271F1BC4D@baidu.com>

How to handle udp/quic logical connection with reuseport when reload/restart. For tcp, old worker can only handle old established connection, and do not accept new connection ever.
It is possible with BPF? Although it is not good solution


PS, why do not add conn after accept when using epoll?

if (ngx_add_conn && (ngx_event_flags & NGX_USE_EPOLL_EVENT) == 0) {
    if (ngx_add_conn(c) == NGX_ERROR) {
        ngx_close_accepted_connection(c);
        return;
    }
}


From ct at flyingcircus.io  Tue Sep 22 06:31:25 2020
From: ct at flyingcircus.io (Christian Theune)
Date: Tue, 22 Sep 2020 08:31:25 +0200
Subject: (draft) Privacy by design - offer more convenient way to
 anonymize IPs in access log by default 
In-Reply-To: <97F30C1F-9577-408B-9EFF-CC10B066EDBF@flyingcircus.io>
References: <117E0BAF-302D-41DC-97FE-C6346E018833@flyingcircus.io>
 <97F30C1F-9577-408B-9EFF-CC10B066EDBF@flyingcircus.io>
Message-ID: <C19D4C19-E38A-4799-870D-BABC168B0880@flyingcircus.io>

Hi,

as this is the first time I?m interacting with this list, I?d like to understand how to interpret silence. :)

Would someone mind educating me whether silence means:

- you?re on the wrong list, please go elsewhere (where?)
- the idea is not interesting enough to be considered
- we currently don?t have time to deal with this
- you?re completely wrong doing this and nobody wants to tell you ;)
- you?re not adhering to some rule (sorry if that is the case)
- something else (what?)

Thanks a lot,
Christian

> On 16. Sep 2020, at 21:20, Christian Theune <ct at flyingcircus.io> wrote:
> 
> Replying to myself for now ;)
> 
> One thing I can imagine being desirable would be to make the number of bits (v4/v6) that are masked configurable.
> 
> The current choice of 8 bits for IPv4 and 64-bit on IPv6 is based on our tradition that was considered a reasonable practice with respect to GDPR by courts in Germany. Unfortunately I can?t find a reliable source at the moment, but would take the time to dig one up if needed for the discussion.
> 
> Cheers,
> Christian
> 
>> On 16. Sep 2020, at 17:28, Christian Theune <ct at flyingcircus.io> wrote:
>> 
>> Signed PGP part
>> Hi!
>> 
>> first time posting here. I started working on a builtin-way for nginx to provide GDPR conformaing access logs reliably by default:
>> 
>> https://github.com/nginx/nginx/compare/branches/stable-1.18...flyingcircusio:ctheune-anonymize-by-default
>> 
>> This isn?t ready to be merged at this point as it suffers at least from two things:
>> 
>> a) I?m not a C programmer by nature and might be making stupid mistakes. This is ?monkey see monkey do code?.
>> 
>> b) You likely do not want to have this on by default for everyone, so I?m expecting that this requires a config option (runtime or compile, with a slight preference to runtime from me).
>>  The new built-in variable remote_addr_anon should be available by default in any case, though.
>> 
>> Some background: we tried implementing this purely by using a mapping approach, but this doesn?t make it a proper default as everybody defining an access log has to mention the proper format or accidentally IPs will leak. This happens over and over and we?d prefer a ?privacy by design? approach very much.
>> 
>> There is a trac ticket (https://trac.nginx.org/nginx/ticket/868) that already discusses this, however the .1 is IMHO not a good approach to recommend as it suggests a real IP whereas our impression is that nulling the last byte in IPv4 and nulling the last 8 bytes in IPv6 is the proper approach and it?s directly visible that this is an anonymized IP.
>> 
>> Let me know what you think about this and - if you?d like to see this in the mainstream code - I?d appreciate if you give me the necessary hints to bring this code to the level of quality that is proper for nginx.
>> 
>> Kind regards,
>> Christian
>> 
>> --
>> Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
>> Flying Circus Internet Operations GmbH ? http://flyingcircus.io
>> Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
>> HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick
> 
> --
> Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
> Flying Circus Internet Operations GmbH ? http://flyingcircus.io
> Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
> HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick
> 

Liebe Gr??e,
Christian Theune

--
Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
Flying Circus Internet Operations GmbH ? http://flyingcircus.io
Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: Message signed with OpenPGP
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20200922/1c0c72fe/attachment-0001.bin>

From hungnv at opensource.com.vn  Tue Sep 22 06:42:43 2020
From: hungnv at opensource.com.vn (Hung Nguyen)
Date: Tue, 22 Sep 2020 13:42:43 +0700
Subject: (draft) Privacy by design - offer more convenient way to
 anonymize IPs in access log by default
In-Reply-To: <C19D4C19-E38A-4799-870D-BABC168B0880@flyingcircus.io>
References: <C19D4C19-E38A-4799-870D-BABC168B0880@flyingcircus.io>
Message-ID: <C7B80A4C-989A-4975-A18B-4EE51B43E160@opensource.com.vn>

Hi Christian,

In my opinion your use case (GDPR) is not widely used, since Nginx offers developer number of ways to change nginx behaviour and add more feature other than default, you should consider to write your own module to archive what you want

--
H?ng

> On Sep 22, 2020, at 13:31, Christian Theune <ct at flyingcircus.io> wrote:
> 
> ?Hi,
> 
> as this is the first time I?m interacting with this list, I?d like to understand how to interpret silence. :)
> 
> Would someone mind educating me whether silence means:
> 
> - you?re on the wrong list, please go elsewhere (where?)
> - the idea is not interesting enough to be considered
> - we currently don?t have time to deal with this
> - you?re completely wrong doing this and nobody wants to tell you ;)
> - you?re not adhering to some rule (sorry if that is the case)
> - something else (what?)
> 
> Thanks a lot,
> Christian
> 
>> On 16. Sep 2020, at 21:20, Christian Theune <ct at flyingcircus.io> wrote:
>> 
>> Replying to myself for now ;)
>> 
>> One thing I can imagine being desirable would be to make the number of bits (v4/v6) that are masked configurable.
>> 
>> The current choice of 8 bits for IPv4 and 64-bit on IPv6 is based on our tradition that was considered a reasonable practice with respect to GDPR by courts in Germany. Unfortunately I can?t find a reliable source at the moment, but would take the time to dig one up if needed for the discussion.
>> 
>> Cheers,
>> Christian
>> 
>>>> On 16. Sep 2020, at 17:28, Christian Theune <ct at flyingcircus.io> wrote:
>>> 
>>> Signed PGP part
>>> Hi!
>>> 
>>> first time posting here. I started working on a builtin-way for nginx to provide GDPR conformaing access logs reliably by default:
>>> 
>>> https://github.com/nginx/nginx/compare/branches/stable-1.18...flyingcircusio:ctheune-anonymize-by-default
>>> 
>>> This isn?t ready to be merged at this point as it suffers at least from two things:
>>> 
>>> a) I?m not a C programmer by nature and might be making stupid mistakes. This is ?monkey see monkey do code?.
>>> 
>>> b) You likely do not want to have this on by default for everyone, so I?m expecting that this requires a config option (runtime or compile, with a slight preference to runtime from me).
>>> The new built-in variable remote_addr_anon should be available by default in any case, though.
>>> 
>>> Some background: we tried implementing this purely by using a mapping approach, but this doesn?t make it a proper default as everybody defining an access log has to mention the proper format or accidentally IPs will leak. This happens over and over and we?d prefer a ?privacy by design? approach very much.
>>> 
>>> There is a trac ticket (https://trac.nginx.org/nginx/ticket/868) that already discusses this, however the .1 is IMHO not a good approach to recommend as it suggests a real IP whereas our impression is that nulling the last byte in IPv4 and nulling the last 8 bytes in IPv6 is the proper approach and it?s directly visible that this is an anonymized IP.
>>> 
>>> Let me know what you think about this and - if you?d like to see this in the mainstream code - I?d appreciate if you give me the necessary hints to bring this code to the level of quality that is proper for nginx.
>>> 
>>> Kind regards,
>>> Christian
>>> 
>>> --
>>> Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
>>> Flying Circus Internet Operations GmbH ? http://flyingcircus.io
>>> Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
>>> HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick
>> 
>> --
>> Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
>> Flying Circus Internet Operations GmbH ? http://flyingcircus.io
>> Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
>> HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick
>> 
> 
> Liebe Gr??e,
> Christian Theune
> 
> --
> Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
> Flying Circus Internet Operations GmbH ? http://flyingcircus.io
> Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
> HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick
> 
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

From turchanov at farpost.com  Tue Sep 22 06:53:25 2020
From: turchanov at farpost.com (turchanov at farpost.com)
Date: Tue, 22 Sep 2020 16:53:25 +1000
Subject: [PATCH] Conditional application of limit_conn directive
Message-ID: <37619b9652dccd8e09a2affe139d063c@vl.ru>

Patch with tests will follow

# HG changeset patch
# User Sergei Turchanov <turchanov at farpost.com>
# Date 1600756248 -36000
#      Tue Sep 22 16:30:48 2020 +1000
# Node ID 2cf4e00bb96b17142a82e5f1868197465cf5c194
# Parent  052ecc68d35038b1b4adde12efe6249a92055f09
Conditional application of limit_conn directive

Implemented conditional application of limit_conn directive.
Since limit_conn directive cannot be used inside 'if' block, an optional
parameter 'if=condition' was introduced to control whether the limit is
applied or not. The limit will not be applied if the condition evaluates
to "0" or an empty string.

diff -r 052ecc68d350 -r 2cf4e00bb96b 
src/http/modules/ngx_http_limit_conn_module.c
--- a/src/http/modules/ngx_http_limit_conn_module.c	Wed Sep 16 18:26:25 
2020 +0300
+++ b/src/http/modules/ngx_http_limit_conn_module.c	Tue Sep 22 16:30:48 
2020 +1000
@@ -45,6 +45,7 @@
  typedef struct {
      ngx_shm_zone_t               *shm_zone;
      ngx_uint_t                    conn;
+    ngx_http_complex_value_t     *filter;
  } ngx_http_limit_conn_limit_t;


@@ -98,7 +99,7 @@
        NULL },

      { ngx_string("limit_conn"),
-      
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE2,
+      
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE23,
        ngx_http_limit_conn,
        NGX_HTTP_LOC_CONF_OFFSET,
        0,
@@ -181,7 +182,7 @@
  {
      size_t                          n;
      uint32_t                        hash;
-    ngx_str_t                       key;
+    ngx_str_t                       key, val;
      ngx_uint_t                      i;
      ngx_rbtree_node_t              *node;
      ngx_pool_cleanup_t             *cln;
@@ -199,6 +200,16 @@
      limits = lccf->limits.elts;

      for (i = 0; i < lccf->limits.nelts; i++) {
+        if (limits[i].filter) {
+            if (ngx_http_complex_value(r, limits[i].filter, &val) != 
NGX_OK) {
+                return NGX_ERROR;
+            }
+
+            if (val.len == 0 || (val.len == 1 && val.data[0] == '0')) {
+                continue;
+            }
+        }
+
          ctx = limits[i].shm_zone->data;

          if (ngx_http_complex_value(r, &ctx->key, &key) != NGX_OK) {
@@ -662,11 +673,12 @@
  static char *
  ngx_http_limit_conn(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
  {
-    ngx_shm_zone_t               *shm_zone;
-    ngx_http_limit_conn_conf_t   *lccf = conf;
-    ngx_http_limit_conn_limit_t  *limit, *limits;
+    ngx_shm_zone_t                     *shm_zone;
+    ngx_http_limit_conn_conf_t         *lccf = conf;
+    ngx_http_limit_conn_limit_t        *limit, *limits;
+    ngx_http_compile_complex_value_t    ccv;

-    ngx_str_t  *value;
+    ngx_str_t  *value, s;
      ngx_int_t   n;
      ngx_uint_t  i;

@@ -712,10 +724,38 @@
      if (limit == NULL) {
          return NGX_CONF_ERROR;
      }
+    ngx_memzero(limit, sizeof(ngx_http_limit_conn_limit_t));

      limit->conn = n;
      limit->shm_zone = shm_zone;

+    if (cf->args->nelts == 4) {
+        if (ngx_strncmp(value[3].data, "if=", 3) == 0) {
+            s.len = value[3].len - 3;
+            s.data = value[3].data + 3;
+
+            ngx_memzero(&ccv, 
sizeof(ngx_http_compile_complex_value_t));
+
+            ccv.cf = cf;
+            ccv.value = &s;
+            ccv.complex_value = ngx_palloc(cf->pool,
+                                           
sizeof(ngx_http_complex_value_t));
+            if (ccv.complex_value == NULL) {
+                return NGX_CONF_ERROR;
+            }
+
+            if (ngx_http_compile_complex_value(&ccv) != NGX_OK) {
+                return NGX_CONF_ERROR;
+            }
+
+            limit->filter = ccv.complex_value;
+        } else {
+            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                               "invalid parameter \"%V\"", &value[3]);
+            return NGX_CONF_ERROR;
+        }
+    }
+
      return NGX_CONF_OK;
  }


From turchanov at farpost.com  Tue Sep 22 06:53:46 2020
From: turchanov at farpost.com (turchanov at farpost.com)
Date: Tue, 22 Sep 2020 16:53:46 +1000
Subject: [PATCH] tests: Conditional application of limit_conn directive
Message-ID: <2bfdb4129a0e514c8366ea7fd10de26e@vl.ru>

# HG changeset patch
# User Sergei Turchanov <turchanov at farpost.com>
# Date 1600754712 -36000
#      Tue Sep 22 16:05:12 2020 +1000
# Node ID ed28bf1ce143fda23a16128d00099079ea18a66a
# Parent  e682d5ad3861af0a2117892a1aefc17b1303a1fa
Tests: test conditional application of limit_conn directive

diff -r e682d5ad3861 -r ed28bf1ce143 limit_conn_if.t
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/limit_conn_if.t	Tue Sep 22 16:05:12 2020 +1000
@@ -0,0 +1,114 @@
+#!/usr/bin/perl
+
+# (C) Sergei Turchanov
+
+# limit_req based tests for conditional application of limit_conn 
directive.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http proxy limit_conn limit_req/);
+
+$t->write_file_expand('nginx.conf', <<'EOF')->plan(11);
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    limit_req_zone   $binary_remote_addr  zone=req:1m rate=30r/m;
+
+    limit_conn_zone  $binary_remote_addr  zone=zone:1m;
+    limit_conn_zone  $binary_remote_addr  zone=zone2:1m;
+    limit_conn_zone  $binary_remote_addr  zone=custom:1m;
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+
+        location /w {
+            limit_req  zone=req burst=10;
+        }
+    }
+
+    map $arg_unlim $apply_limit {
+        1       "";
+        2       0;
+        3       "foo-bar";
+        default 1;
+    }
+
+    server {
+        listen       127.0.0.1:8080;
+        server_name  localhost;
+
+        location / {
+            proxy_pass http://127.0.0.1:8081;
+            limit_conn zone 1;
+        }
+
+        location /1 {
+            limit_conn zone 1;
+        }
+
+        location /cond {
+            limit_conn zone 1 if=$apply_limit;
+        }
+
+        location /zone {
+            limit_conn zone2 1;
+        }
+    }
+}
+
+EOF
+
+$t->run();
+
+###############################################################################
+
+# charge limit_req
+
+http_get('/w');
+
+# same and other zones in different locations
+
+my $s = http_get('/w', start => 1);
+like(http_get('/'), qr/^HTTP\/1.. 503 /, 'rejected');
+like(http_get('/1'), qr/^HTTP\/1.. 503 /, 'rejected different 
location');
+like(http_get('/cond'), qr/^HTTP\/1.. 503 /,
+    'rejected due to condition (if=1)');
+like(http_get('/cond?unlim=3'), qr/^HTTP\/1.. 503 /,
+    'rejected due to condition (if=somerandomstring)');
+unlike(http_get('/cond?unlim=1'), qr/^HTTP\/1.. 503 /,
+    'passed due to condition (if="")');
+like(http_get('/cond'), qr/^HTTP\/1.. 503 /, 'rejected again (if=1)');
+unlike(http_get('/cond?unlim=2'), qr/^HTTP\/1.. 503 /,
+    'passed due to condition (if=0)');
+like(http_get('/cond'), qr/^HTTP\/1.. 503 /, 'still rejected (if=1)');
+unlike(http_get('/zone'), qr/^HTTP\/1.. 503 /, 'passed different 
zone');
+
+close $s;
+unlike(http_get('/1'), qr/^HTTP\/1.. 503 /, 'passed');
+unlike(http_get('/cond'), qr/^HTTP\/1.. 503 /, 'passed (if=1)');
+
+###############################################################################

From ct at flyingcircus.io  Tue Sep 22 06:55:59 2020
From: ct at flyingcircus.io (Christian Theune)
Date: Tue, 22 Sep 2020 08:55:59 +0200
Subject: (draft) Privacy by design - offer more convenient way to
 anonymize IPs in access log by default 
In-Reply-To: <C7B80A4C-989A-4975-A18B-4EE51B43E160@opensource.com.vn>
References: <C19D4C19-E38A-4799-870D-BABC168B0880@flyingcircus.io>
 <C7B80A4C-989A-4975-A18B-4EE51B43E160@opensource.com.vn>
Message-ID: <75130859-3832-497F-A176-17A7B6D8B38A@flyingcircus.io>

Hi,

Thanks for that input!

> On 22. Sep 2020, at 08:42, Hung Nguyen <hungnv at opensource.com.vn> wrote:
> 
> Hi Christian,
> 
> In my opinion your use case (GDPR) is not widely used, since Nginx offers developer number of ways to change nginx behaviour and add more feature other than default, you should consider to write your own module to archive what you want

Interesting. Technically one can currently make a single nginx config that is GDPR compliant WRT IP logging. However, it?s AFAICT impossible to set up nginx in a way so that delegating virtual host configuration to another party doesn?t automatically lead to accidents (we weren?t able to avoid accidents even without delegation).

I was surprised that there is no way to change the default logging format reliably - and this could be an alternative path more relevant to the core with two options that I see:

1. allow redefining the ?combined? log format, or
2. allow explicitly setting another format as default (might be easier when I look at the current structure)

Having the anonymized IP as a separate value (remote_addr_anon) could easily be extracted into a separate module but maybe its so lightweight that adding it to the core makes sense as well. We originally did it purely using maps, my gut feeling tells me that that?s much slower but we don?t have any evidence of it making a significant impact at our traffic levels.

I would have thought that GDPR would be more relevant as nginx is so widely spread and privacy compliance has been such a big topic in Europe over the last years ? Googling for ?nginx gdpr? gives ?only? 833k results. Not nothing but I kind of expected a larger result set.

Cheers,
Christian

--
Christian Theune ? ct at flyingcircus.io ? +49 345 219401 0
Flying Circus Internet Operations GmbH ? http://flyingcircus.io
Leipziger Str. 70/71 ? 06108 Halle (Saale) ? Deutschland
HR Stendal HRB 21169 ? Gesch?ftsf?hrer: Christian Theune, Christian Zagrodnick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: Message signed with OpenPGP
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20200922/225a14e9/attachment.bin>

From turchanov at farpost.com  Tue Sep 22 07:05:45 2020
From: turchanov at farpost.com (turchanov at farpost.com)
Date: Tue, 22 Sep 2020 17:05:45 +1000
Subject: [PATCH... kind of] documentation: Conditional application of
 limit_conn directive
Message-ID: <819ce0f06df2d3558bc2f183091f028e@vl.ru>

Hello!

Couldn't find (https://nginx.org/ru/docs/contributing_changes.html) 
information on
were to commit changes to documentation describing "limit_conn". I will 
post them here.
Sorry about that. (Or point me to the resource were it is explained how 
to make changes to
documentation).

============

Syntax:	limit_conn zone number [if=condition];
Default:	?
Context:	http, server, location

...

The "if" parameter controls conditional application of the limit.
The limit will not be applied if the condition evaluates to "0" or an 
empty string.


=============

From jan.prachar at gmail.com  Tue Sep 22 09:40:09 2020
From: jan.prachar at gmail.com (Jan =?UTF-8?Q?Pracha=C5=99?=)
Date: Tue, 22 Sep 2020 11:40:09 +0200
Subject: [PATCH] Proxy: set u->keepalive also if the whole body has
 already been read.
In-Reply-To: <20200921190700.GD1136@mdounin.ru>
References: <772fee1c477f3e1415d87b0b938cdd0113807b6a.camel@gmail.com>
 <20200921190700.GD1136@mdounin.ru>
Message-ID: <df9997125544f981dc9e9699aa1aa5fb4db1a459.camel@gmail.com>

On Po, 2020-09-21 at 22:07 +0300, Maxim Dounin wrote:
> @@ -1915,7 +1916,7 @@
> >                  || u->headers_in.status_n == NGX_HTTP_NOT_MODIFIED
> >                  || ctx->head
> >                  || (!u->headers_in.chunked
> > -                    && u->headers_in.content_length_n == 0))
> > +                    && u->headers_in.content_length_n <= u->buffer.last-u->buffer.pos))
> >              {
> >                  u->keepalive = !u->headers_in.connection_close;
> >              }
> 
> I can't say I like the idea of connection cacheability to depend 
> on timing and buffer size factors.
> 
> Further, the "<=" comparison looks wrong: we shouldn't cache 
> connections if there are more data than expected.

Hello, I changed it in the following patch, in case you would like to use it. Now the
condition is more strict though, because previously was responsonses with Content-Lenght:
0 and non empty upstream buffer kept in connection cache.

# HG changeset patch
# User Jan Pracha?  <jan.prachar at gmail.com>
# Date 1600710589 -7200
#      Mon Sep 21 19:49:49 2020 +0200
# Node ID e35b529b03781e64912e0d8a72bd0f957dc08cd2
# Parent  052ecc68d35038b1b4adde12efe6249a92055f09
set u->keepalive also if the whole body has already been read

diff -r 052ecc68d350 -r e35b529b0378 src/http/modules/ngx_http_proxy_module.c
--- a/src/http/modules/ngx_http_proxy_module.c	Wed Sep 16 18:26:25 2020 +0300
+++ b/src/http/modules/ngx_http_proxy_module.c	Mon Sep 21 19:49:49 2020 +0200
@@ -1905,7 +1905,8 @@
             }
 
             /*
-             * set u->keepalive if response has no body; this allows to keep
+             * set u->keepalive if response has no body or if the whole body
+             * has been already read to u->buffer; this allows to keep
              * connections alive in case of r->header_only or X-Accel-Redirect
              */
 
@@ -1915,7 +1916,7 @@
                 || u->headers_in.status_n == NGX_HTTP_NOT_MODIFIED
                 || ctx->head
                 || (!u->headers_in.chunked
-                    && u->headers_in.content_length_n == 0))
+                    && u->headers_in.content_length_n == u->buffer.last - u->buffer.pos))
             {
                 u->keepalive = !u->headers_in.connection_close;
             }


From pluknet at nginx.com  Tue Sep 22 11:05:48 2020
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Tue, 22 Sep 2020 12:05:48 +0100
Subject: udp reuseport when reload/restart 
In-Reply-To: <4FF90486-41EC-413C-87E5-ACC271F1BC4D@baidu.com>
References: <4FF90486-41EC-413C-87E5-ACC271F1BC4D@baidu.com>
Message-ID: <D571D733-4DC6-4824-B04A-32CEDAC0EA10@nginx.com>


> On 22 Sep 2020, at 05:34, Gao,Yan(ACG VCP) <gaoyan09 at baidu.com> wrote:
> 
> How to handle udp/quic logical connection with reuseport when reload/restart. For tcp, old worker can only handle old established connection, and do not accept new connection ever.

The current behaviour is to reject new QUIC connections
with CONNECTION_REFUSED after graceful shutdown was initiated,
and let already established connections to complete.

-- 
Sergey Kandaurov


From mdounin at mdounin.ru  Tue Sep 22 13:19:23 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 22 Sep 2020 16:19:23 +0300
Subject: [PATCH] Conditional application of limit_conn directive
In-Reply-To: <37619b9652dccd8e09a2affe139d063c@vl.ru>
References: <37619b9652dccd8e09a2affe139d063c@vl.ru>
Message-ID: <20200922131923.GE1136@mdounin.ru>

Hello!

On Tue, Sep 22, 2020 at 04:53:25PM +1000, turchanov at farpost.com wrote:

> Patch with tests will follow
> 
> # HG changeset patch
> # User Sergei Turchanov <turchanov at farpost.com>
> # Date 1600756248 -36000
> #      Tue Sep 22 16:30:48 2020 +1000
> # Node ID 2cf4e00bb96b17142a82e5f1868197465cf5c194
> # Parent  052ecc68d35038b1b4adde12efe6249a92055f09
> Conditional application of limit_conn directive
> 
> Implemented conditional application of limit_conn directive.
> Since limit_conn directive cannot be used inside 'if' block, an optional
> parameter 'if=condition' was introduced to control whether the limit is
> applied or not. The limit will not be applied if the condition evaluates
> to "0" or an empty string.

Thank you for the patch.

Quoting the documentation (http://nginx.org/r/limit_conn_zone):

: Requests with an empty key value are not accounted.

That is, limit_conn can be used conditionally without any 
code changes.  The same applies to limit_req.

-- 
Maxim Dounin
http://mdounin.ru/

From turchanov at farpost.com  Tue Sep 22 14:39:32 2020
From: turchanov at farpost.com (turchanov at farpost.com)
Date: Wed, 23 Sep 2020 00:39:32 +1000
Subject: [PATCH] Conditional application of limit_conn directive
In-Reply-To: <20200922131923.GE1136@mdounin.ru>
References: <37619b9652dccd8e09a2affe139d063c@vl.ru>
 <20200922131923.GE1136@mdounin.ru>
Message-ID: <54c602bf82cd03c12ca51bd30924578d@vl.ru>

Hello!

...
Maxim Dounin ????? 2020-09-22 23:19:
> Quoting the documentation (http://nginx.org/r/limit_conn_zone):
> 
> : Requests with an empty key value are not accounted.
> 
> That is, limit_conn can be used conditionally without any
> code changes.  The same applies to limit_req.

Yes, I know about that feature. The problem with that approach is a 
complication of the
process to compute the value of a "key". We tried that before (with a 
cascade of "map" directives)
and configuration became unreadable and very hard to understand when you 
add another indirection
level to force a "key" to be an empty value depending on another 
condition. That was an incentive
for this patch.
This is especially true when there are several "limit_conn" directives 
in one location
to limit user sessions by different identification attributes (and all 
of those attirbutes
need to be "emptied" on a condition).

With best regards,
Sergei Turchanov.

From xeioex at nginx.com  Wed Sep 23 14:17:31 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 23 Sep 2020 14:17:31 +0000
Subject: [njs] Adding support for Buffer objects in crypto methods.
Message-ID: <hg.eef4ab1bee70.1600870651.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/eef4ab1bee70
branches:  
changeset: 1527:eef4ab1bee70
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Tue Sep 22 18:42:21 2020 +0000
description:
Adding support for Buffer objects in crypto methods.

diffstat:

 src/njs_buffer.c         |  212 +++++++++++-------------
 src/njs_buffer.h         |   22 ++
 src/njs_crypto.c         |  402 +++++++++++++++++++++-------------------------
 src/test/njs_unit_test.c |  291 ++++++++++++++++++---------------
 4 files changed, 463 insertions(+), 464 deletions(-)

diffs (truncated from 1417 to 1000 lines):

diff -r 00fcf5b00ce3 -r eef4ab1bee70 src/njs_buffer.c
--- a/src/njs_buffer.c	Mon Sep 21 17:15:10 2020 +0000
+++ b/src/njs_buffer.c	Tue Sep 22 18:42:21 2020 +0000
@@ -17,19 +17,6 @@
     ((size << 2) | (sign << 1) | little)
 
 
-typedef njs_int_t (*njs_buffer_encode)(njs_vm_t *vm, njs_value_t *value,
-    const njs_str_t *src);
-typedef size_t (*njs_buffer_encode_length)(const njs_str_t *src,
-                                           size_t *out_size);
-
-typedef struct {
-    njs_str_t                 name;
-    njs_buffer_encode         encode;
-    njs_buffer_encode         decode;
-    njs_buffer_encode_length  decode_length;
-} njs_buffer_encoding_t;
-
-
 static njs_buffer_encoding_t  njs_buffer_encodings[] =
 {
     {
@@ -83,18 +70,15 @@ static njs_int_t njs_buffer_write_string
     njs_typed_array_t *array, const njs_buffer_encoding_t *encoding,
     uint64_t offset, uint64_t length);
 static njs_int_t njs_buffer_fill(njs_vm_t *vm, njs_typed_array_t *array,
-    njs_value_t *value, const njs_buffer_encoding_t *encoding, uint64_t offset,
+    const njs_value_t *fill, const njs_value_t *encoding, uint64_t offset,
     uint64_t end);
-static njs_int_t njs_buffer_fill_string(njs_vm_t *vm, njs_value_t *value,
+static njs_int_t njs_buffer_fill_string(njs_vm_t *vm, const njs_value_t *value,
     njs_typed_array_t *array, const njs_buffer_encoding_t *encoding,
     uint8_t *start, uint8_t *end);
-static njs_int_t njs_buffer_fill_typed_array(njs_vm_t *vm, njs_value_t *value,
-    njs_typed_array_t *array, uint8_t *start, uint8_t *end);
-static const njs_buffer_encoding_t *njs_buffer_encoding(njs_vm_t *vm,
-    njs_value_t *value);
-static njs_int_t njs_buffer_decode_string(njs_vm_t *vm, njs_value_t *value,
-    njs_value_t *dst, const njs_buffer_encoding_t *encoding);
-static void njs_buffer_decode_destroy(njs_vm_t *vm, njs_value_t *source,
+static njs_int_t njs_buffer_fill_typed_array(njs_vm_t *vm,
+    const njs_value_t *value, njs_typed_array_t *array, uint8_t *start,
+    uint8_t *end);
+static void njs_buffer_decode_destroy(njs_vm_t *vm, const njs_value_t *source,
     njs_value_t *target);
 
 
@@ -149,8 +133,8 @@ njs_buffer_set(njs_vm_t *vm, njs_value_t
 }
 
 
-static njs_typed_array_t *
-njs_buffer_alloc_array(njs_vm_t *vm, size_t size, njs_bool_t zeroing)
+njs_typed_array_t *
+njs_buffer_alloc(njs_vm_t *vm, size_t size, njs_bool_t zeroing)
 {
     njs_value_t        value;
     njs_typed_array_t  *array;
@@ -169,6 +153,25 @@ njs_buffer_alloc_array(njs_vm_t *vm, siz
 }
 
 
+njs_int_t
+njs_buffer_new(njs_vm_t *vm, njs_value_t *value, const u_char *start,
+    uint32_t size)
+{
+    njs_typed_array_t  *buffer;
+
+    buffer = njs_buffer_alloc(vm, size, 0);
+    if (njs_slow_path(buffer == NULL)) {
+        return NJS_ERROR;
+    }
+
+    memcpy(njs_typed_array_buffer(buffer)->u.u8, start, size);
+
+    njs_set_typed_array(value, buffer);
+
+    return NJS_OK;
+}
+
+
 static njs_int_t
 njs_buffer_constructor(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t unused)
@@ -180,13 +183,13 @@ njs_buffer_constructor(njs_vm_t *vm, njs
 
 
 static njs_int_t
-njs_buffer_alloc(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
+njs_buffer_alloc_safe(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t safe)
 {
-    double                       size;
-    njs_int_t                    ret;
-    njs_typed_array_t            *array;
-    const njs_buffer_encoding_t  *encoding;
+    double             size;
+    njs_int_t          ret;
+    njs_typed_array_t  *array;
+    const njs_value_t  *fill;
 
     if (njs_slow_path(!njs_is_number(njs_arg(args, nargs, 1)))) {
         njs_type_error(vm, "\"size\" argument must be of type number");
@@ -199,22 +202,15 @@ njs_buffer_alloc(njs_vm_t *vm, njs_value
         return NJS_ERROR;
     }
 
-    array = njs_buffer_alloc_array(vm, size, safe || nargs <= 2);
+    array = njs_buffer_alloc(vm, size, safe || nargs <= 2);
     if (njs_slow_path(array == NULL)) {
         return NJS_ERROR;
     }
 
-    if (safe && nargs > 2) {
-        encoding = njs_buffer_utf8_encoding();
-
-        if (nargs > 3 && njs_is_string(njs_argument(args, 2))) {
-            encoding = njs_buffer_encoding(vm, njs_argument(args, 3));
-            if (njs_slow_path(encoding == NULL)) {
-                return NJS_ERROR;
-            }
-        }
-
-        ret = njs_buffer_fill(vm, array, njs_argument(args, 2), encoding, 0,
+    fill = njs_arg(args, nargs, 2);
+
+    if (safe && njs_is_defined(fill)) {
+        ret = njs_buffer_fill(vm, array, fill, njs_arg(args, nargs, 3), 0,
                               array->byte_length);
         if (njs_slow_path(ret != NJS_OK)) {
             return NJS_ERROR;
@@ -246,13 +242,9 @@ njs_buffer_from(njs_vm_t *vm, njs_value_
                                             njs_arg(args, nargs, 3));
 
     case NJS_STRING:
-        encoding = njs_buffer_utf8_encoding();
-
-        if (nargs > 2) {
-            encoding = njs_buffer_encoding(vm, njs_argument(args, 2));
-            if (njs_slow_path(encoding == NULL)) {
-                return NJS_ERROR;
-            }
+        encoding = njs_buffer_encoding(vm, njs_arg(args, nargs, 2));
+        if (njs_slow_path(encoding == NULL)) {
+            return NJS_ERROR;
         }
 
         return njs_buffer_from_string(vm, value, encoding);
@@ -334,7 +326,7 @@ next:
         return ret;
     }
 
-    buffer = njs_buffer_alloc_array(vm, len, 0);
+    buffer = njs_buffer_alloc(vm, len, 0);
     if (njs_slow_path(buffer == NULL)) {
         return NJS_ERROR;
     }
@@ -448,7 +440,7 @@ njs_buffer_from_typed_array(njs_vm_t *vm
 
     length = njs_typed_array_length(array);
 
-    buffer = njs_buffer_alloc_array(vm, length, 0);
+    buffer = njs_buffer_alloc(vm, length, 0);
     if (njs_slow_path(buffer == NULL)) {
         return NJS_ERROR;
     }
@@ -481,7 +473,7 @@ njs_buffer_from_string(njs_vm_t *vm, njs
 
     njs_string_get(&dst, &str);
 
-    buffer = njs_buffer_alloc_array(vm, str.length, 0);
+    buffer = njs_buffer_alloc(vm, str.length, 0);
     if (njs_slow_path(buffer == NULL)) {
         return NJS_ERROR;
     }
@@ -812,7 +804,7 @@ njs_buffer_concat(njs_vm_t *vm, njs_valu
         }
     }
 
-    buffer = njs_buffer_alloc_array(vm, len, 0);
+    buffer = njs_buffer_alloc(vm, len, 0);
     if (njs_slow_path(buffer == NULL)) {
         return NJS_ERROR;
     }
@@ -884,8 +876,11 @@ static njs_int_t
 njs_buffer_is_encoding(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t unused)
 {
-    njs_set_boolean(&vm->retval,
-                    njs_buffer_encoding(vm, njs_arg(args, nargs, 1)) != NULL);
+    const njs_value_t  *value;
+
+    value = njs_arg(args, nargs, 1);
+    njs_set_boolean(&vm->retval, njs_is_defined(value)
+                                 && njs_buffer_encoding(vm, value) != NULL);
 
     return NJS_OK;
 }
@@ -1563,12 +1558,11 @@ static njs_int_t
 njs_buffer_prototype_fill(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t unused)
 {
-    uint64_t                     offset, end;
-    njs_int_t                    ret;
-    njs_value_t                  *this, *value, *value_offset, *value_end,
-                                 *encode;
-    njs_typed_array_t            *array;
-    const njs_buffer_encoding_t  *encoding;
+    uint64_t           offset, end;
+    njs_int_t          ret;
+    njs_value_t        *this, *value, *value_offset, *value_end,
+                       *encode;
+    njs_typed_array_t  *array;
 
     this = njs_argument(args, 0);
     if (njs_slow_path(nargs < 2)) {
@@ -1584,8 +1578,6 @@ njs_buffer_prototype_fill(njs_vm_t *vm, 
     value_offset = njs_arg(args, nargs, 2);
     value_end = njs_arg(args, nargs, 3);
     encode = njs_arg(args, nargs, 4);
-    encoding = njs_buffer_utf8_encoding();
-
 
     offset = 0;
     end = array->byte_length;
@@ -1593,7 +1585,7 @@ njs_buffer_prototype_fill(njs_vm_t *vm, 
     if (njs_is_defined(value_offset)) {
         if (njs_is_string(value) && njs_is_string(value_offset)) {
             encode = value_offset;
-            goto encoding;
+            goto fill;
         }
 
         ret = njs_value_to_index(vm, value_offset, &offset);
@@ -1605,7 +1597,7 @@ njs_buffer_prototype_fill(njs_vm_t *vm, 
     if (njs_is_defined(value_end)) {
         if (njs_is_string(value) && njs_is_string(value_end)) {
             encode = value_end;
-            goto encoding;
+            goto fill;
         }
 
         ret = njs_value_to_index(vm, value_end, &end);
@@ -1614,21 +1606,9 @@ njs_buffer_prototype_fill(njs_vm_t *vm, 
         }
     }
 
-    if (njs_is_defined(encode)) {
-        if (njs_slow_path(!njs_is_string(encode))) {
-            njs_type_error(vm, "\"encoding\" argument must be of type string");
-            return NJS_ERROR;
-        }
-
-    encoding:
-
-        encoding = njs_buffer_encoding(vm, encode);
-        if (njs_slow_path(encoding == NULL)) {
-            return NJS_ERROR;
-        }
-    }
-
-    ret = njs_buffer_fill(vm, array, value, encoding, offset, end);
+fill:
+
+    ret = njs_buffer_fill(vm, array, value, encode, offset, end);
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
@@ -1642,13 +1622,14 @@ done:
 
 
 static njs_int_t
-njs_buffer_fill(njs_vm_t *vm, njs_typed_array_t *array, njs_value_t *value,
-    const njs_buffer_encoding_t *encoding, uint64_t offset, uint64_t end)
+njs_buffer_fill(njs_vm_t *vm, njs_typed_array_t *array, const njs_value_t *fill,
+    const njs_value_t *encode, uint64_t offset, uint64_t end)
 {
-    double              num;
-    uint8_t             *start, *stop;
-    njs_int_t           ret;
-    njs_array_buffer_t  *buffer;
+    double                       num;
+    uint8_t                      *start, *stop;
+    njs_int_t                    ret;
+    njs_array_buffer_t           *buffer;
+    const njs_buffer_encoding_t  *encoding;
 
     buffer = njs_typed_array_writable(vm, array);
     if (njs_slow_path(buffer == NULL)) {
@@ -1672,15 +1653,20 @@ njs_buffer_fill(njs_vm_t *vm, njs_typed_
     start = &buffer->u.u8[array->offset + offset];
     stop = &buffer->u.u8[array->offset + end];
 
-    switch (value->type) {
+    switch (fill->type) {
     case NJS_STRING:
-        return njs_buffer_fill_string(vm, value, array, encoding, start, stop);
+        encoding = njs_buffer_encoding(vm, encode);
+        if (njs_slow_path(encoding == NULL)) {
+            return NJS_ERROR;
+        }
+
+        return njs_buffer_fill_string(vm, fill, array, encoding, start, stop);
 
     case NJS_TYPED_ARRAY:
-        return njs_buffer_fill_typed_array(vm, value, array, start, stop);
+        return njs_buffer_fill_typed_array(vm, fill, array, start, stop);
 
     default:
-        ret = njs_value_to_number(vm, value, &num);
+        ret = njs_value_to_number(vm, njs_value_arg(fill), &num);
         if (njs_slow_path(ret != NJS_OK)) {
             return ret;
         }
@@ -1698,7 +1684,7 @@ njs_buffer_fill(njs_vm_t *vm, njs_typed_
 
 
 static njs_int_t
-njs_buffer_fill_string(njs_vm_t *vm, njs_value_t *value,
+njs_buffer_fill_string(njs_vm_t *vm, const njs_value_t *value,
     njs_typed_array_t *array, const njs_buffer_encoding_t *encoding,
     uint8_t *start, uint8_t *end)
 {
@@ -1733,7 +1719,7 @@ done:
 
 
 static njs_int_t
-njs_buffer_fill_typed_array(njs_vm_t *vm, njs_value_t *value,
+njs_buffer_fill_typed_array(njs_vm_t *vm, const njs_value_t *value,
     njs_typed_array_t *array, uint8_t *to, uint8_t *end)
 {
     size_t              byte_length;
@@ -1773,7 +1759,7 @@ njs_buffer_prototype_to_string(njs_vm_t 
     uint64_t                     start, end;
     njs_int_t                    ret;
     njs_str_t                    str;
-    njs_value_t                  *this, *enc, *value_start, *value_end;
+    njs_value_t                  *this, *value_start, *value_end;
     njs_typed_array_t            *array;
     const njs_buffer_encoding_t  *encoding;
 
@@ -1783,7 +1769,6 @@ njs_buffer_prototype_to_string(njs_vm_t 
         return NJS_ERROR;
     }
 
-    enc = njs_arg(args, nargs, 1);
     value_start = njs_arg(args, nargs, 2);
     value_end = njs_arg(args, nargs, 3);
 
@@ -1791,11 +1776,9 @@ njs_buffer_prototype_to_string(njs_vm_t 
     end = array->byte_length;
     encoding = njs_buffer_utf8_encoding();
 
-    if (njs_is_defined(enc)) {
-        encoding = njs_buffer_encoding(vm, njs_argument(args, 1));
-        if (njs_slow_path(encoding == NULL)) {
-            return NJS_ERROR;
-        }
+    encoding = njs_buffer_encoding(vm,  njs_arg(args, nargs, 1));
+    if (njs_slow_path(encoding == NULL)) {
+        return NJS_ERROR;
     }
 
     if (njs_is_defined(value_start)) {
@@ -2265,40 +2248,41 @@ njs_buffer_prototype_to_json(njs_vm_t *v
 }
 
 
-static const njs_buffer_encoding_t *
-njs_buffer_encoding(njs_vm_t *vm, njs_value_t *value)
+const njs_buffer_encoding_t *
+njs_buffer_encoding(njs_vm_t *vm, const njs_value_t *value)
 {
-    njs_str_t              str;
-    njs_int_t              ret;
+    njs_str_t              name;
     njs_buffer_encoding_t  *encoding;
 
     if (njs_slow_path(!njs_is_string(value))) {
-        ret = njs_value_to_string(vm, value, value);
-        if (njs_slow_path(ret != NJS_OK)) {
+        if (njs_is_defined(value)) {
+            njs_type_error(vm, "encoding must be a string");
             return NULL;
         }
+
+        return &njs_buffer_encodings[0];
     }
 
-    njs_string_get(value, &str);
+    njs_string_get(value, &name);
 
     for (encoding = &njs_buffer_encodings[0];
          encoding->name.length != 0;
          encoding++)
     {
-        if (njs_strstr_eq(&str, &encoding->name)) {
+        if (njs_strstr_eq(&name, &encoding->name)) {
             return encoding;
         }
     }
 
-    njs_type_error(vm, "\"%V\" encoding is not supported", &str);
+    njs_type_error(vm, "\"%V\" encoding is not supported", &name);
 
     return NULL;
 }
 
 
-static njs_int_t
-njs_buffer_decode_string(njs_vm_t *vm, njs_value_t *value, njs_value_t *dst,
-    const njs_buffer_encoding_t *encoding)
+njs_int_t
+njs_buffer_decode_string(njs_vm_t *vm, const njs_value_t *value,
+    njs_value_t *dst, const njs_buffer_encoding_t *encoding)
 {
     njs_int_t          ret;
     njs_str_t          str;
@@ -2325,7 +2309,7 @@ njs_buffer_decode_string(njs_vm_t *vm, n
 
 
 static void
-njs_buffer_decode_destroy(njs_vm_t *vm, njs_value_t *source,
+njs_buffer_decode_destroy(njs_vm_t *vm, const njs_value_t *source,
     njs_value_t *target)
 {
     njs_str_t  src, trg;
@@ -2833,7 +2817,7 @@ static const njs_object_prop_t  njs_buff
     {
         .type = NJS_PROPERTY,
         .name = njs_string("alloc"),
-        .value = njs_native_function2(njs_buffer_alloc, 0, 1),
+        .value = njs_native_function2(njs_buffer_alloc_safe, 0, 1),
         .writable = 1,
         .configurable = 1,
     },
@@ -2841,7 +2825,7 @@ static const njs_object_prop_t  njs_buff
     {
         .type = NJS_PROPERTY,
         .name = njs_string("allocUnsafe"),
-        .value = njs_native_function2(njs_buffer_alloc, 1, 0),
+        .value = njs_native_function2(njs_buffer_alloc_safe, 1, 0),
         .writable = 1,
         .configurable = 1,
     },
@@ -2849,7 +2833,7 @@ static const njs_object_prop_t  njs_buff
     {
         .type = NJS_PROPERTY,
         .name = njs_long_string("allocUnsafeSlow"),
-        .value = njs_native_function2(njs_buffer_alloc, 1, 0),
+        .value = njs_native_function2(njs_buffer_alloc_safe, 1, 0),
         .writable = 1,
         .configurable = 1,
     },
diff -r 00fcf5b00ce3 -r eef4ab1bee70 src/njs_buffer.h
--- a/src/njs_buffer.h	Mon Sep 21 17:15:10 2020 +0000
+++ b/src/njs_buffer.h	Tue Sep 22 18:42:21 2020 +0000
@@ -8,8 +8,30 @@
 #define _NJS_BUFFER_H_INCLUDED_
 
 
+typedef njs_int_t (*njs_buffer_encode_t)(njs_vm_t *vm, njs_value_t *value,
+    const njs_str_t *src);
+typedef size_t (*njs_buffer_encode_length_t)(const njs_str_t *src,
+                                             size_t *out_size);
+
+typedef struct {
+    njs_str_t                   name;
+    njs_buffer_encode_t         encode;
+    njs_buffer_encode_t         decode;
+    njs_buffer_encode_length_t  decode_length;
+} njs_buffer_encoding_t;
+
+
 njs_int_t njs_buffer_set(njs_vm_t *vm, njs_value_t *value, const u_char *start,
     uint32_t size);
+njs_int_t njs_buffer_new(njs_vm_t *vm, njs_value_t *value, const u_char *start,
+    uint32_t size);
+njs_typed_array_t *njs_buffer_alloc(njs_vm_t *vm, size_t size,
+    njs_bool_t zeroing);
+
+const njs_buffer_encoding_t *njs_buffer_encoding(njs_vm_t *vm,
+    const njs_value_t *value);
+njs_int_t njs_buffer_decode_string(njs_vm_t *vm, const njs_value_t *value,
+    njs_value_t *dst, const njs_buffer_encoding_t *encoding);
 
 
 extern const njs_object_type_init_t  njs_buffer_type_init;
diff -r 00fcf5b00ce3 -r eef4ab1bee70 src/njs_crypto.c
--- a/src/njs_crypto.c	Mon Sep 21 17:15:10 2020 +0000
+++ b/src/njs_crypto.c	Tue Sep 22 18:42:21 2020 +0000
@@ -56,6 +56,14 @@ typedef struct {
 } njs_crypto_enc_t;
 
 
+static njs_hash_alg_t *njs_crypto_algorithm(njs_vm_t *vm,
+    const njs_value_t *value);
+static njs_crypto_enc_t *njs_crypto_encoding(njs_vm_t *vm,
+    const njs_value_t *value);
+static njs_int_t njs_buffer_digest(njs_vm_t *vm, njs_value_t *value,
+    const njs_str_t *src);
+
+
 static njs_hash_alg_t njs_hash_algorithms[] = {
 
    {
@@ -92,9 +100,15 @@ static njs_hash_alg_t njs_hash_algorithm
 
 };
 
+
 static njs_crypto_enc_t njs_encodings[] = {
 
    {
+     njs_str("buffer"),
+     njs_buffer_digest
+   },
+
+   {
      njs_str("hex"),
      njs_string_hex
    },
@@ -116,11 +130,6 @@ static njs_crypto_enc_t njs_encodings[] 
 };
 
 
-static njs_hash_alg_t *njs_crypto_alg(njs_vm_t *vm, const njs_str_t *name);
-static njs_crypto_enc_t *njs_crypto_encoding(njs_vm_t *vm,
-    const njs_str_t *name);
-
-
 static njs_object_value_t *
 njs_crypto_object_value_alloc(njs_vm_t *vm, njs_object_type_t type)
 {
@@ -152,19 +161,11 @@ static njs_int_t
 njs_crypto_create_hash(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t unused)
 {
-    njs_str_t           alg_name;
     njs_digest_t        *dgst;
     njs_hash_alg_t      *alg;
     njs_object_value_t  *hash;
 
-    if (njs_slow_path(nargs < 2 || !njs_is_string(&args[1]))) {
-        njs_type_error(vm, "algorithm must be a string");
-        return NJS_ERROR;
-    }
-
-    njs_string_get(&args[1], &alg_name);
-
-    alg = njs_crypto_alg(vm, &alg_name);
+    alg = njs_crypto_algorithm(vm, njs_arg(args, nargs, 1));
     if (njs_slow_path(alg == NULL)) {
         return NJS_ERROR;
     }
@@ -193,34 +194,79 @@ njs_crypto_create_hash(njs_vm_t *vm, njs
 
 static njs_int_t
 njs_hash_prototype_update(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
-    njs_index_t unused)
+    njs_index_t tag)
 {
-    njs_str_t     data;
-    njs_value_t   *this;
-    njs_digest_t  *dgst;
-
-    if (njs_slow_path(nargs < 2 || !njs_is_string(&args[1]))) {
-        njs_type_error(vm, "data must be a string");
-        return NJS_ERROR;
-    }
+    njs_str_t                    data;
+    njs_int_t                    ret;
+    njs_hmac_t                   *ctx;
+    njs_value_t                  *this, dst;
+    njs_digest_t                 *dgst;
+    njs_typed_array_t            *array;
+    const njs_value_t            *value;
+    njs_array_buffer_t           *buffer;
+    const njs_buffer_encoding_t  *encoding;
 
     this = njs_argument(args, 0);
-
-    if (njs_slow_path(!njs_is_object_data(this, NJS_DATA_TAG_CRYPTO_HASH))) {
+    if (njs_slow_path(!njs_is_object_data(this, tag))) {
         njs_type_error(vm, "\"this\" is not a hash object");
         return NJS_ERROR;
     }
 
-    njs_string_get(&args[1], &data);
+    value = njs_arg(args, nargs, 1);
+
+    switch (value->type) {
+    case NJS_STRING:
+        encoding = njs_buffer_encoding(vm, njs_arg(args, nargs, 2));
+        if (njs_slow_path(encoding == NULL)) {
+            return NJS_ERROR;
+        }
+
+        ret = njs_buffer_decode_string(vm, value, &dst, encoding);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_ERROR;
+        }
+
+        njs_string_get(&dst, &data);
+        break;
 
-    dgst = njs_object_data(this);
+    case NJS_TYPED_ARRAY:
+    case NJS_DATA_VIEW:
+        array = njs_typed_array(value);
+        buffer = array->buffer;
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
 
-    if (njs_slow_path(dgst->alg == NULL)) {
-        njs_error(vm, "Digest already called");
+        data.start = &buffer->u.u8[array->offset];
+        data.length = array->byte_length;
+        break;
+
+    default:
+        njs_type_error(vm, "data argument \"%s\" is not a string "
+                       "or Buffer-like object", njs_type_string(value->type));
+
         return NJS_ERROR;
     }
 
-    dgst->alg->update(&dgst->u, data.start, data.length);
+    if (tag == NJS_DATA_TAG_CRYPTO_HASH) {
+        dgst = njs_object_data(this);
+        if (njs_slow_path(dgst->alg == NULL)) {
+            njs_error(vm, "Digest already called");
+            return NJS_ERROR;
+        }
+
+        dgst->alg->update(&dgst->u, data.start, data.length);
+
+    } else {
+        ctx = njs_object_data(this);
+        if (njs_slow_path(ctx->alg == NULL)) {
+            njs_error(vm, "Digest already called");
+            return NJS_ERROR;
+        }
+
+        ctx->alg->update(&ctx->u, data.start, data.length);
+    }
 
     vm->retval = *this;
 
@@ -230,71 +276,62 @@ njs_hash_prototype_update(njs_vm_t *vm, 
 
 static njs_int_t
 njs_hash_prototype_digest(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
-    njs_index_t unused)
+    njs_index_t tag)
 {
-    u_char            digest[32], *p;
-    njs_int_t         ret;
-    njs_str_t         enc_name, str;
+    njs_str_t         str;
+    njs_hmac_t        *ctx;
     njs_value_t       *this;
     njs_digest_t      *dgst;
     njs_hash_alg_t    *alg;
     njs_crypto_enc_t  *enc;
-
-    if (njs_slow_path(nargs > 1 && !njs_is_string(&args[1]))) {
-        njs_type_error(vm, "encoding must be a string");
-        return NJS_ERROR;
-    }
+    u_char            hash1[32], digest[32];
 
     this = njs_argument(args, 0);
-
-    if (njs_slow_path(!njs_is_object_data(this, NJS_DATA_TAG_CRYPTO_HASH))) {
+    if (njs_slow_path(!njs_is_object_data(this, tag))) {
         njs_type_error(vm, "\"this\" is not a hash object");
         return NJS_ERROR;
     }
 
-    enc = NULL;
-
-    if (nargs > 1) {
-        njs_string_get(&args[1], &enc_name);
-
-        enc = njs_crypto_encoding(vm, &enc_name);
-        if (njs_slow_path(enc == NULL)) {
-            return NJS_ERROR;
-        }
-    }
-
-    dgst = njs_object_data(this);
-
-    if (njs_slow_path(dgst->alg == NULL)) {
-        njs_error(vm, "Digest already called");
+    enc = njs_crypto_encoding(vm, njs_arg(args, nargs, 1));
+    if (njs_slow_path(enc == NULL)) {
         return NJS_ERROR;
     }
 
-    alg = dgst->alg;
+    if (tag == NJS_DATA_TAG_CRYPTO_HASH) {
+        dgst = njs_object_data(this);
+        if (njs_slow_path(dgst->alg == NULL)) {
+            goto exception;
+        }
+
+        alg = dgst->alg;
+        alg->final(digest, &dgst->u);
+        dgst->alg = NULL;
 
-    alg->final(digest, &dgst->u);
+    } else {
+        ctx = njs_object_data(this);
+        if (njs_slow_path(ctx->alg == NULL)) {
+            goto exception;
+        }
+
+        alg = ctx->alg;
+        alg->final(hash1, &ctx->u);
+
+        alg->init(&ctx->u);
+        alg->update(&ctx->u, ctx->opad, 64);
+        alg->update(&ctx->u, hash1, alg->size);
+        alg->final(digest, &ctx->u);
+        ctx->alg = NULL;
+    }
 
     str.start = digest;
     str.length = alg->size;
 
-    if (enc == NULL) {
-        p = njs_string_alloc(vm, &vm->retval, str.length, 0);
-
-        if (njs_fast_path(p != NULL)) {
-            memcpy(p, str.start, str.length);
-            ret = NJS_OK;
+    return enc->encode(vm, &vm->retval, &str);
 
-        } else {
-            ret = NJS_ERROR;
-        }
+exception:
 
-    } else {
-        ret = enc->encode(vm, &vm->retval, &str);
-    }
-
-    dgst->alg = NULL;
-
-    return ret;
+    njs_error(vm, "Digest already called");
+    return NJS_ERROR;
 }
 
 
@@ -317,7 +354,8 @@ static const njs_object_prop_t  njs_hash
     {
         .type = NJS_PROPERTY,
         .name = njs_string("update"),
-        .value = njs_native_function(njs_hash_prototype_update, 0),
+        .value = njs_native_function2(njs_hash_prototype_update, 0,
+                                      NJS_DATA_TAG_CRYPTO_HASH),
         .writable = 1,
         .configurable = 1,
     },
@@ -325,7 +363,8 @@ static const njs_object_prop_t  njs_hash
     {
         .type = NJS_PROPERTY,
         .name = njs_string("digest"),
-        .value = njs_native_function(njs_hash_prototype_digest, 0),
+        .value = njs_native_function2(njs_hash_prototype_digest, 0,
+                                      NJS_DATA_TAG_CRYPTO_HASH),
         .writable = 1,
         .configurable = 1,
     },
@@ -397,31 +436,47 @@ static njs_int_t
 njs_crypto_create_hmac(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t unused)
 {
-    u_char              digest[32], key_buf[64];
-    njs_str_t           alg_name, key;
+    njs_str_t           key;
     njs_uint_t          i;
     njs_hmac_t          *ctx;
     njs_hash_alg_t      *alg;
+    njs_typed_array_t   *array;
+    const njs_value_t   *value;
+    njs_array_buffer_t  *buffer;
     njs_object_value_t  *hmac;
-
-    if (njs_slow_path(nargs < 2 || !njs_is_string(&args[1]))) {
-        njs_type_error(vm, "algorithm must be a string");
-        return NJS_ERROR;
-    }
+    u_char              digest[32], key_buf[64];
 
-    if (njs_slow_path(nargs < 3 || !njs_is_string(&args[2]))) {
-        njs_type_error(vm, "key must be a string");
-        return NJS_ERROR;
-    }
-
-    njs_string_get(&args[1], &alg_name);
-
-    alg = njs_crypto_alg(vm, &alg_name);
+    alg = njs_crypto_algorithm(vm, njs_arg(args, nargs, 1));
     if (njs_slow_path(alg == NULL)) {
         return NJS_ERROR;
     }
 
-    njs_string_get(&args[2], &key);
+    value = njs_arg(args, nargs, 2);
+
+    switch (value->type) {
+    case NJS_STRING:
+        njs_string_get(value, &key);
+        break;
+
+    case NJS_TYPED_ARRAY:
+    case NJS_DATA_VIEW:
+        array = njs_typed_array(value);
+        buffer = array->buffer;
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
+        key.start = &buffer->u.u8[array->offset];
+        key.length = array->byte_length;
+        break;
+
+    default:
+        njs_type_error(vm, "key argument \"%s\" is not a string "
+                       "or Buffer-like object", njs_type_string(value->type));
+
+        return NJS_ERROR;
+    }
 
     ctx = njs_mp_alloc(vm->mem_pool, sizeof(njs_hmac_t));
     if (njs_slow_path(ctx == NULL)) {
@@ -468,118 +523,6 @@ njs_crypto_create_hmac(njs_vm_t *vm, njs
 }
 
 
-static njs_int_t
-njs_hmac_prototype_update(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
-    njs_index_t unused)
-{
-    njs_str_t    data;
-    njs_hmac_t   *ctx;
-    njs_value_t  *this;
-
-    if (njs_slow_path(nargs < 2 || !njs_is_string(&args[1]))) {
-        njs_type_error(vm, "data must be a string");
-        return NJS_ERROR;
-    }
-
-    this = njs_argument(args, 0);
-
-    if (njs_slow_path(!njs_is_object_data(this, NJS_DATA_TAG_CRYPTO_HMAC))) {
-        njs_type_error(vm, "\"this\" is not a hash object");
-        return NJS_ERROR;
-    }
-
-    njs_string_get(&args[1], &data);
-
-    ctx = njs_object_data(this);
-
-    if (njs_slow_path(ctx->alg == NULL)) {
-        njs_error(vm, "Digest already called");
-        return NJS_ERROR;
-    }
-
-    ctx->alg->update(&ctx->u, data.start, data.length);
-
-    vm->retval = *this;
-
-    return NJS_OK;
-}
-
-
-static njs_int_t
-njs_hmac_prototype_digest(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
-    njs_index_t unused)
-{
-    u_char            hash1[32], digest[32], *p;
-    njs_str_t         enc_name, str;
-    njs_int_t         ret;
-    njs_hmac_t        *ctx;
-    njs_value_t       *this;
-    njs_hash_alg_t    *alg;
-    njs_crypto_enc_t  *enc;
-
-    if (njs_slow_path(nargs > 1 && !njs_is_string(&args[1]))) {
-        njs_type_error(vm, "encoding must be a string");
-        return NJS_ERROR;
-    }
-
-    this = njs_argument(args, 0);
-
-    if (njs_slow_path(!njs_is_object_data(this, NJS_DATA_TAG_CRYPTO_HMAC))) {
-        njs_type_error(vm, "\"this\" is not a hash object");
-        return NJS_ERROR;
-    }
-
-    enc = NULL;
-
-    if (nargs > 1) {
-        njs_string_get(&args[1], &enc_name);
-
-        enc = njs_crypto_encoding(vm, &enc_name);
-        if (njs_slow_path(enc == NULL)) {
-            return NJS_ERROR;
-        }
-    }
-
-    ctx = njs_object_data(this);
-
-    if (njs_slow_path(ctx->alg == NULL)) {
-        njs_error(vm, "Digest already called");
-        return NJS_ERROR;
-    }
-
-    alg = ctx->alg;
-
-    alg->final(hash1, &ctx->u);
-
-    alg->init(&ctx->u);
-    alg->update(&ctx->u, ctx->opad, 64);
-    alg->update(&ctx->u, hash1, alg->size);
-    alg->final(digest, &ctx->u);
-
-    str.start = digest;
-    str.length = alg->size;
-
-    if (enc == NULL) {
-        p = njs_string_alloc(vm, &vm->retval, str.length, 0);
-
-        if (njs_fast_path(p != NULL)) {
-            memcpy(p, str.start, str.length);
-            ret = NJS_OK;
-
-        } else {
-            ret = NJS_ERROR;
-        }
-
-    } else {
-        ret = enc->encode(vm, &vm->retval, &str);
-    }
-
-    ctx->alg = NULL;
-
-    return ret;
-}
-
-
 static const njs_object_prop_t  njs_hmac_prototype_properties[] =
 {
     {
@@ -599,7 +542,8 @@ static const njs_object_prop_t  njs_hmac
     {
         .type = NJS_PROPERTY,
         .name = njs_string("update"),
-        .value = njs_native_function(njs_hmac_prototype_update, 0),
+        .value = njs_native_function2(njs_hash_prototype_update, 0,
+                                      NJS_DATA_TAG_CRYPTO_HMAC),
         .writable = 1,
         .configurable = 1,
     },
@@ -607,7 +551,8 @@ static const njs_object_prop_t  njs_hmac
     {
         .type = NJS_PROPERTY,
         .name = njs_string("digest"),
-        .value = njs_native_function(njs_hmac_prototype_digest, 0),
+        .value = njs_native_function2(njs_hash_prototype_digest, 0,
+                                      NJS_DATA_TAG_CRYPTO_HMAC),
         .writable = 1,
         .configurable = 1,
     },
@@ -716,34 +661,61 @@ const njs_object_type_init_t  njs_hmac_t
 
 
 static njs_hash_alg_t *
-njs_crypto_alg(njs_vm_t *vm, const njs_str_t *name)
+njs_crypto_algorithm(njs_vm_t *vm, const njs_value_t *value)
 {
-    njs_hash_alg_t *e;
+    njs_str_t       name;
+    njs_hash_alg_t  *e;
+
+    if (njs_slow_path(!njs_is_string(value))) {
+        njs_type_error(vm, "algorithm must be a string");
+        return NULL;
+    }
+
+    njs_string_get(value, &name);
 

From mdounin at mdounin.ru  Wed Sep 23 18:36:56 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 23 Sep 2020 18:36:56 +0000
Subject: [nginx] HTTP/2: fixed segfault on DATA frames after 400 errors.
Message-ID: <hg.097f578a4a8f.1600886216.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/097f578a4a8f
branches:  
changeset: 7710:097f578a4a8f
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Sep 23 19:50:49 2020 +0300
description:
HTTP/2: fixed segfault on DATA frames after 400 errors.

If 400 errors were redirected to an upstream server using the error_page
directive, DATA frames from the client might cause segmentation fault
due to null pointer dereference.  The bug had appeared in 6989:2c4dbcd6f2e4
(1.13.0).

Fix is to skip such frames in ngx_http_v2_state_read_data() (similarly
to 7561:9f1f9d6e056a).  With the fix, behaviour of 400 errors in HTTP/2
is now similar to one in HTTP/1.x, that is, nginx doesn't try to read the
request body.

Note that proxying 400 errors, as well as other early stage errors, to
upstream servers might not be a good idea anyway.  These errors imply
that reading and processing of the request (and the request headers)
wasn't complete, and proxying of such incomplete request might lead to
various errors.

Reported by Chenglong Zhang.

diffstat:

 src/http/v2/ngx_http_v2.c |  7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diffs (17 lines):

diff -r 052ecc68d350 -r 097f578a4a8f src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Wed Sep 16 18:26:25 2020 +0300
+++ b/src/http/v2/ngx_http_v2.c	Wed Sep 23 19:50:49 2020 +0300
@@ -1084,6 +1084,13 @@ ngx_http_v2_state_read_data(ngx_http_v2_
         return ngx_http_v2_state_skip_padded(h2c, pos, end);
     }
 
+    if (r->headers_in.content_length_n < 0 && !r->headers_in.chunked) {
+        ngx_log_debug0(NGX_LOG_DEBUG_HTTP, h2c->connection->log, 0,
+                       "skipping http2 DATA frame");
+
+        return ngx_http_v2_state_skip_padded(h2c, pos, end);
+    }
+
     size = end - pos;
 
     if (size >= h2c->state.length) {

From mdounin at mdounin.ru  Wed Sep 23 18:37:00 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 23 Sep 2020 18:37:00 +0000
Subject: [nginx] HTTP/2: run posted requests after reading body.
Message-ID: <hg.526dddf637bb.1600886220.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/526dddf637bb
branches:  
changeset: 7711:526dddf637bb
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Sep 23 19:52:31 2020 +0300
description:
HTTP/2: run posted requests after reading body.

HTTP/2 code failed to run posted requests after calling the request body
handler, and this resulted in connection hang if a subrequest was created
in the body handler and no other actions were made.

diffstat:

 src/http/v2/ngx_http_v2.c |  4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diffs (28 lines):

diff -r 097f578a4a8f -r 526dddf637bb src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Wed Sep 23 19:50:49 2020 +0300
+++ b/src/http/v2/ngx_http_v2.c	Wed Sep 23 19:52:31 2020 +0300
@@ -1058,6 +1058,7 @@ ngx_http_v2_state_read_data(ngx_http_v2_
     size_t                   size;
     ngx_buf_t               *buf;
     ngx_int_t                rc;
+    ngx_connection_t        *fc;
     ngx_http_request_t      *r;
     ngx_http_v2_stream_t    *stream;
     ngx_http_v2_srv_conf_t  *h2scf;
@@ -1076,6 +1077,7 @@ ngx_http_v2_state_read_data(ngx_http_v2_
     }
 
     r = stream->request;
+    fc = r->connection;
 
     if (r->reading_body && !r->request_body_no_buffering) {
         ngx_log_debug0(NGX_LOG_DEBUG_HTTP, h2c->connection->log, 0,
@@ -1108,6 +1110,8 @@ ngx_http_v2_state_read_data(ngx_http_v2_
             ngx_http_finalize_request(r, rc);
         }
 
+        ngx_http_run_posted_requests(fc);
+
     } else if (size) {
         buf = stream->preread;
 

From qiao.liu at intel.com  Thu Sep 24 00:58:45 2020
From: qiao.liu at intel.com (Liu, Qiao)
Date: Thu, 24 Sep 2020 00:58:45 +0000
Subject: [PATCH] Use BPF to distribute packet to different work thread.
In-Reply-To: <SN6PR11MB3517E97F77028B2A11C3B9EF87200@SN6PR11MB3517.namprd11.prod.outlook.com>
References: <SN6PR11MB35172BC6F818934BACB7FA3A87270@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200910101132.GA38727@vl.krasnogorsk.ru>
 <SN6PR11MB351752D836F2B1659E8FF67787240@SN6PR11MB3517.namprd11.prod.outlook.com>
 <20200913233936.GV18881@mdounin.ru>
 <SN6PR11MB3517F8A7E457669E49D3483C87230@SN6PR11MB3517.namprd11.prod.outlook.com>
 <SN6PR11MB3517E97F77028B2A11C3B9EF87200@SN6PR11MB3517.namprd11.prod.outlook.com>
Message-ID: <SN6PR11MB351734F2EEDC51AB02E12EBF87390@SN6PR11MB3517.namprd11.prod.outlook.com>

Remove printf 

# HG changeset patch
# User Liu Qiao <qiao.liu at intel.com>
# Date 1599735293 14400
#      Thu Sep 10 06:54:53 2020 -0400
# Node ID c2eabe9168d0cbefc030807a0808568d86c93e4f
# Parent  da5e3f5b16733167142b599b6af3ce9469a07d52
Use BPF to distribute packet to different work thread.
Use Berkeley Packet Filter to get packet queue_mapping number,
and use this queue_mapping number to distribute the packet to
different work thread, this will improve CPU utilization and http
latency.
Author: Samudrala, Sridhar <sridhar.samudrala at intel.com>

diff -r da5e3f5b1673 -r c2eabe9168d0 auto/os/linux
--- a/auto/os/linux	Wed Sep 02 23:13:36 2020 +0300
+++ b/auto/os/linux	Thu Sep 10 06:54:53 2020 -0400
@@ -32,6 +32,10 @@
     have=NGX_HAVE_POSIX_FADVISE . auto/nohave
 fi
 
+if [ $version -lt 263680 ]; then
+    have=NGX_HAVE_REUSEPORT_CBPF . auto/nohave
+fi
+
 # epoll, EPOLLET version
 
 ngx_feature="epoll"
diff -r da5e3f5b1673 -r c2eabe9168d0 auto/unix
--- a/auto/unix	Wed Sep 02 23:13:36 2020 +0300
+++ b/auto/unix	Thu Sep 10 06:54:53 2020 -0400
@@ -331,6 +331,17 @@
 ngx_feature_test="setsockopt(0, SOL_SOCKET, SO_REUSEPORT, NULL, 0)"
 . auto/feature
 
+ngx_feature="SO_REUSEPORT_CBPF"
+ngx_feature_name="NGX_HAVE_REUSEPORT_CBPF"
+ngx_feature_run=no
+ngx_feature_incs="#include <sys/socket.h>
+                  #include <linux/filter.h>
+                  #include <error.h>"
+ngx_feature_path=
+ngx_feature_libs=
+ngx_feature_test="setsockopt(0, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, NULL, 0)"
+. auto/feature
+
 
 ngx_feature="SO_ACCEPTFILTER"
 ngx_feature_name="NGX_HAVE_DEFERRED_ACCEPT"
diff -r da5e3f5b1673 -r c2eabe9168d0 src/core/ngx_connection.c
--- a/src/core/ngx_connection.c	Wed Sep 02 23:13:36 2020 +0300
+++ b/src/core/ngx_connection.c	Thu Sep 10 06:54:53 2020 -0400
@@ -8,7 +8,10 @@
 #include <ngx_config.h>
 #include <ngx_core.h>
 #include <ngx_event.h>
-
+#if (NGX_HAVE_REUSEPORT_CBPF)
+#include <linux/filter.h>
+#include <error.h>
+#endif
 
 ngx_os_io_t  ngx_io;
 
@@ -708,6 +711,35 @@
     return NGX_OK;
 }
 
+#if(NGX_HAVE_REUSEPORT)
+#if(NGX_HAVE_REUSEPORT_CBPF)
+#ifndef ARRAY_SIZE
+#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
+#endif
+
+static ngx_int_t  attach_bpf(int fd, uint16_t n)
+{
+	struct sock_filter code[] = {
+		/* A = skb->queue_mapping */
+		{ BPF_LD  | BPF_W | BPF_ABS, 0, 0, SKF_AD_OFF + SKF_AD_QUEUE },
+		/* A = A % n */
+		{ BPF_ALU | BPF_MOD, 0, 0, n },
+		/* return A */
+		{ BPF_RET | BPF_A, 0, 0, 0 },
+	};
+	struct sock_fprog p = {
+		.len = ARRAY_SIZE(code),
+		.filter = code,
+	};
+
+	if (setsockopt(fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, &p, sizeof(p)))
+	    return NGX_ERROR;
+	else
+            return NGX_OK;
+}
+#endif
+#endif
+
 
 void
 ngx_configure_listening_sockets(ngx_cycle_t *cycle)
@@ -719,6 +751,11 @@
 #if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER)
     struct accept_filter_arg   af;
 #endif
+#if (NGX_HAVE_REUSEPORT)
+#if (NGX_HAVE_REUSEPORT_CBPF)
+        ngx_core_conf_t* ccf ;
+#endif
+#endif
 
     ls = cycle->listening.elts;
     for (i = 0; i < cycle->listening.nelts; i++) {
@@ -1011,6 +1048,31 @@
         }
 
 #endif
+#if (NGX_HAVE_REUSEPORT)
+#if (NGX_HAVE_REUSEPORT_CBPF)
+    if(ls[i].reuseport)
+    {
+        ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,ngx_core_module);
+        if(ccf)
+        {
+            if( NGX_OK == attach_bpf(ls[i].fd, ccf->worker_processes) )
+            {
+                ngx_log_error(NGX_LOG_INFO,cycle->log ,ngx_socket_errno,\
+                      "bpf prog attached to fd:%d\n", ls[i].fd);
+            }
+            else
+            {
+                ngx_log_error(NGX_LOG_ERR,cycle->log ,ngx_socket_errno,\
+                      "failed to set SO_ATTACH_REUSEPORT_CBPF\n");
+            }
+        }
+        else
+           ngx_log_error(NGX_LOG_ERR,cycle->log ,ngx_socket_errno,\
+                      "can not get config, attach bpf failed\n");
+        
+    }
+#endif
+#endif
     }
 
     return;

-----Original Message-----
From: Liu, Qiao 
Sent: Tuesday, September 15, 2020 10:09 AM
To: nginx-devel at nginx.org
Subject: RE: [PATCH] Use BPF to distribute packet to different work thread.

Below is 5 times test result compare, 112 threads, 10000 connections, 1M object http request. Seems P99 have great improvement, and Max is also reduced



                                   AVG          Stdev            Max        P99
                  test 1      1.32s        447.09ms     5.48s      2.82s
BPF           test 2      1.39s        513.8ms       9.42s      3.1s
                  test 3      1.4s          341.38ms     5.63s      2.55s
                  test 4      1.41s        407.45ms     6.96s      2.77s
                  test 5      1.29s        644.81ms     9.45s      3.74s
                 Average  1.362s      470.906ms   7.388s    2.996s

NonBPF   test 1      1.48s         916.88ms     9.44s       5.08s
                 test 2      1.43s         658.48ms     9.54s       3.92s
                 test 3      1.41s         650.38ms     8.63s       3.59s
                 test 4      1.29s         1010ms        10s           5.21s
                 test 5      1.31s         875.01ms     9.53s       4.39s
             Average     1.384s        822.15ms    9.428s    4.438s


Thanks
LQ
-----Original Message-----
From: nginx-devel <nginx-devel-bounces at nginx.org> On Behalf Of Liu, Qiao
Sent: Monday, September 14, 2020 9:18 AM
To: nginx-devel at nginx.org
Subject: RE: [PATCH] Use BPF to distribute packet to different work thread.

Hi, Maxim Dounin:
Thanks for your reply, this server is random selected, we just do BPF and no-BPF test, I think the latency based on server configuration, not related with BPF patch, also the NIC of the server is Mellanox, not ADQ capable hardware , we will do more test Thanks LQ

-----Original Message-----
From: nginx-devel <nginx-devel-bounces at nginx.org> On Behalf Of Maxim Dounin
Sent: Monday, September 14, 2020 7:40 AM
To: nginx-devel at nginx.org
Subject: Re: [PATCH] Use BPF to distribute packet to different work thread.

Hello!

On Fri, Sep 11, 2020 at 05:41:47AM +0000, Liu, Qiao wrote:

> Hi, Vladimir Homutov:
> The below is our WRK test result output with BPF enable
> 
>   112 threads and 10000 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   608.23ms  820.71ms  10.00s    87.48%
>     Connect    16.52ms   54.53ms   1.99s    94.73%
>     Delay     153.13ms  182.17ms   2.00s    90.74%
>     Req/Sec   244.79    142.32     1.99k    68.40%
>   Latency Distribution
>   50.00%  293.50ms
>   75.00%  778.33ms
>   90.00%    1.61s
>   99.00%    3.71s
>   99.90%    7.03s
>   99.99%    8.94s
>   Connect Distribution
>   50.00%    1.93ms
>   75.00%    2.85ms
>   90.00%   55.76ms
>   99.00%  229.19ms
>   99.90%  656.79ms
>   99.99%    1.43s
>   Delay Distribution
>   50.00%  110.96ms
>   75.00%  193.67ms
>   90.00%  321.77ms
>   99.00%  959.27ms
>   99.90%    1.57s
>   99.99%    1.91s
> Compared with no BPF but enable reuseport as below
> 
> 112 threads and 10000 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   680.50ms  943.69ms  10.00s    87.18%
>     Connect    58.44ms  238.08ms   2.00s    94.58%
>     Delay     158.84ms  256.28ms   2.00s    90.92%
>     Req/Sec   244.51    151.00     1.41k    69.67%
>   Latency Distribution
>   50.00%  317.61ms
>   75.00%  913.52ms
>   90.00%    1.90s
>   99.00%    4.30s
>   99.90%    6.52s
>   99.99%    8.80s
>   Connect Distribution
>   50.00%    1.88ms
>   75.00%    2.21ms
>   90.00%   55.94ms
>   99.00%    1.45s
>   99.90%    1.95s
>   99.99%    2.00s
>   Delay Distribution
>   50.00%   73.01ms
>   75.00%  190.40ms
>   90.00%  387.01ms
>   99.00%    1.34s
>   99.90%    1.86s
>   99.99%    1.99s
> 
> 
> From the above results, there shows almost 20% percent latency 
> reduction. P99 latency of BPF is 3.71s , but without BPF is 4.3s.

Thank you for the results.

Given that latency stdev is way higher than the average latency, I don't think the "20% percent latency reduction" observed is statistically significant.  Please try running several tests and use ministat(1) to check the results.

Also, the latency values look very high, and request rate very low.  What's on the server side?

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

From pluknet at nginx.com  Thu Sep 24 12:53:50 2020
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Thu, 24 Sep 2020 12:53:50 +0000
Subject: [nginx] SSL: abort handshake on SSL_set_SSL_CTX() errors.
Message-ID: <hg.eb940fe579cf.1600952030.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/eb940fe579cf
branches:  
changeset: 7712:eb940fe579cf
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Thu Sep 24 13:51:29 2020 +0100
description:
SSL: abort handshake on SSL_set_SSL_CTX() errors.

In rare cases, such as memory allocation failure, SSL_set_SSL_CTX() returns
NULL, which could mean that a different SSL configuration has not been set.
Note that this new behaviour seemingly originated in OpenSSL-1.1.0 release.

diffstat:

 src/http/ngx_http_request.c |  5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diffs (15 lines):

diff -r 526dddf637bb -r eb940fe579cf src/http/ngx_http_request.c
--- a/src/http/ngx_http_request.c	Wed Sep 23 19:52:31 2020 +0300
+++ b/src/http/ngx_http_request.c	Thu Sep 24 13:51:29 2020 +0100
@@ -932,7 +932,10 @@ ngx_http_ssl_servername(ngx_ssl_conn_t *
     c->ssl->buffer_size = sscf->buffer_size;
 
     if (sscf->ssl.ctx) {
-        SSL_set_SSL_CTX(ssl_conn, sscf->ssl.ctx);
+        if (SSL_set_SSL_CTX(ssl_conn, sscf->ssl.ctx) == NULL) {
+            *ad = SSL_AD_INTERNAL_ERROR;
+            return SSL_TLSEXT_ERR_ALERT_FATAL;
+        }
 
         /*
          * SSL_set_SSL_CTX() only changes certs as of 1.0.0d

From xeioex at nginx.com  Thu Sep 24 19:03:33 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 24 Sep 2020 19:03:33 +0000
Subject: [njs] Fixed dead store assignment after eef4ab1bee70.
Message-ID: <hg.8498c810d853.1600974213.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/8498c810d853
branches:  
changeset: 1528:8498c810d853
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Thu Sep 24 19:00:38 2020 +0000
description:
Fixed dead store assignment after eef4ab1bee70.

Found by Clang static analyzer.

diffstat:

 src/njs_buffer.c |  49 ++++++++++++++-----------------------------------
 1 files changed, 14 insertions(+), 35 deletions(-)

diffs (108 lines):

diff -r eef4ab1bee70 -r 8498c810d853 src/njs_buffer.c
--- a/src/njs_buffer.c	Tue Sep 22 18:42:21 2020 +0000
+++ b/src/njs_buffer.c	Thu Sep 24 19:00:38 2020 +0000
@@ -55,8 +55,6 @@ static njs_buffer_encoding_t  njs_buffer
     },
 
     { njs_null_str, 0, 0, 0 }
-
-#define njs_buffer_utf8_encoding()  &njs_buffer_encodings[0]
 };
 
 
@@ -517,7 +515,7 @@ njs_buffer_byte_length(njs_vm_t *vm, njs
     njs_index_t unused)
 {
     size_t                       size;
-    njs_value_t                  *value, *enc;
+    njs_value_t                  *value;
     const njs_buffer_encoding_t  *encoding;
 
     value = njs_arg(args, nargs, 1);
@@ -536,14 +534,9 @@ njs_buffer_byte_length(njs_vm_t *vm, njs
         return NJS_OK;
 
     case NJS_STRING:
-        enc = njs_arg(args, nargs, 2);
-        encoding = njs_buffer_utf8_encoding();
-
-        if (njs_is_defined(enc)) {
-            encoding = njs_buffer_encoding(vm, enc);
-            if (njs_slow_path(encoding == NULL)) {
-                return NJS_ERROR;
-            }
+        encoding = njs_buffer_encoding(vm, njs_arg(args, nargs, 2));
+        if (njs_slow_path(encoding == NULL)) {
+            return NJS_ERROR;
         }
 
         size = njs_buffer_decode_string_length(value, encoding);
@@ -1452,7 +1445,6 @@ njs_buffer_prototype_write(njs_vm_t *vm,
 
     offset = 0;
     length = array->byte_length;
-    encoding = njs_buffer_utf8_encoding();
 
     if (njs_slow_path(!njs_is_string(value))) {
         njs_type_error(vm, "first argument must be a string");
@@ -1483,18 +1475,11 @@ njs_buffer_prototype_write(njs_vm_t *vm,
         }
     }
 
-    if (njs_is_defined(enc)) {
-        if (njs_slow_path(!njs_is_string(enc))) {
-            njs_type_error(vm, "\"encoding\" argument must be of type string");
-            return NJS_ERROR;
-        }
-
-    encoding:
-
-        encoding = njs_buffer_encoding(vm, enc);
-        if (njs_slow_path(encoding == NULL)) {
-            return NJS_ERROR;
-        }
+encoding:
+
+    encoding = njs_buffer_encoding(vm, enc);
+    if (njs_slow_path(encoding == NULL)) {
+        return NJS_ERROR;
     }
 
     buffer = njs_typed_array_writable(vm, array);
@@ -1774,7 +1759,6 @@ njs_buffer_prototype_to_string(njs_vm_t 
 
     start = 0;
     end = array->byte_length;
-    encoding = njs_buffer_utf8_encoding();
 
     encoding = njs_buffer_encoding(vm,  njs_arg(args, nargs, 1));
     if (njs_slow_path(encoding == NULL)) {
@@ -1921,8 +1905,6 @@ njs_buffer_prototype_index_of(njs_vm_t *
     njs_array_buffer_t           *buffer;
     const njs_buffer_encoding_t  *encoding;
 
-    encoding = njs_buffer_utf8_encoding();
-
     this = njs_argument(args, 0);
     value = njs_arg(args, nargs, 1);
     value_from = njs_arg(args, nargs, 2);
@@ -1990,14 +1972,11 @@ njs_buffer_prototype_index_of(njs_vm_t *
         }
     }
 
-    if (njs_is_defined(enc)) {
-
-    encoding:
-
-        encoding = njs_buffer_encoding(vm, enc);
-        if (njs_slow_path(encoding == NULL)) {
-            return NJS_ERROR;
-        }
+encoding:
+
+    encoding = njs_buffer_encoding(vm, enc);
+    if (njs_slow_path(encoding == NULL)) {
+        return NJS_ERROR;
     }
 
     buffer = njs_typed_array_buffer(array);

From p.pautov at f5.com  Thu Sep 24 21:22:12 2020
From: p.pautov at f5.com (Pavel Pautov)
Date: Thu, 24 Sep 2020 21:22:12 +0000
Subject: [nginx] Stream: set module.
Message-ID: <hg.45e9281c6c5b.1600982532.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/45e9281c6c5b
branches:  
changeset: 7713:45e9281c6c5b
user:      Pavel Pautov <p.pautov at f5.com>
date:      Fri Aug 28 14:10:54 2020 -0700
description:
Stream: set module.

Adds 'set' directive to the stream server context.

diffstat:

 auto/modules                       |   10 +
 auto/options                       |    3 +
 src/stream/ngx_stream_set_module.c |  226 +++++++++++++++++++++++++++++++++++++
 3 files changed, 239 insertions(+), 0 deletions(-)

diffs (277 lines):

diff -r eb940fe579cf -r 45e9281c6c5b auto/modules
--- a/auto/modules	Thu Sep 24 13:51:29 2020 +0100
+++ b/auto/modules	Fri Aug 28 14:10:54 2020 -0700
@@ -1119,6 +1119,16 @@ if [ $STREAM != NO ]; then
         . auto/module
     fi
 
+    if [ $STREAM_SET = YES ]; then
+        ngx_module_name=ngx_stream_set_module
+        ngx_module_deps=
+        ngx_module_srcs=src/stream/ngx_stream_set_module.c
+        ngx_module_libs=
+        ngx_module_link=$STREAM_SET
+
+        . auto/module
+    fi
+
     if [ $STREAM_UPSTREAM_HASH = YES ]; then
         ngx_module_name=ngx_stream_upstream_hash_module
         ngx_module_deps=
diff -r eb940fe579cf -r 45e9281c6c5b auto/options
--- a/auto/options	Thu Sep 24 13:51:29 2020 +0100
+++ b/auto/options	Fri Aug 28 14:10:54 2020 -0700
@@ -124,6 +124,7 @@ STREAM_GEOIP=NO
 STREAM_MAP=YES
 STREAM_SPLIT_CLIENTS=YES
 STREAM_RETURN=YES
+STREAM_SET=YES
 STREAM_UPSTREAM_HASH=YES
 STREAM_UPSTREAM_LEAST_CONN=YES
 STREAM_UPSTREAM_RANDOM=YES
@@ -324,6 +325,7 @@ use the \"--with-mail_ssl_module\" optio
         --without-stream_split_clients_module)
                                          STREAM_SPLIT_CLIENTS=NO    ;;
         --without-stream_return_module)  STREAM_RETURN=NO           ;;
+        --without-stream_set_module)     STREAM_SET=NO              ;;
         --without-stream_upstream_hash_module)
                                          STREAM_UPSTREAM_HASH=NO    ;;
         --without-stream_upstream_least_conn_module)
@@ -538,6 +540,7 @@ cat << END
   --without-stream_split_clients_module
                                      disable ngx_stream_split_clients_module
   --without-stream_return_module     disable ngx_stream_return_module
+  --without-stream_set_module        disable ngx_stream_set_module
   --without-stream_upstream_hash_module
                                      disable ngx_stream_upstream_hash_module
   --without-stream_upstream_least_conn_module
diff -r eb940fe579cf -r 45e9281c6c5b src/stream/ngx_stream_set_module.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/src/stream/ngx_stream_set_module.c	Fri Aug 28 14:10:54 2020 -0700
@@ -0,0 +1,226 @@
+
+/*
+ * Copyright (C) Pavel Pautov
+ * Copyright (C) Nginx, Inc.
+ */
+
+
+#include <ngx_config.h>
+#include <ngx_core.h>
+#include <ngx_stream.h>
+
+
+typedef struct {
+    ngx_int_t                   index;
+    ngx_stream_set_variable_pt  set_handler;
+    uintptr_t                   data;
+    ngx_stream_complex_value_t  value;
+} ngx_stream_set_cmd_t;
+
+
+typedef struct {
+    ngx_array_t                 commands;
+} ngx_stream_set_srv_conf_t;
+
+
+static ngx_int_t ngx_stream_set_handler(ngx_stream_session_t *s);
+static ngx_int_t ngx_stream_set_var(ngx_stream_session_t *s,
+    ngx_stream_variable_value_t *v, uintptr_t data);
+static ngx_int_t ngx_stream_set_init(ngx_conf_t *cf);
+static void *ngx_stream_set_create_srv_conf(ngx_conf_t *cf);
+static char *ngx_stream_set(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
+
+
+static ngx_command_t  ngx_stream_set_commands[] = {
+
+    { ngx_string("set"),
+      NGX_STREAM_SRV_CONF|NGX_CONF_TAKE2,
+      ngx_stream_set,
+      NGX_STREAM_SRV_CONF_OFFSET,
+      0,
+      NULL },
+
+      ngx_null_command
+};
+
+
+static ngx_stream_module_t  ngx_stream_set_module_ctx = {
+    NULL,                                  /* preconfiguration */
+    ngx_stream_set_init,                   /* postconfiguration */
+
+    NULL,                                  /* create main configuration */
+    NULL,                                  /* init main configuration */
+
+    ngx_stream_set_create_srv_conf,        /* create server configuration */
+    NULL                                   /* merge server configuration */
+};
+
+
+ngx_module_t  ngx_stream_set_module = {
+    NGX_MODULE_V1,
+    &ngx_stream_set_module_ctx,            /* module context */
+    ngx_stream_set_commands,               /* module directives */
+    NGX_STREAM_MODULE,                     /* module type */
+    NULL,                                  /* init master */
+    NULL,                                  /* init module */
+    NULL,                                  /* init process */
+    NULL,                                  /* init thread */
+    NULL,                                  /* exit thread */
+    NULL,                                  /* exit process */
+    NULL,                                  /* exit master */
+    NGX_MODULE_V1_PADDING
+};
+
+
+static ngx_int_t
+ngx_stream_set_handler(ngx_stream_session_t *s)
+{
+    ngx_str_t                     str;
+    ngx_uint_t                    i;
+    ngx_stream_set_cmd_t         *cmds;
+    ngx_stream_set_srv_conf_t    *scf;
+    ngx_stream_variable_value_t   vv;
+
+    scf = ngx_stream_get_module_srv_conf(s, ngx_stream_set_module);
+    cmds = scf->commands.elts;
+    vv = ngx_stream_variable_null_value;
+
+    for (i = 0; i < scf->commands.nelts; i++) {
+        if (ngx_stream_complex_value(s, &cmds[i].value, &str) != NGX_OK) {
+            return NGX_ERROR;
+        }
+
+        if (cmds[i].set_handler != NULL) {
+            vv.len = str.len;
+            vv.data = str.data;
+            cmds[i].set_handler(s, &vv, cmds[i].data);
+
+        } else {
+            s->variables[cmds[i].index].len = str.len;
+            s->variables[cmds[i].index].valid = 1;
+            s->variables[cmds[i].index].no_cacheable = 0;
+            s->variables[cmds[i].index].not_found = 0;
+            s->variables[cmds[i].index].data = str.data;
+        }
+    }
+
+    return NGX_DECLINED;
+}
+
+
+static ngx_int_t
+ngx_stream_set_var(ngx_stream_session_t *s, ngx_stream_variable_value_t *v,
+    uintptr_t data)
+{
+    *v = ngx_stream_variable_null_value;
+
+    return NGX_OK;
+}
+
+
+static ngx_int_t
+ngx_stream_set_init(ngx_conf_t *cf)
+{
+    ngx_stream_handler_pt        *h;
+    ngx_stream_core_main_conf_t  *cmcf;
+
+    cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module);
+
+    h = ngx_array_push(&cmcf->phases[NGX_STREAM_PREACCESS_PHASE].handlers);
+    if (h == NULL) {
+        return NGX_ERROR;
+    }
+
+    *h = ngx_stream_set_handler;
+
+    return NGX_OK;
+}
+
+
+static void *
+ngx_stream_set_create_srv_conf(ngx_conf_t *cf)
+{
+    ngx_stream_set_srv_conf_t  *conf;
+
+    conf = ngx_pcalloc(cf->pool, sizeof(ngx_stream_set_srv_conf_t));
+    if (conf == NULL) {
+        return NULL;
+    }
+
+    /*
+     * set by ngx_pcalloc():
+     *
+     *     conf->commands = { NULL };
+     */
+
+    return conf;
+}
+
+
+static char *
+ngx_stream_set(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
+{
+    ngx_stream_set_srv_conf_t  *scf = conf;
+
+    ngx_str_t                           *args;
+    ngx_int_t                            index;
+    ngx_stream_set_cmd_t                *set_cmd;
+    ngx_stream_variable_t               *v;
+    ngx_stream_compile_complex_value_t   ccv;
+
+    args = cf->args->elts;
+
+    if (args[1].data[0] != '$') {
+        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                           "invalid variable name \"%V\"", &args[1]);
+        return NGX_CONF_ERROR;
+    }
+
+    args[1].len--;
+    args[1].data++;
+
+    v = ngx_stream_add_variable(cf, &args[1],
+                                NGX_STREAM_VAR_CHANGEABLE|NGX_STREAM_VAR_WEAK);
+    if (v == NULL) {
+        return NGX_CONF_ERROR;
+    }
+
+    index = ngx_stream_get_variable_index(cf, &args[1]);
+    if (index == NGX_ERROR) {
+        return NGX_CONF_ERROR;
+    }
+
+    if (v->get_handler == NULL) {
+        v->get_handler = ngx_stream_set_var;
+    }
+
+    if (scf->commands.elts == NULL) {
+        if (ngx_array_init(&scf->commands, cf->pool, 1,
+                           sizeof(ngx_stream_set_cmd_t))
+            != NGX_OK)
+        {
+            return NGX_CONF_ERROR;
+        }
+    }
+
+    set_cmd = ngx_array_push(&scf->commands);
+    if (set_cmd == NULL) {
+        return NGX_CONF_ERROR;
+    }
+
+    set_cmd->index = index;
+    set_cmd->set_handler = v->set_handler;
+    set_cmd->data = v->data;
+
+    ngx_memzero(&ccv, sizeof(ngx_stream_compile_complex_value_t));
+
+    ccv.cf = cf;
+    ccv.value = &args[2];
+    ccv.complex_value = &set_cmd->value;
+
+    if (ngx_stream_compile_complex_value(&ccv) != NGX_OK) {
+        return NGX_CONF_ERROR;
+    }
+
+    return NGX_CONF_OK;
+}

From ru at nginx.com  Mon Sep 28 09:43:43 2020
From: ru at nginx.com (Ruslan Ermilov)
Date: Mon, 28 Sep 2020 09:43:43 +0000
Subject: [nginx] Proxy: strengthen syntax checking for some directives.
Message-ID: <hg.8dbc9ee97998.1601286223.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/8dbc9ee97998
branches:  
changeset: 7714:8dbc9ee97998
user:      Ruslan Ermilov <ru at nginx.com>
date:      Sun Sep 27 23:21:09 2020 +0300
description:
Proxy: strengthen syntax checking for some directives.

The "false" parameter of the proxy_redirect directive is deprecated.
Warning has been emitted since c2230102df6f (0.7.54).

The "off" parameter of the proxy_redirect, proxy_cookie_domain, and
proxy_cookie_path directives tells nginx not to inherit the
configuration from the previous configuration level.

Previously, after specifying the directive with the "off" parameter,
any other directives were ignored, and syntax checking was disabled.

The syntax was enforced to allow either one directive with the "off"
parameter, or several directives with other parameters.

Also, specifying "proxy_redirect default foo" no longer works like
"proxy_redirect default".

diffstat:

 src/http/modules/ngx_http_proxy_module.c |  34 +++++++++++++++++++------------
 1 files changed, 21 insertions(+), 13 deletions(-)

diffs (87 lines):

diff -r 45e9281c6c5b -r 8dbc9ee97998 src/http/modules/ngx_http_proxy_module.c
--- a/src/http/modules/ngx_http_proxy_module.c	Fri Aug 28 14:10:54 2020 -0700
+++ b/src/http/modules/ngx_http_proxy_module.c	Sun Sep 27 23:21:09 2020 +0300
@@ -3766,7 +3766,7 @@ ngx_http_proxy_redirect(ngx_conf_t *cf, 
     ngx_http_compile_complex_value_t   ccv;
 
     if (plcf->redirect == 0) {
-        return NGX_CONF_OK;
+        return "is duplicate";
     }
 
     plcf->redirect = 1;
@@ -3775,16 +3775,12 @@ ngx_http_proxy_redirect(ngx_conf_t *cf, 
 
     if (cf->args->nelts == 2) {
         if (ngx_strcmp(value[1].data, "off") == 0) {
+
+            if (plcf->redirects) {
+                return "is duplicate";
+            }
+
             plcf->redirect = 0;
-            plcf->redirects = NULL;
-            return NGX_CONF_OK;
-        }
-
-        if (ngx_strcmp(value[1].data, "false") == 0) {
-            ngx_conf_log_error(NGX_LOG_ERR, cf, 0,
-                           "invalid parameter \"false\", use \"off\" instead");
-            plcf->redirect = 0;
-            plcf->redirects = NULL;
             return NGX_CONF_OK;
         }
 
@@ -3808,7 +3804,9 @@ ngx_http_proxy_redirect(ngx_conf_t *cf, 
         return NGX_CONF_ERROR;
     }
 
-    if (ngx_strcmp(value[1].data, "default") == 0) {
+    if (cf->args->nelts == 2
+        && ngx_strcmp(value[1].data, "default") == 0)
+    {
         if (plcf->proxy_lengths) {
             ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                "\"proxy_redirect default\" cannot be used "
@@ -3911,7 +3909,7 @@ ngx_http_proxy_cookie_domain(ngx_conf_t 
     ngx_http_compile_complex_value_t   ccv;
 
     if (plcf->cookie_domains == NULL) {
-        return NGX_CONF_OK;
+        return "is duplicate";
     }
 
     value = cf->args->elts;
@@ -3919,6 +3917,11 @@ ngx_http_proxy_cookie_domain(ngx_conf_t 
     if (cf->args->nelts == 2) {
 
         if (ngx_strcmp(value[1].data, "off") == 0) {
+
+            if (plcf->cookie_domains != NGX_CONF_UNSET_PTR) {
+                return "is duplicate";
+            }
+
             plcf->cookie_domains = NULL;
             return NGX_CONF_OK;
         }
@@ -3998,7 +4001,7 @@ ngx_http_proxy_cookie_path(ngx_conf_t *c
     ngx_http_compile_complex_value_t   ccv;
 
     if (plcf->cookie_paths == NULL) {
-        return NGX_CONF_OK;
+        return "is duplicate";
     }
 
     value = cf->args->elts;
@@ -4006,6 +4009,11 @@ ngx_http_proxy_cookie_path(ngx_conf_t *c
     if (cf->args->nelts == 2) {
 
         if (ngx_strcmp(value[1].data, "off") == 0) {
+
+            if (plcf->cookie_paths != NGX_CONF_UNSET_PTR) {
+                return "is duplicate";
+            }
+
             plcf->cookie_paths = NULL;
             return NGX_CONF_OK;
         }

From ru at nginx.com  Mon Sep 28 09:43:46 2020
From: ru at nginx.com (Ruslan Ermilov)
Date: Mon, 28 Sep 2020 09:43:46 +0000
Subject: [nginx] Proxy: changed interface of some internal functions.
Message-ID: <hg.5c7917292b29.1601286226.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/5c7917292b29
branches:  
changeset: 7715:5c7917292b29
user:      Ruslan Ermilov <ru at nginx.com>
date:      Sun Sep 27 23:21:10 2020 +0300
description:
Proxy: changed interface of some internal functions.

This is in preparation for the next change.

Also, moved optimization from ngx_http_proxy_rewrite_regex_handler()
to ngx_http_proxy_rewrite().

diffstat:

 src/http/modules/ngx_http_proxy_module.c |  66 +++++++++++++++----------------
 1 files changed, 32 insertions(+), 34 deletions(-)

diffs (178 lines):

diff -r 8dbc9ee97998 -r 5c7917292b29 src/http/modules/ngx_http_proxy_module.c
--- a/src/http/modules/ngx_http_proxy_module.c	Sun Sep 27 23:21:09 2020 +0300
+++ b/src/http/modules/ngx_http_proxy_module.c	Sun Sep 27 23:21:10 2020 +0300
@@ -18,7 +18,7 @@ typedef struct {
 typedef struct ngx_http_proxy_rewrite_s  ngx_http_proxy_rewrite_t;
 
 typedef ngx_int_t (*ngx_http_proxy_rewrite_pt)(ngx_http_request_t *r,
-    ngx_table_elt_t *h, size_t prefix, size_t len,
+    ngx_str_t *value, size_t prefix, size_t len,
     ngx_http_proxy_rewrite_t *pr);
 
 struct ngx_http_proxy_rewrite_s {
@@ -161,7 +161,7 @@ static ngx_int_t ngx_http_proxy_rewrite_
 static ngx_int_t ngx_http_proxy_rewrite_cookie_value(ngx_http_request_t *r,
     ngx_table_elt_t *h, u_char *value, ngx_array_t *rewrites);
 static ngx_int_t ngx_http_proxy_rewrite(ngx_http_request_t *r,
-    ngx_table_elt_t *h, size_t prefix, size_t len, ngx_str_t *replacement);
+    ngx_str_t *value, size_t prefix, size_t len, ngx_str_t *replacement);
 
 static ngx_int_t ngx_http_proxy_add_variables(ngx_conf_t *cf);
 static void *ngx_http_proxy_create_main_conf(ngx_conf_t *cf);
@@ -2584,7 +2584,7 @@ ngx_http_proxy_rewrite_redirect(ngx_http
     len = h->value.len - prefix;
 
     for (i = 0; i < plcf->redirects->nelts; i++) {
-        rc = pr[i].handler(r, h, prefix, len, &pr[i]);
+        rc = pr[i].handler(r, &h->value, prefix, len, &pr[i]);
 
         if (rc != NGX_DECLINED) {
             return rc;
@@ -2669,7 +2669,7 @@ ngx_http_proxy_rewrite_cookie_value(ngx_
     pr = rewrites->elts;
 
     for (i = 0; i < rewrites->nelts; i++) {
-        rc = pr[i].handler(r, h, prefix, len, &pr[i]);
+        rc = pr[i].handler(r, &h->value, prefix, len, &pr[i]);
 
         if (rc != NGX_DECLINED) {
             return rc;
@@ -2681,8 +2681,8 @@ ngx_http_proxy_rewrite_cookie_value(ngx_
 
 
 static ngx_int_t
-ngx_http_proxy_rewrite_complex_handler(ngx_http_request_t *r,
-    ngx_table_elt_t *h, size_t prefix, size_t len, ngx_http_proxy_rewrite_t *pr)
+ngx_http_proxy_rewrite_complex_handler(ngx_http_request_t *r, ngx_str_t *value,
+    size_t prefix, size_t len, ngx_http_proxy_rewrite_t *pr)
 {
     ngx_str_t  pattern, replacement;
 
@@ -2691,8 +2691,7 @@ ngx_http_proxy_rewrite_complex_handler(n
     }
 
     if (pattern.len > len
-        || ngx_rstrncmp(h->value.data + prefix, pattern.data,
-                        pattern.len) != 0)
+        || ngx_rstrncmp(value->data + prefix, pattern.data, pattern.len) != 0)
     {
         return NGX_DECLINED;
     }
@@ -2701,20 +2700,20 @@ ngx_http_proxy_rewrite_complex_handler(n
         return NGX_ERROR;
     }
 
-    return ngx_http_proxy_rewrite(r, h, prefix, pattern.len, &replacement);
+    return ngx_http_proxy_rewrite(r, value, prefix, pattern.len, &replacement);
 }
 
 
 #if (NGX_PCRE)
 
 static ngx_int_t
-ngx_http_proxy_rewrite_regex_handler(ngx_http_request_t *r, ngx_table_elt_t *h,
+ngx_http_proxy_rewrite_regex_handler(ngx_http_request_t *r, ngx_str_t *value,
     size_t prefix, size_t len, ngx_http_proxy_rewrite_t *pr)
 {
     ngx_str_t  pattern, replacement;
 
     pattern.len = len;
-    pattern.data = h->value.data + prefix;
+    pattern.data = value->data + prefix;
 
     if (ngx_http_regex_exec(r, pr->pattern.regex, &pattern) != NGX_OK) {
         return NGX_DECLINED;
@@ -2724,20 +2723,15 @@ ngx_http_proxy_rewrite_regex_handler(ngx
         return NGX_ERROR;
     }
 
-    if (prefix == 0 && h->value.len == len) {
-        h->value = replacement;
-        return NGX_OK;
-    }
-
-    return ngx_http_proxy_rewrite(r, h, prefix, len, &replacement);
+    return ngx_http_proxy_rewrite(r, value, prefix, len, &replacement);
 }
 
 #endif
 
 
 static ngx_int_t
-ngx_http_proxy_rewrite_domain_handler(ngx_http_request_t *r,
-    ngx_table_elt_t *h, size_t prefix, size_t len, ngx_http_proxy_rewrite_t *pr)
+ngx_http_proxy_rewrite_domain_handler(ngx_http_request_t *r, ngx_str_t *value,
+    size_t prefix, size_t len, ngx_http_proxy_rewrite_t *pr)
 {
     u_char     *p;
     ngx_str_t   pattern, replacement;
@@ -2746,7 +2740,7 @@ ngx_http_proxy_rewrite_domain_handler(ng
         return NGX_ERROR;
     }
 
-    p = h->value.data + prefix;
+    p = value->data + prefix;
 
     if (p[0] == '.') {
         p++;
@@ -2762,18 +2756,23 @@ ngx_http_proxy_rewrite_domain_handler(ng
         return NGX_ERROR;
     }
 
-    return ngx_http_proxy_rewrite(r, h, prefix, len, &replacement);
+    return ngx_http_proxy_rewrite(r, value, prefix, len, &replacement);
 }
 
 
 static ngx_int_t
-ngx_http_proxy_rewrite(ngx_http_request_t *r, ngx_table_elt_t *h, size_t prefix,
+ngx_http_proxy_rewrite(ngx_http_request_t *r, ngx_str_t *value, size_t prefix,
     size_t len, ngx_str_t *replacement)
 {
     u_char  *p, *data;
     size_t   new_len;
 
-    new_len = replacement->len + h->value.len - len;
+    if (len == value->len) {
+        *value = *replacement;
+        return NGX_OK;
+    }
+
+    new_len = replacement->len + value->len - len;
 
     if (replacement->len > len) {
 
@@ -2782,23 +2781,22 @@ ngx_http_proxy_rewrite(ngx_http_request_
             return NGX_ERROR;
         }
 
-        p = ngx_copy(data, h->value.data, prefix);
+        p = ngx_copy(data, value->data, prefix);
         p = ngx_copy(p, replacement->data, replacement->len);
 
-        ngx_memcpy(p, h->value.data + prefix + len,
-                   h->value.len - len - prefix + 1);
-
-        h->value.data = data;
+        ngx_memcpy(p, value->data + prefix + len,
+                   value->len - len - prefix + 1);
+
+        value->data = data;
 
     } else {
-        p = ngx_copy(h->value.data + prefix, replacement->data,
-                     replacement->len);
-
-        ngx_memmove(p, h->value.data + prefix + len,
-                    h->value.len - len - prefix + 1);
+        p = ngx_copy(value->data + prefix, replacement->data, replacement->len);
+
+        ngx_memmove(p, value->data + prefix + len,
+                    value->len - len - prefix + 1);
     }
 
-    h->value.len = new_len;
+    value->len = new_len;
 
     return NGX_OK;
 }

From ru at nginx.com  Mon Sep 28 09:43:49 2020
From: ru at nginx.com (Ruslan Ermilov)
Date: Mon, 28 Sep 2020 09:43:49 +0000
Subject: [nginx] Proxy: added the "proxy_cookie_flags" directive.
Message-ID: <hg.d6a5e14aa3e4.1601286229.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/d6a5e14aa3e4
branches:  
changeset: 7716:d6a5e14aa3e4
user:      Ruslan Ermilov <ru at nginx.com>
date:      Sun Sep 27 23:21:11 2020 +0300
description:
Proxy: added the "proxy_cookie_flags" directive.

diffstat:

 src/http/modules/ngx_http_proxy_module.c |  586 +++++++++++++++++++++++++++++-
 1 files changed, 554 insertions(+), 32 deletions(-)

diffs (725 lines):

diff -r 5c7917292b29 -r d6a5e14aa3e4 src/http/modules/ngx_http_proxy_module.c
--- a/src/http/modules/ngx_http_proxy_module.c	Sun Sep 27 23:21:10 2020 +0300
+++ b/src/http/modules/ngx_http_proxy_module.c	Sun Sep 27 23:21:11 2020 +0300
@@ -10,6 +10,19 @@
 #include <ngx_http.h>
 
 
+#define  NGX_HTTP_PROXY_COOKIE_SECURE           0x0001
+#define  NGX_HTTP_PROXY_COOKIE_SECURE_ON        0x0002
+#define  NGX_HTTP_PROXY_COOKIE_SECURE_OFF       0x0004
+#define  NGX_HTTP_PROXY_COOKIE_HTTPONLY         0x0008
+#define  NGX_HTTP_PROXY_COOKIE_HTTPONLY_ON      0x0010
+#define  NGX_HTTP_PROXY_COOKIE_HTTPONLY_OFF     0x0020
+#define  NGX_HTTP_PROXY_COOKIE_SAMESITE         0x0040
+#define  NGX_HTTP_PROXY_COOKIE_SAMESITE_STRICT  0x0080
+#define  NGX_HTTP_PROXY_COOKIE_SAMESITE_LAX     0x0100
+#define  NGX_HTTP_PROXY_COOKIE_SAMESITE_NONE    0x0200
+#define  NGX_HTTP_PROXY_COOKIE_SAMESITE_OFF     0x0400
+
+
 typedef struct {
     ngx_array_t                    caches;  /* ngx_http_file_cache_t * */
 } ngx_http_proxy_main_conf_t;
@@ -36,6 +49,19 @@ struct ngx_http_proxy_rewrite_s {
 
 
 typedef struct {
+    union {
+        ngx_http_complex_value_t   complex;
+#if (NGX_PCRE)
+        ngx_http_regex_t          *regex;
+#endif
+    } cookie;
+
+    ngx_uint_t                     flags;
+    ngx_uint_t                     regex;
+} ngx_http_proxy_cookie_flags_t;
+
+
+typedef struct {
     ngx_str_t                      key_start;
     ngx_str_t                      schema;
     ngx_str_t                      host_header;
@@ -72,6 +98,7 @@ typedef struct {
     ngx_array_t                   *redirects;
     ngx_array_t                   *cookie_domains;
     ngx_array_t                   *cookie_paths;
+    ngx_array_t                   *cookie_flags;
 
     ngx_http_complex_value_t      *method;
     ngx_str_t                      location;
@@ -158,8 +185,14 @@ static ngx_int_t ngx_http_proxy_rewrite_
     ngx_table_elt_t *h, size_t prefix);
 static ngx_int_t ngx_http_proxy_rewrite_cookie(ngx_http_request_t *r,
     ngx_table_elt_t *h);
+static ngx_int_t ngx_http_proxy_parse_cookie(ngx_str_t *value,
+    ngx_array_t *attrs);
 static ngx_int_t ngx_http_proxy_rewrite_cookie_value(ngx_http_request_t *r,
-    ngx_table_elt_t *h, u_char *value, ngx_array_t *rewrites);
+    ngx_str_t *value, ngx_array_t *rewrites);
+static ngx_int_t ngx_http_proxy_rewrite_cookie_flags(ngx_http_request_t *r,
+    ngx_array_t *attrs, ngx_array_t *flags);
+static ngx_int_t ngx_http_proxy_edit_cookie_flags(ngx_http_request_t *r,
+    ngx_array_t *attrs, ngx_uint_t flags);
 static ngx_int_t ngx_http_proxy_rewrite(ngx_http_request_t *r,
     ngx_str_t *value, size_t prefix, size_t len, ngx_str_t *replacement);
 
@@ -180,6 +213,8 @@ static char *ngx_http_proxy_cookie_domai
     void *conf);
 static char *ngx_http_proxy_cookie_path(ngx_conf_t *cf, ngx_command_t *cmd,
     void *conf);
+static char *ngx_http_proxy_cookie_flags(ngx_conf_t *cf, ngx_command_t *cmd,
+    void *conf);
 static char *ngx_http_proxy_store(ngx_conf_t *cf, ngx_command_t *cmd,
     void *conf);
 #if (NGX_HTTP_CACHE)
@@ -282,6 +317,13 @@ static ngx_command_t  ngx_http_proxy_com
       0,
       NULL },
 
+    { ngx_string("proxy_cookie_flags"),
+      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1234,
+      ngx_http_proxy_cookie_flags,
+      NGX_HTTP_LOC_CONF_OFFSET,
+      0,
+      NULL },
+
     { ngx_string("proxy_store"),
       NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
       ngx_http_proxy_store,
@@ -845,6 +887,36 @@ static ngx_path_init_t  ngx_http_proxy_t
 };
 
 
+static ngx_conf_bitmask_t  ngx_http_proxy_cookie_flags_masks[] = {
+
+    { ngx_string("secure"),
+      NGX_HTTP_PROXY_COOKIE_SECURE|NGX_HTTP_PROXY_COOKIE_SECURE_ON },
+
+    { ngx_string("nosecure"),
+      NGX_HTTP_PROXY_COOKIE_SECURE|NGX_HTTP_PROXY_COOKIE_SECURE_OFF },
+
+    { ngx_string("httponly"),
+      NGX_HTTP_PROXY_COOKIE_HTTPONLY|NGX_HTTP_PROXY_COOKIE_HTTPONLY_ON },
+
+    { ngx_string("nohttponly"),
+      NGX_HTTP_PROXY_COOKIE_HTTPONLY|NGX_HTTP_PROXY_COOKIE_HTTPONLY_OFF },
+
+    { ngx_string("samesite=strict"),
+      NGX_HTTP_PROXY_COOKIE_SAMESITE|NGX_HTTP_PROXY_COOKIE_SAMESITE_STRICT },
+
+    { ngx_string("samesite=lax"),
+      NGX_HTTP_PROXY_COOKIE_SAMESITE|NGX_HTTP_PROXY_COOKIE_SAMESITE_LAX },
+
+    { ngx_string("samesite=none"),
+      NGX_HTTP_PROXY_COOKIE_SAMESITE|NGX_HTTP_PROXY_COOKIE_SAMESITE_NONE },
+
+    { ngx_string("nosamesite"),
+      NGX_HTTP_PROXY_COOKIE_SAMESITE|NGX_HTTP_PROXY_COOKIE_SAMESITE_OFF },
+
+    { ngx_null_string, 0 }
+};
+
+
 static ngx_int_t
 ngx_http_proxy_handler(ngx_http_request_t *r)
 {
@@ -906,7 +978,7 @@ ngx_http_proxy_handler(ngx_http_request_
         u->rewrite_redirect = ngx_http_proxy_rewrite_redirect;
     }
 
-    if (plcf->cookie_domains || plcf->cookie_paths) {
+    if (plcf->cookie_domains || plcf->cookie_paths || plcf->cookie_flags) {
         u->rewrite_cookie = ngx_http_proxy_rewrite_cookie;
     }
 
@@ -2598,27 +2670,41 @@ ngx_http_proxy_rewrite_redirect(ngx_http
 static ngx_int_t
 ngx_http_proxy_rewrite_cookie(ngx_http_request_t *r, ngx_table_elt_t *h)
 {
-    size_t                      prefix;
     u_char                     *p;
+    size_t                      len;
     ngx_int_t                   rc, rv;
+    ngx_str_t                  *key, *value;
+    ngx_uint_t                  i;
+    ngx_array_t                 attrs;
+    ngx_keyval_t               *attr;
     ngx_http_proxy_loc_conf_t  *plcf;
 
-    p = (u_char *) ngx_strchr(h->value.data, ';');
-    if (p == NULL) {
+    ngx_array_init(&attrs, r->pool, 2, sizeof(ngx_keyval_t));
+
+    if (ngx_http_proxy_parse_cookie(&h->value, &attrs) != NGX_OK) {
+        return NGX_ERROR;
+    }
+
+    attr = attrs.elts;
+
+    if (attr[0].value.data == NULL) {
         return NGX_DECLINED;
     }
 
-    prefix = p + 1 - h->value.data;
-
     rv = NGX_DECLINED;
 
     plcf = ngx_http_get_module_loc_conf(r, ngx_http_proxy_module);
 
-    if (plcf->cookie_domains) {
-        p = ngx_strcasestrn(h->value.data + prefix, "domain=", 7 - 1);
-
-        if (p) {
-            rc = ngx_http_proxy_rewrite_cookie_value(r, h, p + 7,
+    for (i = 1; i < attrs.nelts; i++) {
+
+        key = &attr[i].key;
+        value = &attr[i].value;
+
+        if (plcf->cookie_domains && key->len == 6
+            && ngx_strncasecmp(key->data, (u_char *) "domain", 6) == 0
+            && value->data)
+        {
+            rc = ngx_http_proxy_rewrite_cookie_value(r, value,
                                                      plcf->cookie_domains);
             if (rc == NGX_ERROR) {
                 return NGX_ERROR;
@@ -2628,13 +2714,12 @@ ngx_http_proxy_rewrite_cookie(ngx_http_r
                 rv = rc;
             }
         }
-    }
-
-    if (plcf->cookie_paths) {
-        p = ngx_strcasestrn(h->value.data + prefix, "path=", 5 - 1);
-
-        if (p) {
-            rc = ngx_http_proxy_rewrite_cookie_value(r, h, p + 5,
+
+        if (plcf->cookie_paths && key->len == 4
+            && ngx_strncasecmp(key->data, (u_char *) "path", 4) == 0
+            && value->data)
+        {
+            rc = ngx_http_proxy_rewrite_cookie_value(r, value,
                                                      plcf->cookie_paths);
             if (rc == NGX_ERROR) {
                 return NGX_ERROR;
@@ -2646,30 +2731,153 @@ ngx_http_proxy_rewrite_cookie(ngx_http_r
         }
     }
 
-    return rv;
+    if (plcf->cookie_flags) {
+        rc = ngx_http_proxy_rewrite_cookie_flags(r, &attrs,
+                                                 plcf->cookie_flags);
+        if (rc == NGX_ERROR) {
+            return NGX_ERROR;
+        }
+
+        if (rc != NGX_DECLINED) {
+            rv = rc;
+        }
+
+        attr = attrs.elts;
+    }
+
+    if (rv != NGX_OK) {
+        return rv;
+    }
+
+    len = 0;
+
+    for (i = 0; i < attrs.nelts; i++) {
+
+        if (attr[i].key.data == NULL) {
+            continue;
+        }
+
+        if (i > 0) {
+            len += 2;
+        }
+
+        len += attr[i].key.len;
+
+        if (attr[i].value.data) {
+            len += 1 + attr[i].value.len;
+        }
+    }
+
+    p = ngx_pnalloc(r->pool, len + 1);
+    if (p == NULL) {
+        return NGX_ERROR;
+    }
+
+    h->value.data = p;
+    h->value.len = len;
+
+    for (i = 0; i < attrs.nelts; i++) {
+
+        if (attr[i].key.data == NULL) {
+            continue;
+        }
+
+        if (i > 0) {
+            *p++ = ';';
+            *p++ = ' ';
+        }
+
+        p = ngx_cpymem(p, attr[i].key.data, attr[i].key.len);
+
+        if (attr[i].value.data) {
+            *p++ = '=';
+            p = ngx_cpymem(p, attr[i].value.data, attr[i].value.len);
+        }
+    }
+
+    *p = '\0';
+
+    return NGX_OK;
 }
 
 
 static ngx_int_t
-ngx_http_proxy_rewrite_cookie_value(ngx_http_request_t *r, ngx_table_elt_t *h,
-    u_char *value, ngx_array_t *rewrites)
+ngx_http_proxy_parse_cookie(ngx_str_t *value, ngx_array_t *attrs)
 {
-    size_t                     len, prefix;
-    u_char                    *p;
+    u_char        *start, *end, *p, *last;
+    ngx_str_t      name, val;
+    ngx_keyval_t  *attr;
+
+    start = value->data;
+    end = value->data + value->len;
+
+    for ( ;; ) {
+
+        last = (u_char *) ngx_strchr(start, ';');
+
+        if (last == NULL) {
+            last = end;
+        }
+
+        while (start < last && *start == ' ') { start++; }
+
+        for (p = start; p < last && *p != '='; p++) { /* void */ }
+
+        name.data = start;
+        name.len = p - start;
+
+        while (name.len && name.data[name.len - 1] == ' ') {
+            name.len--;
+        }
+
+        if (p < last) {
+
+            p++;
+
+            while (p < last && *p == ' ') { p++; }
+
+            val.data = p;
+            val.len = last - val.data;
+
+            while (val.len && val.data[val.len - 1] == ' ') {
+                val.len--;
+            }
+
+        } else {
+            ngx_str_null(&val);
+        }
+
+        attr = ngx_array_push(attrs);
+        if (attr == NULL) {
+            return NGX_ERROR;
+        }
+
+        attr->key = name;
+        attr->value = val;
+
+        if (last == end) {
+            break;
+        }
+
+        start = last + 1;
+    }
+
+    return NGX_OK;
+}
+
+
+static ngx_int_t
+ngx_http_proxy_rewrite_cookie_value(ngx_http_request_t *r, ngx_str_t *value,
+    ngx_array_t *rewrites)
+{
     ngx_int_t                  rc;
     ngx_uint_t                 i;
     ngx_http_proxy_rewrite_t  *pr;
 
-    prefix = value - h->value.data;
-
-    p = (u_char *) ngx_strchr(value, ';');
-
-    len = p ? (size_t) (p - value) : (h->value.len - prefix);
-
     pr = rewrites->elts;
 
     for (i = 0; i < rewrites->nelts; i++) {
-        rc = pr[i].handler(r, &h->value, prefix, len, &pr[i]);
+        rc = pr[i].handler(r, value, 0, value->len, &pr[i]);
 
         if (rc != NGX_DECLINED) {
             return rc;
@@ -2681,6 +2889,192 @@ ngx_http_proxy_rewrite_cookie_value(ngx_
 
 
 static ngx_int_t
+ngx_http_proxy_rewrite_cookie_flags(ngx_http_request_t *r, ngx_array_t *attrs,
+    ngx_array_t *flags)
+{
+    ngx_str_t                       pattern;
+#if (NGX_PCRE)
+    ngx_int_t                       rc;
+#endif
+    ngx_uint_t                      i;
+    ngx_keyval_t                   *attr;
+    ngx_http_proxy_cookie_flags_t  *pcf;
+
+    attr = attrs->elts;
+    pcf = flags->elts;
+
+    for (i = 0; i < flags->nelts; i++) {
+
+#if (NGX_PCRE)
+        if (pcf[i].regex) {
+            rc = ngx_http_regex_exec(r, pcf[i].cookie.regex, &attr[0].key);
+
+            if (rc == NGX_ERROR) {
+                return NGX_ERROR;
+            }
+
+            if (rc == NGX_OK) {
+                break;
+            }
+
+            /* NGX_DECLINED */
+
+            continue;
+        }
+#endif
+
+        if (ngx_http_complex_value(r, &pcf[i].cookie.complex, &pattern)
+            != NGX_OK)
+        {
+            return NGX_ERROR;
+        }
+
+        if (pattern.len == attr[0].key.len
+            && ngx_strncasecmp(attr[0].key.data, pattern.data, pattern.len)
+               == 0)
+        {
+            break;
+        }
+    }
+
+    if (i == flags->nelts) {
+        return NGX_DECLINED;
+    }
+
+    return ngx_http_proxy_edit_cookie_flags(r, attrs, pcf[i].flags);
+}
+
+
+static ngx_int_t
+ngx_http_proxy_edit_cookie_flags(ngx_http_request_t *r, ngx_array_t *attrs,
+    ngx_uint_t flags)
+{
+    ngx_str_t     *key, *value;
+    ngx_uint_t     i;
+    ngx_keyval_t  *attr;
+
+    attr = attrs->elts;
+
+    for (i = 1; i < attrs->nelts; i++) {
+        key = &attr[i].key;
+
+        if (key->len == 6
+            && ngx_strncasecmp(key->data, (u_char *) "secure", 6) == 0)
+        {
+            if (flags & NGX_HTTP_PROXY_COOKIE_SECURE_ON) {
+                flags &= ~NGX_HTTP_PROXY_COOKIE_SECURE_ON;
+
+            } else if (flags & NGX_HTTP_PROXY_COOKIE_SECURE_OFF) {
+                key->data = NULL;
+            }
+
+            continue;
+        }
+
+        if (key->len == 8
+            && ngx_strncasecmp(key->data, (u_char *) "httponly", 8) == 0)
+        {
+            if (flags & NGX_HTTP_PROXY_COOKIE_HTTPONLY_ON) {
+                flags &= ~NGX_HTTP_PROXY_COOKIE_HTTPONLY_ON;
+
+            } else if (flags & NGX_HTTP_PROXY_COOKIE_HTTPONLY_OFF) {
+                key->data = NULL;
+            }
+
+            continue;
+        }
+
+        if (key->len == 8
+            && ngx_strncasecmp(key->data, (u_char *) "samesite", 8) == 0)
+        {
+            value = &attr[i].value;
+
+            if (flags & NGX_HTTP_PROXY_COOKIE_SAMESITE_STRICT) {
+                flags &= ~NGX_HTTP_PROXY_COOKIE_SAMESITE_STRICT;
+
+                if (value->len != 6
+                    || ngx_strncasecmp(value->data, (u_char *) "strict", 6)
+                       != 0)
+                {
+                    ngx_str_set(key, "SameSite");
+                    ngx_str_set(value, "Strict");
+                }
+
+            } else if (flags & NGX_HTTP_PROXY_COOKIE_SAMESITE_LAX) {
+                flags &= ~NGX_HTTP_PROXY_COOKIE_SAMESITE_LAX;
+
+                if (value->len != 3
+                    || ngx_strncasecmp(value->data, (u_char *) "lax", 3) != 0)
+                {
+                    ngx_str_set(key, "SameSite");
+                    ngx_str_set(value, "Lax");
+                }
+
+            } else if (flags & NGX_HTTP_PROXY_COOKIE_SAMESITE_NONE) {
+                flags &= ~NGX_HTTP_PROXY_COOKIE_SAMESITE_NONE;
+
+                if (value->len != 4
+                    || ngx_strncasecmp(value->data, (u_char *) "none", 4) != 0)
+                {
+                    ngx_str_set(key, "SameSite");
+                    ngx_str_set(value, "None");
+                }
+
+            } else if (flags & NGX_HTTP_PROXY_COOKIE_SAMESITE_OFF) {
+                key->data = NULL;
+            }
+
+            continue;
+        }
+    }
+
+    if (flags & NGX_HTTP_PROXY_COOKIE_SECURE_ON) {
+        attr = ngx_array_push(attrs);
+        if (attr == NULL) {
+            return NGX_ERROR;
+        }
+
+        ngx_str_set(&attr->key, "Secure");
+        ngx_str_null(&attr->value);
+    }
+
+    if (flags & NGX_HTTP_PROXY_COOKIE_HTTPONLY_ON) {
+        attr = ngx_array_push(attrs);
+        if (attr == NULL) {
+            return NGX_ERROR;
+        }
+
+        ngx_str_set(&attr->key, "HttpOnly");
+        ngx_str_null(&attr->value);
+    }
+
+    if (flags & (NGX_HTTP_PROXY_COOKIE_SAMESITE_STRICT
+                 |NGX_HTTP_PROXY_COOKIE_SAMESITE_LAX
+                 |NGX_HTTP_PROXY_COOKIE_SAMESITE_NONE))
+    {
+        attr = ngx_array_push(attrs);
+        if (attr == NULL) {
+            return NGX_ERROR;
+        }
+
+        ngx_str_set(&attr->key, "SameSite");
+
+        if (flags & NGX_HTTP_PROXY_COOKIE_SAMESITE_STRICT) {
+            ngx_str_set(&attr->value, "Strict");
+
+        } else if (flags & NGX_HTTP_PROXY_COOKIE_SAMESITE_LAX) {
+            ngx_str_set(&attr->value, "Lax");
+
+        } else {
+            ngx_str_set(&attr->value, "None");
+        }
+    }
+
+    return NGX_OK;
+}
+
+
+static ngx_int_t
 ngx_http_proxy_rewrite_complex_handler(ngx_http_request_t *r, ngx_str_t *value,
     size_t prefix, size_t len, ngx_http_proxy_rewrite_t *pr)
 {
@@ -2742,7 +3136,7 @@ ngx_http_proxy_rewrite_domain_handler(ng
 
     p = value->data + prefix;
 
-    if (p[0] == '.') {
+    if (len && p[0] == '.') {
         p++;
         prefix++;
         len--;
@@ -2955,6 +3349,7 @@ ngx_http_proxy_create_loc_conf(ngx_conf_
 
     conf->cookie_domains = NGX_CONF_UNSET_PTR;
     conf->cookie_paths = NGX_CONF_UNSET_PTR;
+    conf->cookie_flags = NGX_CONF_UNSET_PTR;
 
     conf->http_version = NGX_CONF_UNSET_UINT;
 
@@ -3350,6 +3745,8 @@ ngx_http_proxy_merge_loc_conf(ngx_conf_t
 
     ngx_conf_merge_ptr_value(conf->cookie_paths, prev->cookie_paths, NULL);
 
+    ngx_conf_merge_ptr_value(conf->cookie_flags, prev->cookie_flags, NULL);
+
     ngx_conf_merge_uint_value(conf->http_version, prev->http_version,
                               NGX_HTTP_VERSION_10);
 
@@ -4081,6 +4478,131 @@ ngx_http_proxy_cookie_path(ngx_conf_t *c
 }
 
 
+static char *
+ngx_http_proxy_cookie_flags(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
+{
+    ngx_http_proxy_loc_conf_t *plcf = conf;
+
+    ngx_str_t                         *value;
+    ngx_uint_t                         i, m;
+    ngx_conf_bitmask_t                *mask;
+    ngx_http_proxy_cookie_flags_t     *pcf;
+    ngx_http_compile_complex_value_t   ccv;
+#if (NGX_PCRE)
+    ngx_regex_compile_t                rc;
+    u_char                             errstr[NGX_MAX_CONF_ERRSTR];
+#endif
+
+    if (plcf->cookie_flags == NULL) {
+        return "is duplicate";
+    }
+
+    value = cf->args->elts;
+
+    if (cf->args->nelts == 2) {
+
+        if (ngx_strcmp(value[1].data, "off") == 0) {
+
+            if (plcf->cookie_flags != NGX_CONF_UNSET_PTR) {
+                return "is duplicate";
+            }
+
+            plcf->cookie_flags = NULL;
+            return NGX_CONF_OK;
+        }
+
+        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                           "invalid parameter \"%V\"", &value[1]);
+        return NGX_CONF_ERROR;
+    }
+
+    if (plcf->cookie_flags == NGX_CONF_UNSET_PTR) {
+        plcf->cookie_flags = ngx_array_create(cf->pool, 1,
+                                        sizeof(ngx_http_proxy_cookie_flags_t));
+        if (plcf->cookie_flags == NULL) {
+            return NGX_CONF_ERROR;
+        }
+    }
+
+    pcf = ngx_array_push(plcf->cookie_flags);
+    if (pcf == NULL) {
+        return NGX_CONF_ERROR;
+    }
+
+    pcf->regex = 0;
+
+    if (value[1].data[0] == '~') {
+        value[1].len--;
+        value[1].data++;
+
+#if (NGX_PCRE)
+        ngx_memzero(&rc, sizeof(ngx_regex_compile_t));
+
+        rc.pattern = value[1];
+        rc.err.len = NGX_MAX_CONF_ERRSTR;
+        rc.err.data = errstr;
+        rc.options = NGX_REGEX_CASELESS;
+
+        pcf->cookie.regex = ngx_http_regex_compile(cf, &rc);
+        if (pcf->cookie.regex == NULL) {
+            return NGX_CONF_ERROR;
+        }
+
+        pcf->regex = 1;
+#else
+        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                           "using regex \"%V\" requires PCRE library",
+                           &value[1]);
+        return NGX_CONF_ERROR;
+#endif
+
+    } else {
+
+        ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));
+
+        ccv.cf = cf;
+        ccv.value = &value[1];
+        ccv.complex_value = &pcf->cookie.complex;
+
+        if (ngx_http_compile_complex_value(&ccv) != NGX_OK) {
+            return NGX_CONF_ERROR;
+        }
+    }
+
+    mask = ngx_http_proxy_cookie_flags_masks;
+    pcf->flags = 0;
+
+    for (i = 2; i < cf->args->nelts; i++) {
+        for (m = 0; mask[m].name.len != 0; m++) {
+
+            if (mask[m].name.len != value[i].len
+                || ngx_strcasecmp(mask[m].name.data, value[i].data) != 0)
+            {
+                continue;
+            }
+
+            if (pcf->flags & mask[m].mask) {
+                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                                   "duplicate parameter \"%V\"", &value[i]);
+                return NGX_CONF_ERROR;
+            }
+
+            pcf->flags |= mask[m].mask;
+
+            break;
+        }
+
+        if (mask[m].name.len == 0) {
+            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                               "invalid parameter \"%V\"", &value[i]);
+            return NGX_CONF_ERROR;
+        }
+    }
+
+    return NGX_CONF_OK;
+}
+
+
 static ngx_int_t
 ngx_http_proxy_rewrite_regex(ngx_conf_t *cf, ngx_http_proxy_rewrite_t *pr,
     ngx_str_t *regex, ngx_uint_t caseless)

From corentin.regal at gmail.com  Mon Sep 28 11:54:51 2020
From: corentin.regal at gmail.com (Corentin REGAL)
Date: Mon, 28 Sep 2020 13:54:51 +0200
Subject: Fix regex capture when using map directive and rewrite
In-Reply-To: <CALQyoc8yVwU0taJH3nPrNQsW7tNoV1Ks5w6jaejA9ntOscL_Pw@mail.gmail.com>
References: <CALQyoc8yVwU0taJH3nPrNQsW7tNoV1Ks5w6jaejA9ntOscL_Pw@mail.gmail.com>
Message-ID: <CALQyoc_JSXH4iqDxekUvDDJz-SYQkyU+75=P005K=pGifNrx1g@mail.gmail.com>

Hello,

I didn't receive any reply, is there something wrong with the patch ?

If more context is needed, I want to rewrite an url depending on the
Content-Type header.

Regards.

Le jeu. 17 sept. 2020 ? 10:39, Corentin REGAL <corentin.regal at gmail.com> a
?crit :

> Hello,
>
> I fixed a bug when using map directive and rewrite.
> Captured variables in the map directive override captures of the rewrite.
>
> *nginx.conf*
> daemon off;
> error_log  /var/log/nginx/error.log debug;
> worker_processes  1;
> events {
>     worker_connections  1024;
> }
>
> http {
>     include            mime.types;
>     default_type       application/octet-stream;
>     sendfile           on;
>     keepalive_timeout  0;
>
>     map $http_content_type $input_type {
>         default  json;
>         ~image/  image;
>     }
>
>     server {
>         server_name localhost;
>         listen 80;
>         location / {
>             rewrite "/(.+)" /$1_${input_type} redirect;
>         }
>     }
> }
>
>
> *before patch (/_image)*
> curl -v -H'Content-Type: image/png' localhost:80/predict
> ...
> < Location: http://localhost/_image
> ...
>
> *after patch (/predict_image)*
> curl -v -H'Content-Type: image/png' localhost:80/predict
> ...
> < Location: http://localhost/predict_image
> ...
>
> *patch*
> # HG changeset patch
> # User Corentin Regal <corentin.regal at gmail.com>
> # Date 1600329166 0
> #      Thu Sep 17 07:52:46 2020 +0000
> # Node ID a065d5f865e90a7426d37b30a9faa72e0966756f
> # Parent  052ecc68d35038b1b4adde12efe6249a92055f09
> Fix ngx_http_map_find to not erase rewrite captures
>
> diff -r 052ecc68d350 -r a065d5f865e9 src/http/ngx_http_variables.c
> --- a/src/http/ngx_http_variables.c     Wed Sep 16 18:26:25 2020 +0300
> +++ b/src/http/ngx_http_variables.c     Thu Sep 17 07:52:46 2020 +0000
> @@ -2410,7 +2410,7 @@
>
>          for (i = 0; i < map->nregex; i++) {
>
> -            n = ngx_http_regex_exec(r, reg[i].regex, match);
> +            n = ngx_regex_exec(reg[i].regex->regex, match, NULL, 0);
>
>              if (n == NGX_OK) {
>                  return reg[i].value;
>
> It's my first time using Mercurial or a mailing list, I hope I did it
> right :)
>
> Regards,
>
> Corentin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20200928/db6b5e08/attachment.htm>

From mdounin at mdounin.ru  Mon Sep 28 13:55:22 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 28 Sep 2020 16:55:22 +0300
Subject: Fix regex capture when using map directive and rewrite
In-Reply-To: <CALQyoc_JSXH4iqDxekUvDDJz-SYQkyU+75=P005K=pGifNrx1g@mail.gmail.com>
References: <CALQyoc8yVwU0taJH3nPrNQsW7tNoV1Ks5w6jaejA9ntOscL_Pw@mail.gmail.com>
 <CALQyoc_JSXH4iqDxekUvDDJz-SYQkyU+75=P005K=pGifNrx1g@mail.gmail.com>
Message-ID: <20200928135522.GT1136@mdounin.ru>

Hello!

On Mon, Sep 28, 2020 at 01:54:51PM +0200, Corentin REGAL wrote:

> I didn't receive any reply, is there something wrong with the patch ?

Yes.  In particular, the documentation says (http://nginx.org/r/map):

: A regular expression can contain named and positional captures 
: that can later be used in other directives along with the 
: resulting variable.

With your patch it will no longer work.

Just in case, relevant ticket in Trac is:

https://trac.nginx.org/nginx/ticket/564

-- 
Maxim Dounin
http://mdounin.ru/

From mdounin at mdounin.ru  Mon Sep 28 14:09:12 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 28 Sep 2020 14:09:12 +0000
Subject: [nginx] Userid: userid_flags directive to set cookie flags.
Message-ID: <hg.e3e8b8234f05.1601302152.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/e3e8b8234f05
branches:  
changeset: 7717:e3e8b8234f05
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Sep 28 17:07:48 2020 +0300
description:
Userid: userid_flags directive to set cookie flags.

diffstat:

 src/http/modules/ngx_http_userid_filter_module.c |  72 ++++++++++++++++++++++++
 1 files changed, 72 insertions(+), 0 deletions(-)

diffs (130 lines):

diff -r d6a5e14aa3e4 -r e3e8b8234f05 src/http/modules/ngx_http_userid_filter_module.c
--- a/src/http/modules/ngx_http_userid_filter_module.c	Sun Sep 27 23:21:11 2020 +0300
+++ b/src/http/modules/ngx_http_userid_filter_module.c	Mon Sep 28 17:07:48 2020 +0300
@@ -15,12 +15,20 @@
 #define NGX_HTTP_USERID_V1    2
 #define NGX_HTTP_USERID_ON    3
 
+#define NGX_HTTP_USERID_COOKIE_SECURE           0x0001
+#define NGX_HTTP_USERID_COOKIE_HTTPONLY         0x0002
+#define NGX_HTTP_USERID_COOKIE_SAMESITE         0x0004
+#define NGX_HTTP_USERID_COOKIE_SAMESITE_STRICT  0x0008
+#define NGX_HTTP_USERID_COOKIE_SAMESITE_LAX     0x0010
+#define NGX_HTTP_USERID_COOKIE_SAMESITE_NONE    0x0020
+
 /* 31 Dec 2037 23:55:55 GMT */
 #define NGX_HTTP_USERID_MAX_EXPIRES  2145916555
 
 
 typedef struct {
     ngx_uint_t  enable;
+    ngx_uint_t  flags;
 
     ngx_int_t   service;
 
@@ -88,6 +96,19 @@ static ngx_conf_enum_t  ngx_http_userid_
 };
 
 
+static ngx_conf_bitmask_t  ngx_http_userid_flags[] = {
+    { ngx_string("secure"), NGX_HTTP_USERID_COOKIE_SECURE },
+    { ngx_string("httponly"), NGX_HTTP_USERID_COOKIE_HTTPONLY },
+    { ngx_string("samesite=strict"),
+      NGX_HTTP_USERID_COOKIE_SAMESITE|NGX_HTTP_USERID_COOKIE_SAMESITE_STRICT },
+    { ngx_string("samesite=lax"),
+      NGX_HTTP_USERID_COOKIE_SAMESITE|NGX_HTTP_USERID_COOKIE_SAMESITE_LAX },
+    { ngx_string("samesite=none"),
+      NGX_HTTP_USERID_COOKIE_SAMESITE|NGX_HTTP_USERID_COOKIE_SAMESITE_NONE },
+    { ngx_null_string, 0 }
+};
+
+
 static ngx_conf_post_handler_pt  ngx_http_userid_domain_p =
     ngx_http_userid_domain;
 static ngx_conf_post_handler_pt  ngx_http_userid_path_p = ngx_http_userid_path;
@@ -138,6 +159,13 @@ static ngx_command_t  ngx_http_userid_co
       0,
       NULL },
 
+    { ngx_string("userid_flags"),
+      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE123,
+      ngx_conf_set_bitmask_slot,
+      NGX_HTTP_LOC_CONF_OFFSET,
+      offsetof(ngx_http_userid_conf_t, flags),
+      &ngx_http_userid_flags },
+
     { ngx_string("userid_p3p"),
       NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
       ngx_conf_set_str_slot,
@@ -383,6 +411,26 @@ ngx_http_userid_set_uid(ngx_http_request
         len += conf->domain.len;
     }
 
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_SECURE) {
+        len += sizeof("; secure") - 1;
+    }
+
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_HTTPONLY) {
+        len += sizeof("; httponly") - 1;
+    }
+
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_SAMESITE_STRICT) {
+        len += sizeof("; samesite=strict") - 1;
+    }
+
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_SAMESITE_LAX) {
+        len += sizeof("; samesite=lax") - 1;
+    }
+
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_SAMESITE_NONE) {
+        len += sizeof("; samesite=none") - 1;
+    }
+
     cookie = ngx_pnalloc(r->pool, len);
     if (cookie == NULL) {
         return NGX_ERROR;
@@ -422,6 +470,26 @@ ngx_http_userid_set_uid(ngx_http_request
 
     p = ngx_copy(p, conf->path.data, conf->path.len);
 
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_SECURE) {
+        p = ngx_cpymem(p, "; secure", sizeof("; secure") - 1);
+    }
+
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_HTTPONLY) {
+        p = ngx_cpymem(p, "; httponly", sizeof("; httponly") - 1);
+    }
+
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_SAMESITE_STRICT) {
+        p = ngx_cpymem(p, "; samesite=strict", sizeof("; samesite=strict") - 1);
+    }
+
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_SAMESITE_LAX) {
+        p = ngx_cpymem(p, "; samesite=lax", sizeof("; samesite=lax") - 1);
+    }
+
+    if (conf->flags & NGX_HTTP_USERID_COOKIE_SAMESITE_NONE) {
+        p = ngx_cpymem(p, "; samesite=none", sizeof("; samesite=none") - 1);
+    }
+
     set_cookie = ngx_list_push(&r->headers_out.headers);
     if (set_cookie == NULL) {
         return NGX_ERROR;
@@ -658,6 +726,7 @@ ngx_http_userid_create_conf(ngx_conf_t *
     /*
      * set by ngx_pcalloc():
      *
+     *     conf->flags = 0;
      *     conf->name = { 0, NULL };
      *     conf->domain = { 0, NULL };
      *     conf->path = { 0, NULL };
@@ -682,6 +751,9 @@ ngx_http_userid_merge_conf(ngx_conf_t *c
     ngx_conf_merge_uint_value(conf->enable, prev->enable,
                               NGX_HTTP_USERID_OFF);
 
+    ngx_conf_merge_bitmask_value(conf->flags, prev->flags,
+                                 NGX_CONF_BITMASK_SET);
+
     ngx_conf_merge_str_value(conf->name, prev->name, "uid");
     ngx_conf_merge_str_value(conf->domain, prev->domain, "");
     ngx_conf_merge_str_value(conf->path, prev->path, "; path=/");

From mdounin at mdounin.ru  Mon Sep 28 16:15:46 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 28 Sep 2020 16:15:46 +0000
Subject: [nginx] Resolver: improved error messages (ticket #2024).
Message-ID: <hg.8fe7ebe5adc4.1601309746.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/8fe7ebe5adc4
branches:  
changeset: 7718:8fe7ebe5adc4
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Sep 28 17:41:22 2020 +0300
description:
Resolver: improved error messages (ticket #2024).

diffstat:

 src/core/ngx_resolver.c |  32 ++++++++++++++++----------------
 1 files changed, 16 insertions(+), 16 deletions(-)

diffs (144 lines):

diff -r e3e8b8234f05 -r 8fe7ebe5adc4 src/core/ngx_resolver.c
--- a/src/core/ngx_resolver.c	Mon Sep 28 17:07:48 2020 +0300
+++ b/src/core/ngx_resolver.c	Mon Sep 28 17:41:22 2020 +0300
@@ -1918,7 +1918,7 @@ ngx_resolver_process_a(ngx_resolver_t *r
 
     if (rn == NULL) {
         ngx_log_error(r->log_level, r->log, 0,
-                      "unexpected response for %V", &name);
+                      "unexpected DNS response for %V", &name);
         ngx_resolver_free(r, name.data);
         goto failed;
     }
@@ -1930,7 +1930,7 @@ ngx_resolver_process_a(ngx_resolver_t *r
 
         if (rn->query6 == NULL || rn->naddrs6 != (u_short) -1) {
             ngx_log_error(r->log_level, r->log, 0,
-                          "unexpected response for %V", &name);
+                          "unexpected DNS response for %V", &name);
             ngx_resolver_free(r, name.data);
             goto failed;
         }
@@ -1949,7 +1949,7 @@ ngx_resolver_process_a(ngx_resolver_t *r
 
         if (rn->query == NULL || rn->naddrs != (u_short) -1) {
             ngx_log_error(r->log_level, r->log, 0,
-                          "unexpected response for %V", &name);
+                          "unexpected DNS response for %V", &name);
             ngx_resolver_free(r, name.data);
             goto failed;
         }
@@ -1964,7 +1964,7 @@ ngx_resolver_process_a(ngx_resolver_t *r
 
     if (ident != qident) {
         ngx_log_error(r->log_level, r->log, 0,
-                      "wrong ident %ui response for %V, expect %ui",
+                      "wrong ident %ui in DNS response for %V, expect %ui",
                       ident, &name, qident);
         ngx_resolver_free(r, name.data);
         goto failed;
@@ -2149,7 +2149,7 @@ ngx_resolver_process_a(ngx_resolver_t *r
 
         if (class != 1) {
             ngx_log_error(r->log_level, r->log, 0,
-                          "unexpected RR class %ui", class);
+                          "unexpected RR class %ui in DNS response", class);
             goto failed;
         }
 
@@ -2218,7 +2218,7 @@ ngx_resolver_process_a(ngx_resolver_t *r
         default:
 
             ngx_log_error(r->log_level, r->log, 0,
-                          "unexpected RR type %ui", type);
+                          "unexpected RR type %ui in DNS response", type);
         }
 
         i += len;
@@ -2567,7 +2567,7 @@ ngx_resolver_process_srv(ngx_resolver_t 
 
     if (rn == NULL || rn->query == NULL) {
         ngx_log_error(r->log_level, r->log, 0,
-                      "unexpected response for %V", &name);
+                      "unexpected DNS response for %V", &name);
         ngx_resolver_free(r, name.data);
         goto failed;
     }
@@ -2581,7 +2581,7 @@ ngx_resolver_process_srv(ngx_resolver_t 
 
     if (ident != qident) {
         ngx_log_error(r->log_level, r->log, 0,
-                      "wrong ident %ui response for %V, expect %ui",
+                      "wrong ident %ui in DNS response for %V, expect %ui",
                       ident, &name, qident);
         ngx_resolver_free(r, name.data);
         goto failed;
@@ -2691,7 +2691,7 @@ ngx_resolver_process_srv(ngx_resolver_t 
 
         if (class != 1) {
             ngx_log_error(r->log_level, r->log, 0,
-                          "unexpected RR class %ui", class);
+                          "unexpected RR class %ui in DNS response", class);
             goto failed;
         }
 
@@ -2734,7 +2734,7 @@ ngx_resolver_process_srv(ngx_resolver_t 
         default:
 
             ngx_log_error(r->log_level, r->log, 0,
-                          "unexpected RR type %ui", type);
+                          "unexpected RR type %ui in DNS response", type);
         }
 
         i += len;
@@ -3165,7 +3165,7 @@ valid:
 
     if (rn == NULL || rn->query == NULL) {
         ngx_log_error(r->log_level, r->log, 0,
-                      "unexpected response for %V", &name);
+                      "unexpected DNS response for %V", &name);
         ngx_resolver_free(r, name.data);
         goto failed;
     }
@@ -3174,7 +3174,7 @@ valid:
 
     if (ident != qident) {
         ngx_log_error(r->log_level, r->log, 0,
-                      "wrong ident %ui response for %V, expect %ui",
+                      "wrong ident %ui in DNS response for %V, expect %ui",
                       ident, &name, qident);
         ngx_resolver_free(r, name.data);
         goto failed;
@@ -3256,7 +3256,7 @@ valid:
 
         if (class != 1) {
             ngx_log_error(r->log_level, r->log, 0,
-                          "unexpected RR class %ui", class);
+                          "unexpected RR class %ui in DNS response", class);
             goto failed;
         }
 
@@ -3283,7 +3283,7 @@ valid:
         default:
 
             ngx_log_error(r->log_level, r->log, 0,
-                          "unexpected RR type %ui", type);
+                          "unexpected RR type %ui in DNS response", type);
         }
 
         i += len;
@@ -3952,12 +3952,12 @@ ngx_resolver_copy(ngx_resolver_t *r, ngx
         }
 
         if (p >= last) {
-            err = "name is out of response";
+            err = "name is out of DNS response";
             goto invalid;
         }
     }
 
-    err = "compression pointers loop";
+    err = "compression pointers loop in DNS response";
 
 invalid:
 

From xeioex at nginx.com  Mon Sep 28 17:46:30 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 28 Sep 2020 17:46:30 +0000
Subject: [njs] Fixed njs_buffer_set() introduced in 27bb9caf186c.
Message-ID: <hg.366aa456dc90.1601315190.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/366aa456dc90
branches:  
changeset: 1529:366aa456dc90
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Mon Sep 28 16:59:35 2020 +0000
description:
Fixed njs_buffer_set() introduced in 27bb9caf186c.

Previously an instance of Uint8Array was returned, not Buffer.

diffstat:

 src/njs_buffer.c         |  6 ++++--
 src/test/njs_unit_test.c |  5 ++++-
 2 files changed, 8 insertions(+), 3 deletions(-)

diffs (42 lines):

diff -r 8498c810d853 -r 366aa456dc90 src/njs_buffer.c
--- a/src/njs_buffer.c	Thu Sep 24 19:00:38 2020 +0000
+++ b/src/njs_buffer.c	Mon Sep 28 16:59:35 2020 +0000
@@ -97,7 +97,7 @@ njs_buffer_set(njs_vm_t *vm, njs_value_t
 
     buffer = (njs_array_buffer_t *) &array[1];
 
-    proto = &vm->prototypes[NJS_OBJ_TYPE_BUFFER].object;
+    proto = &vm->prototypes[NJS_OBJ_TYPE_ARRAY_BUFFER].object;
 
     njs_lvlhsh_init(&buffer->object.hash);
     njs_lvlhsh_init(&buffer->object.shared_hash);
@@ -111,10 +111,12 @@ njs_buffer_set(njs_vm_t *vm, njs_value_t
     buffer->u.data = (void *) start;
     buffer->size = size;
 
+    proto = &vm->prototypes[NJS_OBJ_TYPE_BUFFER].object;
+
     array->type = NJS_OBJ_TYPE_UINT8_ARRAY;
     njs_lvlhsh_init(&array->object.hash);
     njs_lvlhsh_init(&array->object.shared_hash);
-    array->object.__proto__ = &vm->prototypes[array->type].object;
+    array->object.__proto__ = proto;
     array->object.slots = NULL;
     array->object.type = NJS_TYPED_ARRAY;
     array->object.shared = 0;
diff -r 8498c810d853 -r 366aa456dc90 src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Thu Sep 24 19:00:38 2020 +0000
+++ b/src/test/njs_unit_test.c	Mon Sep 28 16:59:35 2020 +0000
@@ -19656,8 +19656,11 @@ static njs_unit_test_t  njs_externals_te
               "var s = (new TextDecoder()).decode(u16); [s, s.length]"),
       njs_str("???????????,11") },
 
-    { njs_str("$r.buffer.sort().slice(0,3)"),
+    { njs_str("new Uint8Array($r.buffer.sort().slice(0,3))"),
       njs_str("129,144,145") },
+
+    { njs_str("$r.buffer instanceof Buffer"),
+      njs_str("true") },
 };
 
 static njs_unit_test_t  njs_shared_test[] =

From xeioex at nginx.com  Mon Sep 28 17:46:32 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Mon, 28 Sep 2020 17:46:32 +0000
Subject: [njs] Adding support for Buffer objects in "fs" methods.
Message-ID: <hg.f1ed239edc7d.1601315192.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/f1ed239edc7d
branches:  
changeset: 1530:f1ed239edc7d
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Mon Sep 28 17:45:44 2020 +0000
description:
Adding support for Buffer objects in "fs" methods.

fs.writeFile(), fs.appendFile() and friends may accept an instance of
Buffer as an argument.

Also, fs.readFile() and friends now return an instance of Buffer instead of
Byte-string when encoding is not provided.

Added Buffer encoding for fs.readdir(), fs.realpath() and friends.

diffstat:

 src/njs_fs.c                 |  511 +++++++++++++++++++-----------------------
 src/test/njs_unit_test.c     |  175 +++++++-------
 test/js/fs_appendFile.js     |   61 +++++
 test/js/fs_appendFileSync.js |   59 ++++
 test/js/fs_promises_001.js   |    5 -
 test/js/fs_promises_007.js   |   16 +-
 test/js/fs_readFile.js       |   40 +++
 test/js/fs_readFileSync.js   |   43 +++
 test/js/fs_writeFile.js      |   57 ++++
 test/js/fs_writeFileSync.js  |   54 ++++
 test/njs_expect_test.exp     |  284 +++++++----------------
 11 files changed, 736 insertions(+), 569 deletions(-)

diffs (truncated from 1810 to 1000 lines):

diff -r 366aa456dc90 -r f1ed239edc7d src/njs_fs.c
--- a/src/njs_fs.c	Mon Sep 28 16:59:35 2020 +0000
+++ b/src/njs_fs.c	Mon Sep 28 17:45:44 2020 +0000
@@ -50,13 +50,6 @@ typedef enum {
 } njs_fs_writemode_t;
 
 
-typedef enum {
-    NJS_FS_ENC_INVALID,
-    NJS_FS_ENC_NONE,
-    NJS_FS_ENC_UTF8,
-} njs_fs_encoding_t;
-
-
 typedef struct {
     njs_str_t   name;
     int         value;
@@ -92,19 +85,19 @@ static njs_int_t njs_fs_error(njs_vm_t *
 static njs_int_t njs_fs_result(njs_vm_t *vm, njs_value_t *result,
     njs_index_t calltype, const njs_value_t* callback, njs_uint_t nargs);
 
-static njs_int_t
-njs_file_tree_walk(const char *path, njs_file_tree_walk_cb_t cb, int fd_limit,
-    njs_ftw_flags_t flags);
+static njs_int_t njs_file_tree_walk(const char *path,
+    njs_file_tree_walk_cb_t cb, int fd_limit, njs_ftw_flags_t flags);
 
 static njs_int_t njs_fs_make_path(njs_vm_t *vm, const char *path, mode_t md,
     njs_bool_t recursive, njs_value_t *retval);
 static njs_int_t njs_fs_rmtree(njs_vm_t *vm, const char *path,
     njs_bool_t recursive, njs_value_t *retval);
 
+static njs_int_t njs_fs_path(njs_vm_t *vm, const char **dst,
+    const njs_value_t* src, const njs_str_t *prop_name);
 static int njs_fs_flags(njs_vm_t *vm, njs_value_t *value, int default_flags);
 static mode_t njs_fs_mode(njs_vm_t *vm, njs_value_t *value,
     mode_t default_mode);
-static njs_fs_encoding_t njs_fs_encoding(njs_vm_t *vm, njs_value_t *value);
 
 static njs_int_t njs_fs_add_event(njs_vm_t *vm, const njs_value_t *callback,
     const njs_value_t *args, njs_uint_t nargs);
@@ -135,44 +128,28 @@ static njs_fs_entry_t njs_flags_table[] 
 };
 
 
-njs_inline njs_int_t
-njs_fs_path_arg(njs_vm_t *vm, const char **dst, const njs_value_t* src,
-    const njs_str_t *prop_name)
-{
-    if (njs_slow_path(!njs_is_string(src))) {
-        njs_type_error(vm, "\"%V\" must be a string", prop_name);
-        return NJS_ERROR;
-    }
-
-    *dst = njs_string_to_c_string(vm, njs_value_arg(src));
-    if (njs_slow_path(*dst == NULL)) {
-        return NJS_ERROR;
-    }
-
-    return NJS_OK;
-}
+static const njs_value_t  string_flag = njs_string("flag");
+static const njs_value_t  string_mode = njs_string("mode");
+static const njs_value_t  string_buffer = njs_string("buffer");
+static const njs_value_t  string_encoding = njs_string("encoding");
+static const njs_value_t  string_recursive = njs_string("recursive");
 
 
 static njs_int_t
 njs_fs_read_file(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t calltype)
 {
-    int                fd, flags;
-    u_char             *start;
-    size_t             size;
-    ssize_t            length;
-    njs_str_t          data;
-    njs_int_t          ret;
-    const char         *file_path;
-    njs_value_t        flag, encoding, retval, *callback, *options, *path;
-    struct stat        sb;
-    njs_fs_encoding_t  enc;
-
-    static const njs_value_t  string_flag = njs_string("flag");
-    static const njs_value_t  string_encoding = njs_string("encoding");
+    int                          fd, flags;
+    njs_str_t                    data;
+    njs_int_t                    ret;
+    const char                   *file_path;
+    njs_value_t                  flag, encode, retval, *callback, *options,
+                                 *path;
+    struct stat                  sb;
+    const njs_buffer_encoding_t  *encoding;
 
     path = njs_arg(args, nargs, 1);
-    ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path"));
+    ret = njs_fs_path(vm, &file_path, path, &njs_str_value("path"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
@@ -194,11 +171,11 @@ njs_fs_read_file(njs_vm_t *vm, njs_value
     }
 
     njs_set_undefined(&flag);
-    njs_set_undefined(&encoding);
+    njs_set_undefined(&encode);
 
     switch (options->type) {
     case NJS_STRING:
-        encoding = *options;
+        encode = *options;
         break;
 
     case NJS_UNDEFINED:
@@ -219,7 +196,7 @@ njs_fs_read_file(njs_vm_t *vm, njs_value
         }
 
         ret = njs_value_property(vm, options, njs_value_arg(&string_encoding),
-                                 &encoding);
+                                 &encode);
         if (njs_slow_path(ret == NJS_ERROR)) {
             return ret;
         }
@@ -230,9 +207,12 @@ njs_fs_read_file(njs_vm_t *vm, njs_value
         return NJS_ERROR;
     }
 
-    enc = njs_fs_encoding(vm, &encoding);
-    if (njs_slow_path(enc == NJS_FS_ENC_INVALID)) {
-        return NJS_ERROR;
+    encoding = NULL;
+    if (njs_is_defined(&encode)) {
+        encoding = njs_buffer_encoding(vm, &encode);
+        if (njs_slow_path(encoding == NULL)) {
+            return NJS_ERROR;
+        }
     }
 
     fd = open(file_path, flags);
@@ -252,88 +232,25 @@ njs_fs_read_file(njs_vm_t *vm, njs_value
         goto done;
     }
 
-    if (enc == NJS_FS_ENC_UTF8) {
-        length = sb.st_size;
-
-        if (length > NJS_STRING_MAP_STRIDE) {
-            /*
-             * At this point length is not known, in order to set it to
-             * the correct value after file is read, we need to ensure that
-             * offset_map is allocated by njs_string_alloc(). This can be
-             * achieved by making length != size.
-             */
-            length += 1;
+    data.start = NULL;
+    data.length = sb.st_size;
+
+    ret = njs_fs_fd_read(vm, fd, &data);
+    if (njs_slow_path(ret != NJS_OK)) {
+        if (ret == NJS_DECLINED) {
+            ret = njs_fs_error(vm, "read", strerror(errno), path, errno,
+                               &retval);
         }
 
+        goto done;
+    }
+
+    if (encoding == NULL) {
+        ret = njs_buffer_set(vm, &retval, data.start, data.length);
+
     } else {
-        length = 0;
-    }
-
-    size = sb.st_size;
-
-    if (njs_fast_path(size != 0)) {
-        start = njs_string_alloc(vm, &retval, size, length);
-        if (njs_slow_path(start == NULL)) {
-            ret = NJS_ERROR;
-            goto done;
-        }
-
-        data.start = start;
-        data.length = size;
-
-        ret = njs_fs_fd_read(vm, fd, &data);
-        if (njs_slow_path(ret != NJS_OK)) {
-            if (ret == NJS_DECLINED) {
-                ret = njs_fs_error(vm, "read", strerror(errno), path, errno,
-                                   &retval);
-            }
-
-            goto done;
-        }
-
-        if (njs_slow_path(data.length < size)) {
-            /* Pseudo-files may return less data than declared by st_size. */
-            njs_string_truncate(&retval, data.length, length);
-        }
-
-        size = data.length;
-        start = data.start;
-
-    } else {
-        /* size of the file is not known in advance. */
-
-        data.length = 0;
-
-        ret = njs_fs_fd_read(vm, fd, &data);
-        if (njs_slow_path(ret != NJS_OK)) {
-            if (ret == NJS_DECLINED) {
-                ret = njs_fs_error(vm, "read", strerror(errno), path, errno,
-                                   &retval);
-            }
-
-            goto done;
-        }
-
-        size = data.length;
-        start = data.start;
-
-        ret = njs_string_new(vm, &retval, start, size, length);
-        if (njs_slow_path(ret != NJS_OK)) {
-            goto done;
-        }
-    }
-
-    if (enc == NJS_FS_ENC_UTF8) {
-        length = njs_utf8_length(start, size);
-
-        if (length >= 0) {
-            njs_string_length_set(&retval, length);
-
-        } else {
-            ret = njs_fs_error(vm, NULL, "Non-UTF8 file, convertion "
-                               "is not implemented", path, 0, &retval);
-            goto done;
-        }
+        ret = encoding->encode(vm, &retval, &data);
+        njs_mp_free(vm->mem_pool, data.start);
     }
 
 done:
@@ -354,34 +271,26 @@ static njs_int_t
 njs_fs_write_file(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t magic)
 {
-    int                fd, flags;
-    u_char             *p, *end;
-    mode_t             md;
-    ssize_t            n;
-    njs_str_t          content;
-    njs_int_t          ret;
-    const char         *file_path;
-    njs_value_t        flag, mode, encoding, retval,
-                       *path, *data, *callback, *options;
-    njs_fs_encoding_t  enc;
-    njs_fs_calltype_t  calltype;
-
-    static const njs_value_t  string_flag = njs_string("flag");
-    static const njs_value_t  string_mode = njs_string("mode");
-    static const njs_value_t  string_encoding = njs_string("encoding");
+    int                          fd, flags;
+    u_char                       *p, *end;
+    mode_t                       md;
+    ssize_t                      n;
+    njs_str_t                    content;
+    njs_int_t                    ret;
+    const char                   *file_path;
+    njs_value_t                  flag, mode, encode, retval, *path, *data,
+                                 *callback, *options;
+    njs_typed_array_t            *array;
+    njs_fs_calltype_t            calltype;
+    njs_array_buffer_t           *buffer;
+    const njs_buffer_encoding_t  *encoding;
 
     path = njs_arg(args, nargs, 1);
-    ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path"));
+    ret = njs_fs_path(vm, &file_path, path, &njs_str_value("path"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
 
-    data = njs_arg(args, nargs, 2);
-    if (njs_slow_path(!njs_is_string(data))) {
-        njs_type_error(vm, "\"data\" must be a string");
-        return NJS_ERROR;
-    }
-
     callback = NULL;
     calltype = magic & 3;
     options = njs_arg(args, nargs, 3);
@@ -400,11 +309,11 @@ njs_fs_write_file(njs_vm_t *vm, njs_valu
 
     njs_set_undefined(&flag);
     njs_set_undefined(&mode);
-    njs_set_undefined(&encoding);
+    njs_set_undefined(&encode);
 
     switch (options->type) {
     case NJS_STRING:
-        encoding = *options;
+        encode = *options;
         break;
 
     case NJS_UNDEFINED:
@@ -431,12 +340,49 @@ njs_fs_write_file(njs_vm_t *vm, njs_valu
         }
 
         ret = njs_value_property(vm, options, njs_value_arg(&string_encoding),
-                                 &encoding);
+                                 &encode);
         if (njs_slow_path(ret == NJS_ERROR)) {
             return ret;
         }
     }
 
+    data = njs_arg(args, nargs, 2);
+
+    switch (data->type) {
+    case NJS_TYPED_ARRAY:
+    case NJS_DATA_VIEW:
+        array = njs_typed_array(data);
+        buffer = array->buffer;
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
+        content.start = &buffer->u.u8[array->offset];
+        content.length = array->byte_length;
+        break;
+
+    case NJS_STRING:
+    default:
+        encoding = njs_buffer_encoding(vm, &encode);
+        if (njs_slow_path(encoding == NULL)) {
+            return NJS_ERROR;
+        }
+
+        ret = njs_value_to_string(vm, &retval, data);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_ERROR;
+        }
+
+        ret = njs_buffer_decode_string(vm, &retval, &retval, encoding);
+        if (njs_slow_path(ret != NJS_OK)) {
+            return NJS_ERROR;
+        }
+
+        njs_string_get(&retval, &content);
+        break;
+    }
+
     flags = njs_fs_flags(vm, &flag, O_CREAT | O_WRONLY);
     if (njs_slow_path(flags == -1)) {
         return NJS_ERROR;
@@ -449,19 +395,12 @@ njs_fs_write_file(njs_vm_t *vm, njs_valu
         return NJS_ERROR;
     }
 
-    enc = njs_fs_encoding(vm, &encoding);
-    if (njs_slow_path(enc == NJS_FS_ENC_INVALID)) {
-        return NJS_ERROR;
-    }
-
     fd = open(file_path, flags, md);
     if (njs_slow_path(fd < 0)) {
         ret = njs_fs_error(vm, "open", strerror(errno), path, errno, &retval);
         goto done;
     }
 
-    njs_string_get(data, &content);
-
     p = content.start;
     end = p + content.length;
 
@@ -515,13 +454,13 @@ njs_fs_rename(njs_vm_t *vm, njs_value_t 
         }
     }
 
-    ret = njs_fs_path_arg(vm, &old_path, njs_arg(args, nargs, 1),
+    ret = njs_fs_path(vm, &old_path, njs_arg(args, nargs, 1),
                           &njs_str_value("oldPath"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
 
-    ret = njs_fs_path_arg(vm, &new_path, njs_arg(args, nargs, 2),
+    ret = njs_fs_path(vm, &new_path, njs_arg(args, nargs, 2),
                           &njs_str_value("newPath"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
@@ -552,7 +491,7 @@ njs_fs_access(njs_vm_t *vm, njs_value_t 
     njs_value_t  retval, *path, *callback, *mode;
 
     path = njs_arg(args, nargs, 1);
-    ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path"));
+    ret = njs_fs_path(vm, &file_path, path, &njs_str_value("path"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
@@ -610,13 +549,13 @@ njs_fs_symlink(njs_vm_t *vm, njs_value_t
     njs_value_t  retval, *target, *path, *callback, *type;
 
     target = njs_arg(args, nargs, 1);
-    ret = njs_fs_path_arg(vm, &target_path, target, &njs_str_value("target"));
+    ret = njs_fs_path(vm, &target_path, target, &njs_str_value("target"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
 
     path = njs_arg(args, nargs, 2);
-    ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path"));
+    ret = njs_fs_path(vm, &file_path, path, &njs_str_value("path"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
@@ -666,7 +605,7 @@ njs_fs_unlink(njs_vm_t *vm, njs_value_t 
     njs_value_t  retval, *path, *callback;
 
     path = njs_arg(args, nargs, 1);
-    ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path"));
+    ret = njs_fs_path(vm, &file_path, path, &njs_str_value("path"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
@@ -700,19 +639,15 @@ static njs_int_t
 njs_fs_realpath(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t calltype)
 {
-    u_char             *resolved_path;
-    size_t             size;
-    ssize_t            length;
-    njs_int_t          ret;
-    const char         *file_path;
-    njs_value_t        encoding, retval, *path, *callback, *options;
-    njs_fs_encoding_t  enc;
-    char               path_buf[MAXPATHLEN];
-
-    static const njs_value_t  string_encoding = njs_string("encoding");
+    njs_int_t                    ret;
+    njs_str_t                    s;
+    const char                   *file_path;
+    njs_value_t                  encode, retval, *path, *callback, *options;
+    const njs_buffer_encoding_t  *encoding;
+    char                         path_buf[MAXPATHLEN];
 
     path = njs_arg(args, nargs, 1);
-    ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path"));
+    ret = njs_fs_path(vm, &file_path, path, &njs_str_value("path"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
@@ -732,11 +667,11 @@ njs_fs_realpath(njs_vm_t *vm, njs_value_
         }
     }
 
-    njs_set_undefined(&encoding);
+    njs_set_undefined(&encode);
 
     switch (options->type) {
     case NJS_STRING:
-        encoding = *options;
+        encode = *options;
         break;
 
     case NJS_UNDEFINED:
@@ -751,33 +686,34 @@ njs_fs_realpath(njs_vm_t *vm, njs_value_
         }
 
         ret = njs_value_property(vm, options, njs_value_arg(&string_encoding),
-                                 &encoding);
+                                 &encode);
         if (njs_slow_path(ret == NJS_ERROR)) {
             return ret;
         }
     }
 
-    enc = njs_fs_encoding(vm, &encoding);
-    if (njs_slow_path(enc == NJS_FS_ENC_INVALID)) {
-        return NJS_ERROR;
+    encoding = NULL;
+    if (!njs_is_string(&encode) || !njs_string_eq(&encode, &string_buffer)) {
+        encoding = njs_buffer_encoding(vm, &encode);
+        if (njs_slow_path(encoding == NULL)) {
+            return NJS_ERROR;
+        }
     }
 
-    resolved_path = (u_char *) realpath(file_path, path_buf);
-    if (njs_slow_path(resolved_path == NULL)) {
+    s.start = (u_char *) realpath(file_path, path_buf);
+    if (njs_slow_path(s.start == NULL)) {
         ret = njs_fs_error(vm, "realpath", strerror(errno), path, errno,
                            &retval);
         goto done;
     }
 
-    size = njs_strlen(resolved_path);
-    length = njs_utf8_length(resolved_path, size);
-    if (njs_slow_path(length < 0)) {
-        length = 0;
-    }
-
-    ret = njs_string_new(vm, &retval, resolved_path, size, length);
-    if (njs_slow_path(ret != NJS_OK)) {
-        return NJS_ERROR;
+    s.length = njs_strlen(s.start);
+
+    if (encoding == NULL) {
+        ret = njs_buffer_new(vm, &retval, s.start, s.length);
+
+    } else {
+        ret = encoding->encode(vm, &retval, &s);
     }
 
 done:
@@ -799,11 +735,8 @@ njs_fs_mkdir(njs_vm_t *vm, njs_value_t *
     const char   *file_path;
     njs_value_t  mode, recursive, retval, *path, *callback, *options;
 
-    static const njs_value_t  string_mode = njs_string("mode");
-    static const njs_value_t  string_recursive = njs_string("recursive");
-
     path = njs_arg(args, nargs, 1);
-    ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path"));
+    ret = njs_fs_path(vm, &file_path, path, &njs_str_value("path"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
@@ -878,10 +811,8 @@ njs_fs_rmdir(njs_vm_t *vm, njs_value_t *
     const char   *file_path;
     njs_value_t  recursive, retval, *path, *callback, *options;
 
-    static const njs_value_t  string_recursive = njs_string("recursive");
-
     path = njs_arg(args, nargs, 1);
-    ret = njs_fs_path_arg(vm, &file_path, path, &njs_str_value("path"));
+    ret = njs_fs_path(vm, &file_path, path, &njs_str_value("path"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
@@ -935,23 +866,20 @@ static njs_int_t
 njs_fs_readdir(njs_vm_t *vm, njs_value_t *args, njs_uint_t nargs,
     njs_index_t calltype)
 {
-    DIR                *dir;
-    u_char             *d_name;
-    size_t             size;
-    ssize_t            length;
-    njs_int_t          ret;
-    const char         *dir_path;
-    njs_value_t        encoding, types, ename, etype, retval, *path, *callback,
-                       *options, *value;
-    njs_array_t        *results;
-    struct dirent      *entry;
-    njs_fs_encoding_t  enc;
-
-    static const njs_value_t  string_encoding = njs_string("encoding");
+    DIR                          *dir;
+    njs_str_t                    s;
+    njs_int_t                    ret;
+    const char                   *dir_path;
+    njs_value_t                  encode, types, ename, etype, retval, *path,
+                                 *callback, *options, *value;
+    njs_array_t                  *results;
+    struct dirent                *entry;
+    const njs_buffer_encoding_t  *encoding;
+
     static const njs_value_t  string_types = njs_string("withFileTypes");
 
     path = njs_arg(args, nargs, 1);
-    ret = njs_fs_path_arg(vm, &dir_path, path, &njs_str_value("path"));
+    ret = njs_fs_path(vm, &dir_path, path, &njs_str_value("path"));
     if (njs_slow_path(ret != NJS_OK)) {
         return ret;
     }
@@ -971,11 +899,11 @@ njs_fs_readdir(njs_vm_t *vm, njs_value_t
     }
 
     njs_set_false(&types);
-    njs_set_undefined(&encoding);
+    njs_set_undefined(&encode);
 
     switch (options->type) {
     case NJS_STRING:
-        encoding = *options;
+        encode = *options;
         break;
 
     case NJS_UNDEFINED:
@@ -990,7 +918,7 @@ njs_fs_readdir(njs_vm_t *vm, njs_value_t
         }
 
         ret = njs_value_property(vm, options, njs_value_arg(&string_encoding),
-                                 &encoding);
+                                 &encode);
         if (njs_slow_path(ret == NJS_ERROR)) {
             return ret;
         }
@@ -1002,9 +930,12 @@ njs_fs_readdir(njs_vm_t *vm, njs_value_t
         }
     }
 
-    enc = njs_fs_encoding(vm, &encoding);
-    if (njs_slow_path(enc == NJS_FS_ENC_INVALID)) {
-        return NJS_ERROR;
+    encoding = NULL;
+    if (!njs_is_string(&encode) || !njs_string_eq(&encode, &string_buffer)) {
+        encoding = njs_buffer_encoding(vm, &encode);
+        if (njs_slow_path(encoding == NULL)) {
+            return NJS_ERROR;
+        }
     }
 
     results = njs_array_alloc(vm, 1, 0, NJS_ARRAY_SPARE);
@@ -1033,41 +964,38 @@ njs_fs_readdir(njs_vm_t *vm, njs_value_t
             goto done;
         }
 
-        d_name = (u_char *) entry->d_name;
-
-        size = njs_strlen(d_name);
-        length = njs_utf8_length(d_name, size);
-        if (njs_slow_path(length < 0)) {
-            length = 0;
-        }
-
-        if ((length == 1 && d_name[0] == '.')
-            || (length == 2 && (d_name[0] == '.' && d_name[1] == '.')))
+        s.start = (u_char *) entry->d_name;
+        s.length = njs_strlen(s.start);
+
+        if ((s.length == 1 && s.start[0] == '.')
+            || (s.length == 2 && (s.start[0] == '.' && s.start[1] == '.')))
         {
             continue;
         }
 
-        if (njs_fast_path(!njs_is_true(&types))) {
-            ret = njs_array_string_add(vm, results, d_name, size, length);
-            if (njs_slow_path(ret != NJS_OK)) {
-                goto done;
-            }
-
-            continue;
-        }
-
-        ret = njs_string_new(vm, &ename, d_name, size, length);
-        if (njs_slow_path(ret != NJS_OK)) {
-            goto done;
-        }
-
-        njs_set_number(&etype, njs_dentry_type(entry));
-
         value = njs_array_push(vm, results);
         if (njs_slow_path(value == NULL)) {
             goto done;
         }
 
+        if (encoding == NULL) {
+            ret = njs_buffer_set(vm, &ename, s.start, s.length);
+
+        } else {
+            ret = encoding->encode(vm, &ename, &s);
+        }
+
+        if (njs_slow_path(ret != NJS_OK)) {
+            goto done;
+        }
+
+        if (njs_fast_path(!njs_is_true(&types))) {
+            *value = ename;
+            continue;
+        }
+
+        njs_set_number(&etype, njs_dentry_type(entry));
+
         ret = njs_fs_dirent_create(vm, &ename, &etype, value);
         if (njs_slow_path(ret != NJS_OK)) {
             goto done;
@@ -1099,12 +1027,12 @@ njs_fs_fd_read(njs_vm_t *vm, int fd, njs
 
     if (size == 0) {
         size = 4096;
-
-        data->start = njs_mp_alloc(vm->mem_pool, size);
-        if (data->start == NULL) {
-            njs_memory_error(vm);
-            return NJS_ERROR;
-        }
+    }
+
+    data->start = njs_mp_alloc(vm->mem_pool, size);
+    if (data->start == NULL) {
+        njs_memory_error(vm);
+        return NJS_ERROR;
     }
 
     p = data->start;
@@ -1474,6 +1402,61 @@ njs_fs_rmtree(njs_vm_t *vm, const char *
 }
 
 
+static njs_int_t
+njs_fs_path(njs_vm_t *vm, const char **dst, const njs_value_t* src,
+    const njs_str_t *prop_name)
+{
+    u_char              *data, *p, *start;
+    njs_typed_array_t   *array;
+    njs_array_buffer_t  *buffer;
+
+    switch (src->type) {
+    case NJS_STRING:
+        *dst = njs_string_to_c_string(vm, njs_value_arg(src));
+        if (njs_slow_path(*dst == NULL)) {
+            return NJS_ERROR;
+        }
+
+        break;
+
+    case NJS_TYPED_ARRAY:
+    case NJS_DATA_VIEW:
+        array = njs_typed_array(src);
+        buffer = array->buffer;
+        if (njs_slow_path(njs_is_detached_buffer(buffer))) {
+            njs_type_error(vm, "detached buffer");
+            return NJS_ERROR;
+        }
+
+        start = &buffer->u.u8[array->offset];
+
+        if (njs_slow_path(memchr(start, '\0', array->byte_length) != 0)) {
+            njs_type_error(vm, "\"%V\" must be a Buffer without null bytes",
+                           prop_name);
+            return NJS_ERROR;
+        }
+
+        data = njs_mp_alloc(vm->mem_pool, array->byte_length + 1);
+        if (njs_slow_path(data == NULL)) {
+            njs_memory_error(vm);
+            return NJS_ERROR;
+        }
+
+        p = njs_cpymem(data, start, array->byte_length);
+        *p++ = '\0';
+
+        *dst = (char *) data;
+        break;
+
+    default:
+        njs_type_error(vm, "\"%V\" must be a string or Buffer", prop_name);
+        return NJS_ERROR;
+    }
+
+    return NJS_OK;
+}
+
+
 static int
 njs_fs_flags(njs_vm_t *vm, njs_value_t *value, int default_flags)
 {
@@ -1526,32 +1509,6 @@ njs_fs_mode(njs_vm_t *vm, njs_value_t *v
 }
 
 
-static njs_fs_encoding_t
-njs_fs_encoding(njs_vm_t *vm, njs_value_t *value)
-{
-    njs_str_t  enc;
-    njs_int_t  ret;
-
-    if (njs_is_undefined(value)) {
-        return NJS_FS_ENC_NONE;
-    }
-
-    ret = njs_value_to_string(vm, value, value);
-    if (njs_slow_path(ret != NJS_OK)) {
-        return NJS_FS_ENC_INVALID;
-    }
-
-    njs_string_get(value, &enc);
-
-    if (enc.length != 4 || memcmp(enc.start, "utf8", 4) != 0) {
-        njs_type_error(vm, "Unknown encoding: \"%V\"", &enc);
-        return NJS_FS_ENC_INVALID;
-    }
-
-    return NJS_FS_ENC_UTF8;
-}
-
-
 static njs_int_t
 njs_fs_error(njs_vm_t *vm, const char *syscall, const char *description,
     njs_value_t *path, int errn, njs_value_t *retval)
diff -r 366aa456dc90 -r f1ed239edc7d src/test/njs_unit_test.c
--- a/src/test/njs_unit_test.c	Mon Sep 28 16:59:35 2020 +0000
+++ b/src/test/njs_unit_test.c	Mon Sep 28 17:45:44 2020 +0000
@@ -17013,145 +17013,146 @@ static njs_unit_test_t  njs_test[] =
     /* require('fs').readFile() */
 
     { njs_str("var fs = require('fs');"
-                 "fs.readFile()"),
-      njs_str("TypeError: \"path\" must be a string") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.readFile('/njs_unknown_path')"),
+              "fs.readFile()"),
+      njs_str("TypeError: \"path\" must be a string or Buffer") },
+
+    { njs_str("var fs = require('fs');"
+              "var path = Buffer.from('/broken'); path[3] = 0;"
+              "fs.readFile(path)"),
+      njs_str("TypeError: \"path\" must be a Buffer without null bytes") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.readFile('/njs_unknown_path')"),
       njs_str("TypeError: \"callback\" must be a function") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.readFile('/njs_unknown_path', 'utf8')"),
-      njs_str("TypeError: \"callback\" must be a function") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.readFile('/njs_unknown_path', {flag:'xx'})"),
+              "fs.readFile('/njs_unknown_path', 'utf8')"),
       njs_str("TypeError: \"callback\" must be a function") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.readFile('/njs_unknown_path', {flag:'xx'}, 1)"),
+              "fs.readFile('/njs_unknown_path', {flag:'xx'})"),
       njs_str("TypeError: \"callback\" must be a function") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.readFile('/njs_unknown_path', {flag:'xx'}, function () {})"),
+              "fs.readFile('/njs_unknown_path', {flag:'xx'}, 1)"),
+      njs_str("TypeError: \"callback\" must be a function") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.readFile('/njs_unknown_path', {flag:'xx'}, function () {})"),
       njs_str("TypeError: Unknown file open flags: \"xx\"") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.readFile('/njs_unknown_path', {encoding:'ascii'}, function () {})"),
-      njs_str("TypeError: Unknown encoding: \"ascii\"") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.readFile('/njs_unknown_path', 'ascii', function () {})"),
-      njs_str("TypeError: Unknown encoding: \"ascii\"") },
+              "fs.readFile('/njs_unknown_path', {encoding:'ascii'}, function () {})"),
+      njs_str("TypeError: \"ascii\" encoding is not supported") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.readFile('/njs_unknown_path', 'ascii', function () {})"),
+      njs_str("TypeError: \"ascii\" encoding is not supported") },
 
     /* require('fs').readFileSync() */
 
     { njs_str("var fs = require('fs');"
-                 "fs.readFileSync()"),
-      njs_str("TypeError: \"path\" must be a string") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.readFileSync({})"),
-      njs_str("TypeError: \"path\" must be a string") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.readFileSync('/njs_unknown_path', {flag:'xx'})"),
+              "fs.readFileSync()"),
+      njs_str("TypeError: \"path\" must be a string or Buffer") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.readFileSync({})"),
+      njs_str("TypeError: \"path\" must be a string or Buffer") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.readFileSync('/njs_unknown_path', {flag:'xx'})"),
       njs_str("TypeError: Unknown file open flags: \"xx\"") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.readFileSync('/njs_unknown_path', {encoding:'ascii'})"),
-      njs_str("TypeError: Unknown encoding: \"ascii\"") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.readFileSync('/njs_unknown_path', 'ascii')"),
-      njs_str("TypeError: Unknown encoding: \"ascii\"") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.readFileSync('/njs_unknown_path', true)"),
+              "fs.readFileSync(Buffer.from('/njs_unknown_path'), {encoding:'ascii'})"),
+      njs_str("TypeError: \"ascii\" encoding is not supported") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.readFileSync('/njs_unknown_path', 'ascii')"),
+      njs_str("TypeError: \"ascii\" encoding is not supported") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.readFileSync('/njs_unknown_path', true)"),
       njs_str("TypeError: Unknown options type: \"boolean\" (a string or object required)") },
 
 
     /* require('fs').writeFile() */
 
     { njs_str("var fs = require('fs');"
-                 "fs.writeFile()"),
-      njs_str("TypeError: \"path\" must be a string") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.writeFile({}, '', function () {})"),
-      njs_str("TypeError: \"path\" must be a string") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.writeFile('/njs_unknown_path')"),
-      njs_str("TypeError: \"data\" must be a string") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.writeFile('/njs_unknown_path', '')"),
+              "fs.writeFile()"),
+      njs_str("TypeError: \"path\" must be a string or Buffer") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.writeFile({}, '', function () {})"),
+      njs_str("TypeError: \"path\" must be a string or Buffer") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.writeFile('/njs_unknown_path')"),
       njs_str("TypeError: \"callback\" must be a function") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.writeFile('/njs_unknown_path', '', undefined)"),
+              "fs.writeFile('/njs_unknown_path', '')"),
+      njs_str("TypeError: \"callback\" must be a function") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.writeFile('/njs_unknown_path', '', undefined)"),
       njs_str("TypeError: \"callback\" must be a function") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.writeFile('/njs_unknown_path', '', 'utf8')"),
+              "fs.writeFile('/njs_unknown_path', '', 'utf8')"),
       njs_str("TypeError: \"callback\" must be a function") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.writeFile('/njs_unknown_path', '', {flag:'xx'}, function () {})"),
+              "fs.writeFile('/njs_unknown_path', '', {flag:'xx'}, function () {})"),
       njs_str("TypeError: Unknown file open flags: \"xx\"") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.writeFile('/njs_unknown_path', '', {encoding:'ascii'}, function () {})"),
-      njs_str("TypeError: Unknown encoding: \"ascii\"") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.writeFile('/njs_unknown_path', '', 'ascii', function () {})"),
-      njs_str("TypeError: Unknown encoding: \"ascii\"") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.writeFile('/njs_unknown_path', '', true, function () {})"),
+              "fs.writeFile('/njs_unknown_path', '', {encoding:'ascii'}, function () {})"),
+      njs_str("TypeError: \"ascii\" encoding is not supported") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.writeFile('/njs_unknown_path', '', 'ascii', function () {})"),
+      njs_str("TypeError: \"ascii\" encoding is not supported") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.writeFile('/njs_unknown_path', '', true, function () {})"),
       njs_str("TypeError: Unknown options type: \"boolean\" (a string or object required)") },
 
     /* require('fs').writeFileSync() */
 
     { njs_str("var fs = require('fs');"
-                 "fs.writeFileSync()"),
-      njs_str("TypeError: \"path\" must be a string") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.writeFileSync('/njs_unknown_path')"),
-      njs_str("TypeError: \"data\" must be a string") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.writeFileSync({}, '')"),
-      njs_str("TypeError: \"path\" must be a string") },
-
-    { njs_str("var fs = require('fs');"
-                 "fs.writeFileSync('/njs_unknown_path', '', {flag:'xx'})"),
+              "fs.writeFileSync()"),
+      njs_str("TypeError: \"path\" must be a string or Buffer") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.writeFileSync({}, '')"),
+      njs_str("TypeError: \"path\" must be a string or Buffer") },
+
+    { njs_str("var fs = require('fs');"
+              "fs.writeFileSync('/njs_unknown_path', '', {flag:'xx'})"),
       njs_str("TypeError: Unknown file open flags: \"xx\"") },
 
     { njs_str("var fs = require('fs');"
-                 "fs.writeFileSync('/njs_unknown_path', '', {encoding:'ascii'})"),
-      njs_str("TypeError: Unknown encoding: \"ascii\"") },

From mdounin at mdounin.ru  Tue Sep 29 13:18:09 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 29 Sep 2020 13:18:09 +0000
Subject: [nginx] Userid: userid_flags fixup.
Message-ID: <hg.c0cacad62cc8.1601385489.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/c0cacad62cc8
branches:  
changeset: 7719:c0cacad62cc8
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Sep 29 15:52:18 2020 +0300
description:
Userid: userid_flags fixup.

In 7717:e3e8b8234f05, the 1st bit was incorrectly used.  It shouldn't
be used for bitmask values, as it is used by NGX_CONF_BITMASK_SET.

Additionally, special value "off" added to make it possible to clear
inherited userid_flags value.

diffstat:

 src/http/modules/ngx_http_userid_filter_module.c |  16 +++++++++-------
 1 files changed, 9 insertions(+), 7 deletions(-)

diffs (40 lines):

diff -r 8fe7ebe5adc4 -r c0cacad62cc8 src/http/modules/ngx_http_userid_filter_module.c
--- a/src/http/modules/ngx_http_userid_filter_module.c	Mon Sep 28 17:41:22 2020 +0300
+++ b/src/http/modules/ngx_http_userid_filter_module.c	Tue Sep 29 15:52:18 2020 +0300
@@ -15,12 +15,13 @@
 #define NGX_HTTP_USERID_V1    2
 #define NGX_HTTP_USERID_ON    3
 
-#define NGX_HTTP_USERID_COOKIE_SECURE           0x0001
-#define NGX_HTTP_USERID_COOKIE_HTTPONLY         0x0002
-#define NGX_HTTP_USERID_COOKIE_SAMESITE         0x0004
-#define NGX_HTTP_USERID_COOKIE_SAMESITE_STRICT  0x0008
-#define NGX_HTTP_USERID_COOKIE_SAMESITE_LAX     0x0010
-#define NGX_HTTP_USERID_COOKIE_SAMESITE_NONE    0x0020
+#define NGX_HTTP_USERID_COOKIE_OFF              0x0002
+#define NGX_HTTP_USERID_COOKIE_SECURE           0x0004
+#define NGX_HTTP_USERID_COOKIE_HTTPONLY         0x0008
+#define NGX_HTTP_USERID_COOKIE_SAMESITE         0x0010
+#define NGX_HTTP_USERID_COOKIE_SAMESITE_STRICT  0x0020
+#define NGX_HTTP_USERID_COOKIE_SAMESITE_LAX     0x0040
+#define NGX_HTTP_USERID_COOKIE_SAMESITE_NONE    0x0080
 
 /* 31 Dec 2037 23:55:55 GMT */
 #define NGX_HTTP_USERID_MAX_EXPIRES  2145916555
@@ -97,6 +98,7 @@ static ngx_conf_enum_t  ngx_http_userid_
 
 
 static ngx_conf_bitmask_t  ngx_http_userid_flags[] = {
+    { ngx_string("off"), NGX_HTTP_USERID_COOKIE_OFF },
     { ngx_string("secure"), NGX_HTTP_USERID_COOKIE_SECURE },
     { ngx_string("httponly"), NGX_HTTP_USERID_COOKIE_HTTPONLY },
     { ngx_string("samesite=strict"),
@@ -752,7 +754,7 @@ ngx_http_userid_merge_conf(ngx_conf_t *c
                               NGX_HTTP_USERID_OFF);
 
     ngx_conf_merge_bitmask_value(conf->flags, prev->flags,
-                                 NGX_CONF_BITMASK_SET);
+                            (NGX_CONF_BITMASK_SET|NGX_HTTP_USERID_COOKIE_OFF));
 
     ngx_conf_merge_str_value(conf->name, prev->name, "uid");
     ngx_conf_merge_str_value(conf->domain, prev->domain, "");

From mdounin at mdounin.ru  Tue Sep 29 13:18:12 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 29 Sep 2020 13:18:12 +0000
Subject: [nginx] Proxy: error checking for array init,
 missed in 7716:d6a5e14aa3e4.
Message-ID: <hg.a88384c69d1e.1601385492.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/a88384c69d1e
branches:  
changeset: 7720:a88384c69d1e
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Sep 29 15:54:09 2020 +0300
description:
Proxy: error checking for array init, missed in 7716:d6a5e14aa3e4.

Found by Coverity (CID 1467637).

diffstat:

 src/http/modules/ngx_http_proxy_module.c |  4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diffs (14 lines):

diff -r c0cacad62cc8 -r a88384c69d1e src/http/modules/ngx_http_proxy_module.c
--- a/src/http/modules/ngx_http_proxy_module.c	Tue Sep 29 15:52:18 2020 +0300
+++ b/src/http/modules/ngx_http_proxy_module.c	Tue Sep 29 15:54:09 2020 +0300
@@ -2679,7 +2679,9 @@ ngx_http_proxy_rewrite_cookie(ngx_http_r
     ngx_keyval_t               *attr;
     ngx_http_proxy_loc_conf_t  *plcf;
 
-    ngx_array_init(&attrs, r->pool, 2, sizeof(ngx_keyval_t));
+    if (ngx_array_init(&attrs, r->pool, 2, sizeof(ngx_keyval_t)) != NGX_OK) {
+        return NGX_ERROR;
+    }
 
     if (ngx_http_proxy_parse_cookie(&h->value, &attrs) != NGX_OK) {
         return NGX_ERROR;

From mdounin at mdounin.ru  Tue Sep 29 13:18:15 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 29 Sep 2020 13:18:15 +0000
Subject: [nginx] Updated OpenSSL used for win32 builds.
Message-ID: <hg.5975be6c97f8.1601385495.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/5975be6c97f8
branches:  
changeset: 7721:5975be6c97f8
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Sep 29 15:56:16 2020 +0300
description:
Updated OpenSSL used for win32 builds.

diffstat:

 misc/GNUmakefile |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r a88384c69d1e -r 5975be6c97f8 misc/GNUmakefile
--- a/misc/GNUmakefile	Tue Sep 29 15:54:09 2020 +0300
+++ b/misc/GNUmakefile	Tue Sep 29 15:56:16 2020 +0300
@@ -6,7 +6,7 @@ TEMP =		tmp
 
 CC =		cl
 OBJS =		objs.msvc8
-OPENSSL =	openssl-1.1.1g
+OPENSSL =	openssl-1.1.1h
 ZLIB =		zlib-1.2.11
 PCRE =		pcre-8.44
 

From xeioex at nginx.com  Tue Sep 29 13:34:27 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 29 Sep 2020 13:34:27 +0000
Subject: [njs] Version 0.4.4.
Message-ID: <hg.fdfd580b0dd6.1601386467.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/fdfd580b0dd6
branches:  
changeset: 1531:fdfd580b0dd6
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Tue Sep 29 13:31:47 2020 +0000
description:
Version 0.4.4.

diffstat:

 CHANGES |  55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 55 insertions(+), 0 deletions(-)

diffs (62 lines):

diff -r f1ed239edc7d -r fdfd580b0dd6 CHANGES
--- a/CHANGES	Mon Sep 28 17:45:44 2020 +0000
+++ b/CHANGES	Tue Sep 29 13:31:47 2020 +0000
@@ -1,3 +1,58 @@
+
+Changes with njs 0.4.4                                           29 Sep 2020
+
+    nginx modules:
+
+    *) Bugfix: fixed location merge.
+
+    *) Bugfix: fixed r.httpVersion for HTTP/2.
+
+    Core:
+
+    *) Feature: added support for numeric separators (ES12).
+
+    *) Feature: added remaining methods for %TypedArray%.prototype.
+       The following methods were added: every(), filter(), find(),
+       findIndex(), forEach(), includes(), indexOf(), lastIndexOf(),
+       map(), reduce(), reduceRight(), reverse(), some().
+
+    *) Feature: added %TypedArray% remaining methods.
+       The following methods were added: from(), of().
+
+    *) Feature: added DataView object.
+
+    *) Feature: added Buffer object implementation.
+
+    *) Feature: added support for ArrayBuffer in
+       TextDecoder.prototype.decode().
+
+    *) Feature: added support for Buffer object in "crypto" methods.
+
+    *) Feature: added support for Buffer object in "fs" methods.
+
+    *) Change: Hash.prototype.digest() and Hmac.prototype.digest()
+       now return a Buffer instance instead of a byte string when
+       encoding is not provided.
+
+    *) Change: fs.readFile() and friends now return a Buffer instance
+       instead of a byte string when encoding is not provided.
+
+    *) Bugfix: fixed function "prototype" property handler while
+       setting.
+
+    *) Bugfix: fixed function "constructor" property handler while
+       setting.
+
+    *) Bugfix: fixed String.prototype.indexOf() for byte strings.
+
+    *) Bugfix: fixed RegExpBuiltinExec() with a global flag and
+       byte strings.
+
+    *) Bugfix: fixed RegExp.prototype[Symbol.replace] when the
+       replacement value is a function.
+
+    *) Bugfix: fixed TextDecoder.prototype.decode() with non-zero
+       TypedArray offset.
 
 Changes with njs 0.4.3                                           11 Aug 2020
 

From xeioex at nginx.com  Tue Sep 29 13:34:29 2020
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 29 Sep 2020 13:34:29 +0000
Subject: [njs] Added tag 0.4.4 for changeset fdfd580b0dd6
Message-ID: <hg.e0c26ff9bd97.1601386469.5965299922797593991@dev.nginx>

details:   https://hg.nginx.org/njs/rev/e0c26ff9bd97
branches:  
changeset: 1532:e0c26ff9bd97
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Tue Sep 29 13:32:37 2020 +0000
description:
Added tag 0.4.4 for changeset fdfd580b0dd6

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff -r fdfd580b0dd6 -r e0c26ff9bd97 .hgtags
--- a/.hgtags	Tue Sep 29 13:31:47 2020 +0000
+++ b/.hgtags	Tue Sep 29 13:32:37 2020 +0000
@@ -37,3 +37,4 @@ 6144aafa1472fbdf79bc9d1f858555938ee08452
 9400790bf53843001f94c77b47bc99b05518af78 0.4.1
 b409e86fd02a6f2cb3d741a41b6562471e1b66ef 0.4.2
 1ada1061a040e5cd5ec55744bfa916dfc6744e4c 0.4.3
+fdfd580b0dd617a884ed9287d98341ebef03ee9f 0.4.4

From mdounin at mdounin.ru  Tue Sep 29 14:35:48 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 29 Sep 2020 14:35:48 +0000
Subject: [nginx] nginx-1.19.3-RELEASE
Message-ID: <hg.3cbc2602325f.1601390148.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/3cbc2602325f
branches:  
changeset: 7722:3cbc2602325f
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Sep 29 17:32:10 2020 +0300
description:
nginx-1.19.3-RELEASE

diffstat:

 docs/xml/nginx/changes.xml |  106 +++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 106 insertions(+), 0 deletions(-)

diffs (116 lines):

diff -r 5975be6c97f8 -r 3cbc2602325f docs/xml/nginx/changes.xml
--- a/docs/xml/nginx/changes.xml	Tue Sep 29 15:56:16 2020 +0300
+++ b/docs/xml/nginx/changes.xml	Tue Sep 29 17:32:10 2020 +0300
@@ -5,6 +5,112 @@
 <change_log title="nginx">
 
 
+<changes ver="1.19.3" date="2020-09-29">
+
+<change type="feature">
+<para lang="ru">
+?????? ngx_stream_set_module.
+</para>
+<para lang="en">
+the ngx_stream_set_module.
+</para>
+</change>
+
+<change type="feature">
+<para lang="ru">
+????????? proxy_cookie_flags.
+</para>
+<para lang="en">
+the "proxy_cookie_flags" directive.
+</para>
+</change>
+
+<change type="feature">
+<para lang="ru">
+????????? userid_flags.
+</para>
+<para lang="en">
+the "userid_flags" directive.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+?????????? ?????????? ???????????? stale-if-error
+???????? ???????????, ???? ?????? ????????? ?????
+? ????? 500, 502, 503, 504, 403, 404 ??? 429.
+</para>
+<para lang="en">
+the "stale-if-error" cache control extension
+was erroneously applied if backend returned a response
+with status code 500, 502, 503, 504, 403, 404, or 429.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+???? ?????????????? ???????????
+? ?????? ????????? ?????? ? ??????? ????????? Vary,
+? ????? ????? ?????????? ????????? "[crit] cache file ... has too long header".
+</para>
+<para lang="en">
+"[crit] cache file ... has too long header" messages might appear in logs
+if caching was used
+and the backend returned responses with the "Vary" header line.
+</para>
+</change>
+
+<change type="workaround">
+<para lang="ru">
+??? ????????????? OpenSSL 1.1.1
+? ????? ????? ?????????? ????????? "[crit] SSL_write() failed".
+</para>
+<para lang="en">
+"[crit] SSL_write() failed" messages might appear in logs
+when using OpenSSL 1.1.1.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+? ????? ????? ?????????? ?????????
+"SSL_shutdown() failed (SSL: ... bad write retry)";
+?????? ????????? ? 1.19.2.
+</para>
+<para lang="en">
+"SSL_shutdown() failed (SSL: ... bad write retry)"
+messages might appear in logs;
+the bug had appeared in 1.19.2.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? HTTP/2
+? ??????? ???????? ??? ????????? segmentation fault,
+???? ?????? ? ????? 400 ? ??????? ????????? error_page
+???????????????? ? ???????????? location.
+</para>
+<para lang="en">
+a segmentation fault might occur in a worker process
+when using HTTP/2
+if errors with code 400 were redirected to a proxied location
+using the "error_page" directive.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+?????? ??????? ??? ????????????? HTTP/2 ? ??????????? ? ?????? njs.
+</para>
+<para lang="en">
+socket leak when using HTTP/2 and subrequests in the njs module.
+</para>
+</change>
+
+</changes>
+
+
 <changes ver="1.19.2" date="2020-08-11">
 
 <change type="change">

From mdounin at mdounin.ru  Tue Sep 29 14:35:51 2020
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 29 Sep 2020 14:35:51 +0000
Subject: [nginx] release-1.19.3 tag
Message-ID: <hg.0896ed4ddee4.1601390151.6026610855610030274@dev.nginx>

details:   https://hg.nginx.org/nginx/rev/0896ed4ddee4
branches:  
changeset: 7723:0896ed4ddee4
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Sep 29 17:32:10 2020 +0300
description:
release-1.19.3 tag

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff -r 3cbc2602325f -r 0896ed4ddee4 .hgtags
--- a/.hgtags	Tue Sep 29 17:32:10 2020 +0300
+++ b/.hgtags	Tue Sep 29 17:32:10 2020 +0300
@@ -452,3 +452,4 @@ c44970de01474f6f3e01b0adea85ec1d03e3a5f2
 cbe6ba650211541310618849168631ce0b788f35 release-1.19.0
 062920e2f3bf871ef7a3d8496edec1b3065faf80 release-1.19.1
 a7b46539f507e6c64efa0efda69ad60b6f4ffbce release-1.19.2
+3cbc2602325f0ac08917a4397d76f5155c34b7b1 release-1.19.3