From mdounin at mdounin.ru Wed Aug 1 04:21:33 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 01 Aug 2018 04:21:33 +0000
Subject: [nginx] Dav: fixed ngx_copy_file() to truncate destination file.
Message-ID:
details: http://hg.nginx.org/nginx/rev/3c357206a3b8
branches:
changeset: 7328:3c357206a3b8
user: Maxim Dounin
date: Wed Aug 01 02:11:58 2018 +0300
description:
Dav: fixed ngx_copy_file() to truncate destination file.
Previously, ngx_open_file(NGX_FILE_CREATE_OR_OPEN) was used, resulting
in destination file being partially rewritten if exists. Notably,
this affected WebDAV COPY command (ticket #1576).
diffstat:
src/core/ngx_file.c | 3 +--
1 files changed, 1 insertions(+), 2 deletions(-)
diffs (13 lines):
diff --git a/src/core/ngx_file.c b/src/core/ngx_file.c
--- a/src/core/ngx_file.c
+++ b/src/core/ngx_file.c
@@ -839,8 +839,7 @@ ngx_copy_file(u_char *from, u_char *to,
goto failed;
}
- nfd = ngx_open_file(to, NGX_FILE_WRONLY, NGX_FILE_CREATE_OR_OPEN,
- cf->access);
+ nfd = ngx_open_file(to, NGX_FILE_WRONLY, NGX_FILE_TRUNCATE, cf->access);
if (nfd == NGX_INVALID_FILE) {
ngx_log_error(NGX_LOG_CRIT, cf->log, ngx_errno,
From mdounin at mdounin.ru Wed Aug 1 04:21:35 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 01 Aug 2018 04:21:35 +0000
Subject: [nginx] Dav: changed ngx_copy_file() to preserve access and mtime.
Message-ID:
details: http://hg.nginx.org/nginx/rev/63a76a594dd8
branches:
changeset: 7329:63a76a594dd8
user: Maxim Dounin
date: Wed Aug 01 02:12:11 2018 +0300
description:
Dav: changed ngx_copy_file() to preserve access and mtime.
This fixes wrong permissions and file time after cross-device MOVE
in the DAV module (ticket #1577). Broken in 8101d9101ed8 (0.8.9) when
cross-device copying was introduced in ngx_ext_rename_file().
With this change, ngx_copy_file() always calls ngx_set_file_time(),
either with the time provided, or with the time from the original file.
This is considered acceptable given that copying the file is costly anyway,
and optimizing cases when we do not need to preserve time will require
interface changes.
diffstat:
src/core/ngx_file.c | 22 +++++++++++++---------
1 files changed, 13 insertions(+), 9 deletions(-)
diffs (65 lines):
diff --git a/src/core/ngx_file.c b/src/core/ngx_file.c
--- a/src/core/ngx_file.c
+++ b/src/core/ngx_file.c
@@ -796,10 +796,12 @@ ngx_copy_file(u_char *from, u_char *to,
{
char *buf;
off_t size;
+ time_t time;
size_t len;
ssize_t n;
ngx_fd_t fd, nfd;
ngx_int_t rc;
+ ngx_uint_t access;
ngx_file_info_t fi;
rc = NGX_ERROR;
@@ -814,8 +816,10 @@ ngx_copy_file(u_char *from, u_char *to,
goto failed;
}
- if (cf->size != -1) {
+ if (cf->size != -1 && cf->access != 0 && cf->time != -1) {
size = cf->size;
+ access = cf->access;
+ time = cf->time;
} else {
if (ngx_fd_info(fd, &fi) == NGX_FILE_ERROR) {
@@ -825,7 +829,9 @@ ngx_copy_file(u_char *from, u_char *to,
goto failed;
}
- size = ngx_file_size(&fi);
+ size = (cf->size != -1) ? cf->size : ngx_file_size(&fi);
+ access = cf->access ? cf->access : ngx_file_access(&fi);
+ time = (cf->time != -1) ? cf->time : ngx_file_mtime(&fi);
}
len = cf->buf_size ? cf->buf_size : 65536;
@@ -839,7 +845,7 @@ ngx_copy_file(u_char *from, u_char *to,
goto failed;
}
- nfd = ngx_open_file(to, NGX_FILE_WRONLY, NGX_FILE_TRUNCATE, cf->access);
+ nfd = ngx_open_file(to, NGX_FILE_WRONLY, NGX_FILE_TRUNCATE, access);
if (nfd == NGX_INVALID_FILE) {
ngx_log_error(NGX_LOG_CRIT, cf->log, ngx_errno,
@@ -886,12 +892,10 @@ ngx_copy_file(u_char *from, u_char *to,
size -= n;
}
- if (cf->time != -1) {
- if (ngx_set_file_time(to, nfd, cf->time) != NGX_OK) {
- ngx_log_error(NGX_LOG_ALERT, cf->log, ngx_errno,
- ngx_set_file_time_n " \"%s\" failed", to);
- goto failed;
- }
+ if (ngx_set_file_time(to, nfd, time) != NGX_OK) {
+ ngx_log_error(NGX_LOG_ALERT, cf->log, ngx_errno,
+ ngx_set_file_time_n " \"%s\" failed", to);
+ goto failed;
}
rc = NGX_OK;
From mdounin at mdounin.ru Wed Aug 1 04:21:37 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 01 Aug 2018 04:21:37 +0000
Subject: [nginx] Dav: changed COPY of a file to preserve access mask.
Message-ID:
details: http://hg.nginx.org/nginx/rev/8e7a5de61664
branches:
changeset: 7330:8e7a5de61664
user: Maxim Dounin
date: Wed Aug 01 02:12:21 2018 +0300
description:
Dav: changed COPY of a file to preserve access mask.
The behaviour is now in line with COPY of a directory with contents,
which preserves access masks on individual files, as well as the "cp"
command.
Requested by Roman Arutyunyan.
diffstat:
src/http/modules/ngx_http_dav_module.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diffs (12 lines):
diff --git a/src/http/modules/ngx_http_dav_module.c b/src/http/modules/ngx_http_dav_module.c
--- a/src/http/modules/ngx_http_dav_module.c
+++ b/src/http/modules/ngx_http_dav_module.c
@@ -845,7 +845,7 @@ overwrite_done:
cf.size = ngx_file_size(&fi);
cf.buf_size = 0;
- cf.access = dlcf->access;
+ cf.access = ngx_file_access(&fi);
cf.time = ngx_file_mtime(&fi);
cf.log = r->connection->log;
From robinsonnie at gmail.com Wed Aug 1 07:45:10 2018
From: robinsonnie at gmail.com (Robinson Nie)
Date: Wed, 1 Aug 2018 15:45:10 +0800
Subject: [PATCH] ngx_event_pipe: add judgment conditions for downstream
errors.
In-Reply-To: <20180731145202.GH56558@mdounin.ru>
References:
<20180730172110.GA56558@mdounin.ru>
<20180731145202.GH56558@mdounin.ru>
Message-ID:
When a downstream error occurs, the reading upstream(
p->upstream->recv_chain() ) is executed not only once but multiple
times until it triggers p->upstream->read->ready becomes 0, or p->
upstream_eof / p-> upstream_error / p -> upstream_done becomre !0.
This patch wants to avoid these invalid reading upstreams and finalize
upstream & downstream as soon as possible.
Thank you for your advice!
2018-07-31 22:52 GMT+08:00 Maxim Dounin :
> Hello!
>
> On Tue, Jul 31, 2018 at 11:35:35AM +0800, Robinson Nie wrote:
>
>> Yes, you are right.
>>
>> This patch is somewhat ambiguous, so the updated patch is as follows:
>>
>> # HG changeset patch
>> # User dapeng.ndp
>> # Date 1533007686 -28800
>> # Tue Jul 31 11:28:06 2018 +0800
>> # Node ID 5b7a4bba5238658b9b256cea013d07300c29540d
>> # Parent f7e79596baf209151682f2f7d220161c034657ac
>> ngx_event_pipe: When the downstream error occurs, avoid further
>> reading upstream.
>>
>> When the proxy is turned on and there is no need to cache data locally. If the
>> downstream is wrong(p->downstream_error is set), avoid further reading upstream,
>> and finalize upstream & downstream as soon as possible.
>>
>> diff -r f7e79596baf2 -r 5b7a4bba5238 src/event/ngx_event_pipe.c
>> --- a/src/event/ngx_event_pipe.c Tue Jul 24 18:46:54 2018 +0300
>> +++ b/src/event/ngx_event_pipe.c Tue Jul 31 11:28:06 2018 +0800
>> @@ -112,6 +112,10 @@
>> return NGX_OK;
>> }
>>
>> + if (!p->cacheable && p->downstream_error) {
>> + return NGX_OK;
>> + }
>> +
>> #if (NGX_THREADS)
>>
>> if (p->aio) {
>>
>>
>> Please help review, thanks!
>
> As far as I see, this patch will prevent correct operation with
> threaded writes.
>
> As previously suggested, you may want to first elaborate on what
> problem you are trying to fix. With the current code, the
> decision on whether nginx needs to stop reading from the upstream in
> case of downstream error is made by the caller (in the
> ngx_http_upstream_process_request() in case of the upstream
> module). While moving this decision from the caller to the event
> pipe code is possible, it certainly have some downsides. And the
> possible benefits - saving one iteration of reading from the
> upstream till it blocks - might not be a good enough reason for
> the change.
>
> I suspect that what you are trying to address is actually worker
> monopolization while working with fast clients / upstream servers,
> as described here:
>
> https://trac.nginx.org/nginx/ticket/1431
>
> With the downstream error such monopolization is more likely since
> there is no client to block on, and "one iteration of reading from
> the upstream till it blocks" might be actually something important
> enough. This problem is not limited to the downstream error case
> though, and not even to the event pipe, and addressing it via
> trying to improve some corner cases in the event pipe code looks
> wrong.
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
From mdounin at mdounin.ru Wed Aug 1 13:20:22 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 1 Aug 2018 16:20:22 +0300
Subject: [PATCH] ngx_event_pipe: add judgment conditions for downstream
errors.
In-Reply-To:
References:
<20180730172110.GA56558@mdounin.ru>
<20180731145202.GH56558@mdounin.ru>
Message-ID: <20180801132022.GM56558@mdounin.ru>
Hello!
On Wed, Aug 01, 2018 at 03:45:10PM +0800, Robinson Nie wrote:
> When a downstream error occurs, the reading upstream(
> p->upstream->recv_chain() ) is executed not only once but multiple
> times until it triggers p->upstream->read->ready becomes 0, or p->
> upstream_eof / p-> upstream_error / p -> upstream_done becomre !0.
I wrote "reading from the upstream till it blocks", and
this is what "read->ready becomes 0" means.
--
Maxim Dounin
http://mdounin.ru/
From xeioex at nginx.com Wed Aug 1 15:37:32 2018
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 01 Aug 2018 15:37:32 +0000
Subject: [njs] Returning an internal error for not-implemented Function().
Message-ID:
details: http://hg.nginx.org/njs/rev/4586ae051111
branches:
changeset: 581:4586ae051111
user: Dmitry Volyntsev
date: Wed Aug 01 18:37:15 2018 +0300
description:
Returning an internal error for not-implemented Function().
This fixes #41 issue on GitHub.
diffstat:
njs/njs_function.c | 2 ++
njs/test/njs_unit_test.c | 3 +++
2 files changed, 5 insertions(+), 0 deletions(-)
diffs (25 lines):
diff -r 72aa4d9aa606 -r 4586ae051111 njs/njs_function.c
--- a/njs/njs_function.c Wed Aug 01 18:05:01 2018 +0300
+++ b/njs/njs_function.c Wed Aug 01 18:37:15 2018 +0300
@@ -457,6 +457,8 @@ njs_ret_t
njs_function_constructor(njs_vm_t *vm, njs_value_t *args, nxt_uint_t nargs,
njs_index_t unused)
{
+ njs_internal_error(vm, "Not implemented");
+
return NXT_ERROR;
}
diff -r 72aa4d9aa606 -r 4586ae051111 njs/test/njs_unit_test.c
--- a/njs/test/njs_unit_test.c Wed Aug 01 18:05:01 2018 +0300
+++ b/njs/test/njs_unit_test.c Wed Aug 01 18:37:15 2018 +0300
@@ -6518,6 +6518,9 @@ static njs_unit_test_t njs_test[] =
{ nxt_string("function f() {} f.__proto__ === Function.prototype"),
nxt_string("true") },
+ { nxt_string("Function()"),
+ nxt_string("InternalError: Not implemented") },
+
{ nxt_string("RegExp()"),
nxt_string("/(?:)/") },
From xeioex at nginx.com Wed Aug 1 15:37:32 2018
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Wed, 01 Aug 2018 15:37:32 +0000
Subject: [njs] Adding const qualifiers njs_value_t arguments of public methods.
Message-ID:
details: http://hg.nginx.org/njs/rev/72aa4d9aa606
branches:
changeset: 580:72aa4d9aa606
user: Dmitry Volyntsev
date: Wed Aug 01 18:05:01 2018 +0300
description:
Adding const qualifiers njs_value_t arguments of public methods.
This allows to use the result of safe njs_arg() macro in the
methods.
diffstat:
njs/njs.c | 10 +++++-----
njs/njs.h | 41 ++++++++++++++++++++---------------------
njs/njs_extern.c | 2 +-
njs/njs_function.c | 2 +-
njs/njs_function.h | 2 +-
njs/njs_vm.c | 26 +++++++++++++-------------
6 files changed, 41 insertions(+), 42 deletions(-)
diffs (284 lines):
diff -r c85b07c96e27 -r 72aa4d9aa606 njs/njs.c
--- a/njs/njs.c Tue Jul 31 21:11:44 2018 +0300
+++ b/njs/njs.c Wed Aug 01 18:05:01 2018 +0300
@@ -444,7 +444,7 @@ njs_vm_init(njs_vm_t *vm)
nxt_int_t
-njs_vm_call(njs_vm_t *vm, njs_function_t *function, njs_value_t *args,
+njs_vm_call(njs_vm_t *vm, njs_function_t *function, const njs_value_t *args,
nxt_uint_t nargs)
{
u_char *current;
@@ -531,7 +531,7 @@ njs_vm_pending(njs_vm_t *vm)
nxt_int_t
njs_vm_post_event(njs_vm_t *vm, njs_vm_event_t vm_event,
- njs_value_t *args, nxt_uint_t nargs)
+ const njs_value_t *args, nxt_uint_t nargs)
{
njs_event_t *event;
@@ -654,9 +654,9 @@ njs_vm_retval(njs_vm_t *vm)
nxt_noinline void
-njs_vm_retval_set(njs_vm_t *vm, njs_value_t *value)
+njs_vm_retval_set(njs_vm_t *vm, const njs_value_t *value)
{
- vm->retval = *(njs_value_t *) value;
+ vm->retval = *value;
}
@@ -680,7 +680,7 @@ njs_ret_t njs_vm_retval_to_ext_string(nj
njs_value_t *
-njs_vm_object_prop(njs_vm_t *vm, njs_value_t *value, const nxt_str_t *key)
+njs_vm_object_prop(njs_vm_t *vm, const njs_value_t *value, const nxt_str_t *key)
{
nxt_int_t ret;
njs_object_prop_t *prop;
diff -r c85b07c96e27 -r 72aa4d9aa606 njs/njs.h
--- a/njs/njs.h Tue Jul 31 21:11:44 2018 +0300
+++ b/njs/njs.h Wed Aug 01 18:05:01 2018 +0300
@@ -48,8 +48,7 @@ typedef struct {
extern const njs_value_t njs_value_void;
#define njs_arg(args, nargs, n) \
- (njs_value_t *) ((n < nargs) ? njs_argument(args, n) \
- : &njs_value_void)
+ ((n < nargs) ? njs_argument(args, n) : &njs_value_void)
#define njs_value_assign(dst, src) \
*((njs_opaque_value_t *) dst) = *((njs_opaque_value_t *) src);
@@ -162,7 +161,7 @@ NXT_EXPORT void njs_vm_destroy(njs_vm_t
NXT_EXPORT nxt_int_t njs_vm_compile(njs_vm_t *vm, u_char **start, u_char *end);
NXT_EXPORT njs_vm_t *njs_vm_clone(njs_vm_t *vm, njs_external_ptr_t external);
NXT_EXPORT nxt_int_t njs_vm_call(njs_vm_t *vm, njs_function_t *function,
- njs_value_t *args, nxt_uint_t nargs);
+ const njs_value_t *args, nxt_uint_t nargs);
NXT_EXPORT njs_vm_event_t njs_vm_add_event(njs_vm_t *vm,
njs_function_t *function, njs_host_event_t host_ev,
@@ -170,7 +169,7 @@ NXT_EXPORT njs_vm_event_t njs_vm_add_eve
NXT_EXPORT void njs_vm_del_event(njs_vm_t *vm, njs_vm_event_t vm_event);
NXT_EXPORT nxt_int_t njs_vm_pending(njs_vm_t *vm);
NXT_EXPORT nxt_int_t njs_vm_post_event(njs_vm_t *vm, njs_vm_event_t vm_event,
- njs_value_t *args, nxt_uint_t nargs);
+ const njs_value_t *args, nxt_uint_t nargs);
NXT_EXPORT nxt_int_t njs_vm_run(njs_vm_t *vm);
@@ -179,7 +178,7 @@ NXT_EXPORT const njs_extern_t *njs_vm_ex
NXT_EXPORT nxt_int_t njs_vm_external_create(njs_vm_t *vm,
njs_value_t *value, const njs_extern_t *proto, njs_external_ptr_t object);
NXT_EXPORT nxt_int_t njs_vm_external_bind(njs_vm_t *vm,
- const nxt_str_t *var_name, njs_value_t *value);
+ const nxt_str_t *var_name, const njs_value_t *value);
NXT_EXPORT njs_external_ptr_t njs_vm_external(njs_vm_t *vm,
const njs_value_t *value);
@@ -188,7 +187,7 @@ NXT_EXPORT nxt_array_t *njs_vm_completio
NXT_EXPORT njs_function_t *njs_vm_function(njs_vm_t *vm, nxt_str_t *name);
NXT_EXPORT njs_value_t *njs_vm_retval(njs_vm_t *vm);
-NXT_EXPORT void njs_vm_retval_set(njs_vm_t *vm, njs_value_t *value);
+NXT_EXPORT void njs_vm_retval_set(njs_vm_t *vm, const njs_value_t *value);
NXT_EXPORT u_char * njs_string_alloc(njs_vm_t *vm, njs_value_t *value,
uint32_t size, uint32_t length);
@@ -196,7 +195,7 @@ NXT_EXPORT njs_ret_t njs_string_create(n
u_char *start, uint32_t size, uint32_t length);
NXT_EXPORT nxt_int_t njs_value_string_copy(njs_vm_t *vm, nxt_str_t *retval,
- njs_value_t *value, uintptr_t *next);
+ const njs_value_t *value, uintptr_t *next);
NXT_EXPORT njs_ret_t njs_vm_value_to_ext_string(njs_vm_t *vm, nxt_str_t *dst,
const njs_value_t *src, nxt_uint_t handle_exception);
@@ -212,24 +211,24 @@ NXT_EXPORT void njs_value_data_set(njs_v
NXT_EXPORT void njs_value_error_set(njs_vm_t *vm, njs_value_t *value,
const char *fmt, ...);
-NXT_EXPORT uint8_t njs_value_bool(njs_value_t *value);
-NXT_EXPORT double njs_value_number(njs_value_t *value);
-NXT_EXPORT void *njs_value_data(njs_value_t *value);
-NXT_EXPORT njs_function_t *njs_value_function(njs_value_t *value);
+NXT_EXPORT uint8_t njs_value_bool(const njs_value_t *value);
+NXT_EXPORT double njs_value_number(const njs_value_t *value);
+NXT_EXPORT void *njs_value_data(const njs_value_t *value);
+NXT_EXPORT njs_function_t *njs_value_function(const njs_value_t *value);
-NXT_EXPORT nxt_int_t njs_value_is_null(njs_value_t *value);
-NXT_EXPORT nxt_int_t njs_value_is_void(njs_value_t *value);
-NXT_EXPORT nxt_int_t njs_value_is_boolean(njs_value_t *value);
-NXT_EXPORT nxt_int_t njs_value_is_number(njs_value_t *value);
-NXT_EXPORT nxt_int_t njs_value_is_valid_number(njs_value_t *value);
-NXT_EXPORT nxt_int_t njs_value_is_string(njs_value_t *value);
-NXT_EXPORT nxt_int_t njs_value_is_object(njs_value_t *value);
-NXT_EXPORT nxt_int_t njs_value_is_function(njs_value_t *value);
+NXT_EXPORT nxt_int_t njs_value_is_null(const njs_value_t *value);
+NXT_EXPORT nxt_int_t njs_value_is_void(const njs_value_t *value);
+NXT_EXPORT nxt_int_t njs_value_is_boolean(const njs_value_t *value);
+NXT_EXPORT nxt_int_t njs_value_is_number(const njs_value_t *value);
+NXT_EXPORT nxt_int_t njs_value_is_valid_number(const njs_value_t *value);
+NXT_EXPORT nxt_int_t njs_value_is_string(const njs_value_t *value);
+NXT_EXPORT nxt_int_t njs_value_is_object(const njs_value_t *value);
+NXT_EXPORT nxt_int_t njs_value_is_function(const njs_value_t *value);
NXT_EXPORT njs_ret_t njs_vm_value_dump(njs_vm_t *vm, nxt_str_t *retval,
const njs_value_t *value, nxt_uint_t indent);
-NXT_EXPORT njs_value_t *njs_vm_object_prop(njs_vm_t *vm, njs_value_t *value,
- const nxt_str_t *key);
+NXT_EXPORT njs_value_t *njs_vm_object_prop(njs_vm_t *vm,
+ const njs_value_t *value, const nxt_str_t *key);
extern const nxt_mem_proto_t njs_vm_mem_cache_pool_proto;
diff -r c85b07c96e27 -r 72aa4d9aa606 njs/njs_extern.c
--- a/njs/njs_extern.c Tue Jul 31 21:11:44 2018 +0300
+++ b/njs/njs_extern.c Wed Aug 01 18:05:01 2018 +0300
@@ -193,7 +193,7 @@ njs_vm_external_create(njs_vm_t *vm, njs
nxt_int_t
njs_vm_external_bind(njs_vm_t *vm, const nxt_str_t *var_name,
- njs_value_t *value)
+ const njs_value_t *value)
{
nxt_int_t ret;
njs_extern_value_t *ev;
diff -r c85b07c96e27 -r 72aa4d9aa606 njs/njs_function.c
--- a/njs/njs_function.c Tue Jul 31 21:11:44 2018 +0300
+++ b/njs/njs_function.c Wed Aug 01 18:05:01 2018 +0300
@@ -144,7 +144,7 @@ njs_function_native_frame(njs_vm_t *vm,
nxt_noinline njs_ret_t
njs_function_frame(njs_vm_t *vm, njs_function_t *function,
- const njs_value_t *this, njs_value_t *args, nxt_uint_t nargs,
+ const njs_value_t *this, const njs_value_t *args, nxt_uint_t nargs,
nxt_bool_t ctor)
{
size_t size;
diff -r c85b07c96e27 -r 72aa4d9aa606 njs/njs_function.h
--- a/njs/njs_function.h Tue Jul 31 21:11:44 2018 +0300
+++ b/njs/njs_function.h Wed Aug 01 18:05:01 2018 +0300
@@ -158,7 +158,7 @@ njs_ret_t njs_function_native_frame(njs_
const njs_value_t *this, njs_value_t *args, nxt_uint_t nargs,
size_t reserve, nxt_bool_t ctor);
njs_ret_t njs_function_frame(njs_vm_t *vm, njs_function_t *function,
- const njs_value_t *this, njs_value_t *args, nxt_uint_t nargs,
+ const njs_value_t *this, const njs_value_t *args, nxt_uint_t nargs,
nxt_bool_t ctor);
njs_ret_t njs_function_call(njs_vm_t *vm, njs_index_t retval, size_t advance);
diff -r c85b07c96e27 -r 72aa4d9aa606 njs/njs_vm.c
--- a/njs/njs_vm.c Tue Jul 31 21:11:44 2018 +0300
+++ b/njs/njs_vm.c Wed Aug 01 18:05:01 2018 +0300
@@ -3521,63 +3521,63 @@ memory_error:
nxt_noinline uint8_t
-njs_value_bool(njs_value_t *value)
+njs_value_bool(const njs_value_t *value)
{
return value->data.truth;
}
nxt_noinline double
-njs_value_number(njs_value_t *value)
+njs_value_number(const njs_value_t *value)
{
return value->data.u.number;
}
nxt_noinline void *
-njs_value_data(njs_value_t *value)
+njs_value_data(const njs_value_t *value)
{
return value->data.u.data;
}
nxt_noinline njs_function_t *
-njs_value_function(njs_value_t *value)
+njs_value_function(const njs_value_t *value)
{
return value->data.u.function;
}
nxt_noinline nxt_int_t
-njs_value_is_null(njs_value_t *value)
+njs_value_is_null(const njs_value_t *value)
{
return njs_is_null(value);
}
nxt_noinline nxt_int_t
-njs_value_is_void(njs_value_t *value)
+njs_value_is_void(const njs_value_t *value)
{
return njs_is_void(value);
}
nxt_noinline nxt_int_t
-njs_value_is_boolean(njs_value_t *value)
+njs_value_is_boolean(const njs_value_t *value)
{
return njs_is_boolean(value);
}
nxt_noinline nxt_int_t
-njs_value_is_number(njs_value_t *value)
+njs_value_is_number(const njs_value_t *value)
{
return njs_is_number(value);
}
nxt_noinline nxt_int_t
-njs_value_is_valid_number(njs_value_t *value)
+njs_value_is_valid_number(const njs_value_t *value)
{
return njs_is_number(value)
&& !isnan(value->data.u.number)
@@ -3586,28 +3586,28 @@ njs_value_is_valid_number(njs_value_t *v
nxt_noinline nxt_int_t
-njs_value_is_string(njs_value_t *value)
+njs_value_is_string(const njs_value_t *value)
{
return njs_is_string(value);
}
nxt_noinline nxt_int_t
-njs_value_is_object(njs_value_t *value)
+njs_value_is_object(const njs_value_t *value)
{
return njs_is_object(value);
}
nxt_noinline nxt_int_t
-njs_value_is_function(njs_value_t *value)
+njs_value_is_function(const njs_value_t *value)
{
return njs_is_function(value);
}
nxt_int_t
-njs_value_string_copy(njs_vm_t *vm, nxt_str_t *retval, njs_value_t *value,
+njs_value_string_copy(njs_vm_t *vm, nxt_str_t *retval, const njs_value_t *value,
uintptr_t *next)
{
uintptr_t n;
From xeioex at nginx.com Thu Aug 2 16:40:18 2018
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Thu, 02 Aug 2018 16:40:18 +0000
Subject: [njs] Setting exception values where appropriate.
Message-ID:
details: http://hg.nginx.org/njs/rev/d2cbea77122c
branches:
changeset: 582:d2cbea77122c
user: Dmitry Volyntsev
date: Thu Aug 02 19:39:55 2018 +0300
description:
Setting exception values where appropriate.
This fixes #42 issue on Github.
diffstat:
njs/njs_array.c | 21 ++++++++++++++++-----
njs/njs_crypto.c | 27 ++++++++++-----------------
njs/njs_date.c | 7 ++++---
njs/njs_error.c | 22 ++++++++--------------
njs/njs_extern.c | 15 ++++++++++++---
njs/njs_fs.c | 30 ++++++++++--------------------
njs/njs_function.c | 14 ++++++++++++--
njs/njs_json.c | 2 --
njs/njs_object.c | 28 ++++++++++++++++++++++------
njs/njs_regexp.c | 18 +++++++++++++++---
njs/njs_string.c | 8 ++------
njs/njs_time.c | 1 +
njs/njs_variable.c | 5 +++++
njs/njs_vm.c | 6 ++++++
njs/test/njs_unit_test.c | 17 +++++++++++++----
15 files changed, 136 insertions(+), 85 deletions(-)
diffs (901 lines):
diff -r 4586ae051111 -r d2cbea77122c njs/njs_array.c
--- a/njs/njs_array.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_array.c Thu Aug 02 19:39:55 2018 +0300
@@ -109,20 +109,24 @@ static njs_ret_t njs_array_prototype_sor
nxt_noinline njs_array_t *
njs_array_alloc(njs_vm_t *vm, uint32_t length, uint32_t spare)
{
- uint32_t size;
+ size_t size;
njs_array_t *array;
array = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_array_t));
if (nxt_slow_path(array == NULL)) {
- return NULL;
+ goto memory_error;
}
size = length + spare;
+ if (nxt_slow_path(size * sizeof(njs_value_t) < size)) {
+ goto memory_error;
+ }
+
array->data = nxt_mem_cache_align(vm->mem_cache_pool, sizeof(njs_value_t),
size * sizeof(njs_value_t));
if (nxt_slow_path(array->data == NULL)) {
- return NULL;
+ goto memory_error;
}
array->start = array->data;
@@ -136,6 +140,12 @@ njs_array_alloc(njs_vm_t *vm, uint32_t l
array->length = length;
return array;
+
+memory_error:
+
+ njs_memory_error(vm);
+
+ return NULL;
}
@@ -194,6 +204,7 @@ njs_array_expand(njs_vm_t *vm, njs_array
start = nxt_mem_cache_align(vm->mem_cache_pool, sizeof(njs_value_t),
(prepend + size) * sizeof(njs_value_t));
if (nxt_slow_path(start == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -230,7 +241,7 @@ njs_array_constructor(njs_vm_t *vm, njs_
size = (uint32_t) num;
if ((double) size != num) {
- njs_range_error(vm, NULL);
+ njs_range_error(vm, "Invalid array length");
return NXT_ERROR;
}
@@ -393,7 +404,6 @@ njs_array_prototype_length(njs_vm_t *vm,
if (size > 0) {
ret = njs_array_expand(vm, array, 0, size);
if (nxt_slow_path(ret != NXT_OK)) {
- njs_memory_error(vm);
return NJS_ERROR;
}
@@ -846,6 +856,7 @@ njs_array_prototype_join(njs_vm_t *vm, n
values = nxt_mem_cache_align(vm->mem_cache_pool, sizeof(njs_value_t),
sizeof(njs_value_t) * max);
if (nxt_slow_path(values == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_crypto.c
--- a/njs/njs_crypto.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_crypto.c Thu Aug 02 19:39:55 2018 +0300
@@ -142,9 +142,12 @@ njs_crypto_object_value_alloc(njs_vm_t *
ov->object.extensible = 1;
ov->object.__proto__ = &vm->prototypes[proto].object;
+ return ov;
}
- return ov;
+ njs_memory_error(vm);
+
+ return NULL;
}
@@ -171,12 +174,13 @@ njs_crypto_create_hash(njs_vm_t *vm, njs
hash = njs_crypto_object_value_alloc(vm, NJS_PROTOTYPE_CRYPTO_HASH);
if (nxt_slow_path(hash == NULL)) {
- goto memory_error;
+ return NJS_ERROR;
}
dgst = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_digest_t));
if (nxt_slow_path(dgst == NULL)) {
- goto memory_error;
+ njs_memory_error(vm);
+ return NJS_ERROR;
}
dgst->alg = alg;
@@ -190,12 +194,6 @@ njs_crypto_create_hash(njs_vm_t *vm, njs
vm->retval.data.truth = 1;
return NJS_OK;
-
-memory_error:
-
- njs_memory_error(vm);
-
- return NJS_ERROR;
}
@@ -412,7 +410,8 @@ njs_crypto_create_hmac(njs_vm_t *vm, njs
ctx = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_hmac_t));
if (nxt_slow_path(ctx == NULL)) {
- goto memory_error;
+ njs_memory_error(vm);
+ return NJS_ERROR;
}
ctx->alg = alg;
@@ -443,7 +442,7 @@ njs_crypto_create_hmac(njs_vm_t *vm, njs
hmac = njs_crypto_object_value_alloc(vm, NJS_PROTOTYPE_CRYPTO_HMAC);
if (nxt_slow_path(hmac == NULL)) {
- goto memory_error;
+ return NJS_ERROR;
}
njs_value_data_set(&hmac->value, ctx);
@@ -453,12 +452,6 @@ njs_crypto_create_hmac(njs_vm_t *vm, njs
vm->retval.data.truth = 1;
return NJS_OK;
-
-memory_error:
-
- njs_memory_error(vm);
-
- return NJS_ERROR;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_date.c
--- a/njs/njs_date.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_date.c Thu Aug 02 19:39:55 2018 +0300
@@ -129,6 +129,7 @@ njs_date_constructor(njs_vm_t *vm, njs_v
date = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_date_t));
if (nxt_slow_path(date == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -1052,9 +1053,9 @@ njs_date_to_string(njs_vm_t *vm, njs_val
return njs_string_new(vm, retval, buf, size, size);
}
- njs_range_error(vm, NULL);
-
- return NXT_ERROR;
+ vm->retval = njs_string_invalid_date;
+
+ return NXT_OK;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_error.c
--- a/njs/njs_error.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_error.c Thu Aug 02 19:39:55 2018 +0300
@@ -36,23 +36,15 @@ njs_exception_error_create(njs_vm_t *vm,
ret = njs_string_new(vm, &string, (const u_char *) buf, size, size);
if (nxt_slow_path(ret != NXT_OK)) {
- goto memory_error;
+ return;
}
error = njs_error_alloc(vm, type, NULL, &string);
- if (nxt_slow_path(error == NULL)) {
- goto memory_error;
+ if (nxt_fast_path(error != NULL)) {
+ vm->retval.data.u.object = error;
+ vm->retval.type = type;
+ vm->retval.data.truth = 1;
}
-
- vm->retval.data.u.object = error;
- vm->retval.type = type;
- vm->retval.data.truth = 1;
-
- return;
-
-memory_error:
-
- njs_memory_error(vm);
}
@@ -67,6 +59,7 @@ njs_error_alloc(njs_vm_t *vm, njs_value_
error = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_object_t));
if (nxt_slow_path(error == NULL)) {
+ njs_memory_error(vm);
return NULL;
}
@@ -94,6 +87,7 @@ njs_error_alloc(njs_vm_t *vm, njs_value_
ret = nxt_lvlhsh_insert(&error->hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
return NULL;
}
}
@@ -114,6 +108,7 @@ njs_error_alloc(njs_vm_t *vm, njs_value_
ret = nxt_lvlhsh_insert(&error->hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
return NULL;
}
}
@@ -138,7 +133,6 @@ njs_error_create(njs_vm_t *vm, njs_value
error = njs_error_alloc(vm, type, NULL, value);
if (nxt_slow_path(error == NULL)) {
- njs_memory_error(vm);
return NXT_ERROR;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_extern.c
--- a/njs/njs_extern.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_extern.c Thu Aug 02 19:39:55 2018 +0300
@@ -81,7 +81,7 @@ njs_vm_external_add(njs_vm_t *vm, nxt_lv
do {
ext = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_extern_t));
if (nxt_slow_path(ext == NULL)) {
- return NULL;
+ goto memory_error;
}
ext->name = external->name;
@@ -100,7 +100,7 @@ njs_vm_external_add(njs_vm_t *vm, nxt_lv
function = nxt_mem_cache_zalloc(vm->mem_cache_pool,
sizeof(njs_function_t));
if (nxt_slow_path(function == NULL)) {
- return NULL;
+ goto memory_error;
}
/*
@@ -129,7 +129,7 @@ njs_vm_external_add(njs_vm_t *vm, nxt_lv
child = njs_vm_external_add(vm, &ext->hash, external->properties,
external->nproperties);
if (nxt_slow_path(child == NULL)) {
- return NULL;
+ goto memory_error;
}
}
@@ -144,6 +144,7 @@ njs_vm_external_add(njs_vm_t *vm, nxt_lv
ret = nxt_lvlhsh_insert(hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
return NULL;
}
}
@@ -154,6 +155,12 @@ njs_vm_external_add(njs_vm_t *vm, nxt_lv
} while (n != 0);
return ext;
+
+memory_error:
+
+ njs_memory_error(vm);
+
+ return NULL;
}
@@ -206,6 +213,7 @@ njs_vm_external_bind(njs_vm_t *vm, const
ev = nxt_mem_cache_align(vm->mem_cache_pool, sizeof(njs_value_t),
sizeof(njs_extern_value_t));
if (nxt_slow_path(ev == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -221,6 +229,7 @@ njs_vm_external_bind(njs_vm_t *vm, const
ret = nxt_lvlhsh_insert(&vm->externals_hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
return ret;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_fs.c
--- a/njs/njs_fs.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_fs.c Thu Aug 02 19:39:55 2018 +0300
@@ -221,7 +221,7 @@ njs_fs_read_file(njs_vm_t *vm, njs_value
start = njs_string_alloc(vm, &arguments[2], sb.st_size, length);
if (nxt_slow_path(start == NULL)) {
- goto memory_error;
+ goto fail;
}
p = start;
@@ -286,14 +286,12 @@ done:
return njs_function_apply(vm, callback->data.u.function,
arguments, 3, (njs_index_t) &vm->retval);
-memory_error:
+fail:
if (fd != -1) {
(void) close(fd);
}
- njs_memory_error(vm);
-
return NJS_ERROR;
}
@@ -420,7 +418,7 @@ njs_fs_read_file_sync(njs_vm_t *vm, njs_
start = njs_string_alloc(vm, &vm->retval, sb.st_size, length);
if (nxt_slow_path(start == NULL)) {
- goto memory_error;
+ goto fail;
}
p = start;
@@ -472,14 +470,12 @@ done:
return NJS_OK;
-memory_error:
+fail:
if (fd != -1) {
(void) close(fd);
}
- njs_memory_error(vm);
-
return NJS_ERROR;
}
@@ -898,12 +894,12 @@ static njs_ret_t njs_fs_error(njs_vm_t *
ret = njs_string_new(vm, &string, (u_char *) description, size, size);
if (nxt_slow_path(ret != NXT_OK)) {
- goto memory_error;
+ return NJS_ERROR;
}
error = njs_error_alloc(vm, NJS_OBJECT_ERROR, NULL, &string);
if (nxt_slow_path(error == NULL)) {
- goto memory_error;
+ return NJS_ERROR;
}
lhq.replace = 0;
@@ -920,7 +916,7 @@ static njs_ret_t njs_fs_error(njs_vm_t *
prop = njs_object_prop_alloc(vm, &njs_fs_errno_string, &value, 1);
if (nxt_slow_path(prop == NULL)) {
- goto memory_error;
+ return NJS_ERROR;
}
lhq.value = prop;
@@ -939,7 +935,7 @@ static njs_ret_t njs_fs_error(njs_vm_t *
prop = njs_object_prop_alloc(vm, &njs_fs_path_string, path, 1);
if (nxt_slow_path(prop == NULL)) {
- goto memory_error;
+ return NJS_ERROR;
}
lhq.value = prop;
@@ -955,7 +951,7 @@ static njs_ret_t njs_fs_error(njs_vm_t *
size = strlen(syscall);
ret = njs_string_new(vm, &string, (u_char *) syscall, size, size);
if (nxt_slow_path(ret != NXT_OK)) {
- goto memory_error;
+ return NJS_ERROR;
}
lhq.key = nxt_string_value("sycall");
@@ -964,7 +960,7 @@ static njs_ret_t njs_fs_error(njs_vm_t *
prop = njs_object_prop_alloc(vm, &njs_fs_syscall_string, &string, 1);
if (nxt_slow_path(prop == NULL)) {
- goto memory_error;
+ return NJS_ERROR;
}
lhq.value = prop;
@@ -981,12 +977,6 @@ static njs_ret_t njs_fs_error(njs_vm_t *
retval->data.truth = 1;
return NJS_OK;
-
-memory_error:
-
- njs_memory_error(vm);
-
- return NJS_ERROR;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_function.c
--- a/njs/njs_function.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_function.c Thu Aug 02 19:39:55 2018 +0300
@@ -35,11 +35,16 @@ njs_function_alloc(njs_vm_t *vm)
function->u.lambda = nxt_mem_cache_zalloc(vm->mem_cache_pool,
sizeof(njs_function_lambda_t));
if (nxt_slow_path(function->u.lambda == NULL)) {
+ njs_memory_error(vm);
return NULL;
}
+
+ return function;
}
- return function;
+ njs_memory_error(vm);
+
+ return NULL;
}
@@ -62,7 +67,8 @@ njs_function_value_copy(njs_vm_t *vm, nj
copy = nxt_mem_cache_alloc(vm->mem_cache_pool, size);
if (nxt_slow_path(copy == NULL)) {
- return copy;
+ njs_memory_error(vm);
+ return NULL;
}
value->data.u.function = copy;
@@ -247,6 +253,7 @@ njs_function_frame_alloc(njs_vm_t *vm, s
frame = nxt_mem_cache_align(vm->mem_cache_pool, sizeof(njs_value_t),
spare_size);
if (nxt_slow_path(frame == NULL)) {
+ njs_memory_error(vm);
return NULL;
}
@@ -365,6 +372,7 @@ njs_function_call(njs_vm_t *vm, njs_inde
closure = nxt_mem_cache_align(vm->mem_cache_pool,
sizeof(njs_value_t), size);
if (nxt_slow_path(closure == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -618,6 +626,7 @@ njs_function_prototype_bind(njs_vm_t *vm
function = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_function_t));
if (nxt_slow_path(function == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -640,6 +649,7 @@ njs_function_prototype_bind(njs_vm_t *vm
values = nxt_mem_cache_alloc(vm->mem_cache_pool, size);
if (nxt_slow_path(values == NULL)) {
+ njs_memory_error(vm);
nxt_mem_cache_free(vm->mem_cache_pool, function);
return NXT_ERROR;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_json.c
--- a/njs/njs_json.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_json.c Thu Aug 02 19:39:55 2018 +0300
@@ -535,7 +535,6 @@ njs_json_parse_array(njs_json_parse_ctx_
array = njs_array_alloc(ctx->vm, 0, 0);
if (nxt_slow_path(array == NULL)) {
- njs_memory_error(ctx->vm);
return NULL;
}
@@ -812,7 +811,6 @@ njs_json_parse_string(njs_json_parse_ctx
ret = njs_string_create(ctx->vm, value, (u_char *) start, size, length);
if (nxt_slow_path(ret != NXT_OK)) {
- njs_memory_error(ctx->vm);
return NULL;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_object.c
--- a/njs/njs_object.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_object.c Thu Aug 02 19:39:55 2018 +0300
@@ -34,9 +34,12 @@ njs_object_alloc(njs_vm_t *vm)
object->type = NJS_OBJECT;
object->shared = 0;
object->extensible = 1;
+ return object;
}
- return object;
+ njs_memory_error(vm);
+
+ return NULL;
}
@@ -58,9 +61,12 @@ njs_object_value_copy(njs_vm_t *vm, njs_
object->__proto__ = &vm->prototypes[NJS_PROTOTYPE_OBJECT].object;
object->shared = 0;
value->data.u.object = object;
+ return object;
}
- return object;
+ njs_memory_error(vm);
+
+ return NULL;
}
@@ -83,9 +89,13 @@ njs_object_value_alloc(njs_vm_t *vm, con
ov->object.__proto__ = &vm->prototypes[index].object;
ov->value = *value;
+
+ return &ov->object;
}
- return &ov->object;
+ njs_memory_error(vm);
+
+ return NULL;
}
@@ -107,6 +117,7 @@ njs_object_hash_create(njs_vm_t *vm, nxt
ret = nxt_lvlhsh_insert(hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
return NXT_ERROR;
}
@@ -183,9 +194,12 @@ njs_object_prop_alloc(njs_vm_t *vm, cons
prop->enumerable = attributes;
prop->writable = attributes;
prop->configurable = attributes;
+ return prop;
}
- return prop;
+ njs_memory_error(vm);
+
+ return NULL;
}
@@ -988,6 +1002,7 @@ njs_object_get_own_property_descriptor(n
ret = nxt_lvlhsh_insert(&descriptor->hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
return NXT_ERROR;
}
@@ -1005,6 +1020,7 @@ njs_object_get_own_property_descriptor(n
ret = nxt_lvlhsh_insert(&descriptor->hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
return NXT_ERROR;
}
@@ -1022,6 +1038,7 @@ njs_object_get_own_property_descriptor(n
ret = nxt_lvlhsh_insert(&descriptor->hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
return NXT_ERROR;
}
@@ -1039,6 +1056,7 @@ njs_object_get_own_property_descriptor(n
ret = nxt_lvlhsh_insert(&descriptor->hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
return NXT_ERROR;
}
@@ -1395,7 +1413,6 @@ njs_property_prototype_create(njs_vm_t *
return &prop->value;
}
- /* Memory allocation or NXT_DECLINED error. */
njs_internal_error(vm, NULL);
return NULL;
@@ -1638,7 +1655,6 @@ njs_property_constructor_create(njs_vm_t
return &prop->value;
}
- /* Memory allocation or NXT_DECLINED error. */
njs_internal_error(vm, NULL);
return NULL;
diff -r 4586ae051111 -r d2cbea77122c njs/njs_regexp.c
--- a/njs/njs_regexp.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_regexp.c Thu Aug 02 19:39:55 2018 +0300
@@ -34,11 +34,13 @@ njs_regexp_init(njs_vm_t *vm)
vm->regex_context = nxt_regex_context_create(njs_regexp_malloc,
njs_regexp_free, vm->mem_cache_pool);
if (nxt_slow_path(vm->regex_context == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
vm->single_match_data = nxt_regex_match_data(NULL, vm->regex_context);
if (nxt_slow_path(vm->single_match_data == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -66,6 +68,7 @@ njs_ret_t
njs_regexp_constructor(njs_vm_t *vm, njs_value_t *args, nxt_uint_t nargs,
njs_index_t unused)
{
+ u_char *start;
nxt_str_t string;
njs_regexp_flags_t flags;
@@ -81,9 +84,12 @@ njs_regexp_constructor(njs_vm_t *vm, njs
default:
njs_string_get(&args[2], &string);
- flags = njs_regexp_flags(&string.start, string.start + string.length,
- 1);
+ start = string.start;
+
+ flags = njs_regexp_flags(&start, start + string.length, 1);
if (nxt_slow_path(flags < 0)) {
+ njs_syntax_error(vm, "Invalid RegExp flags \"%.*s\"",
+ (int) string.length, string.start);
return NXT_ERROR;
}
@@ -267,6 +273,7 @@ njs_regexp_pattern_create(njs_vm_t *vm,
sizeof(njs_regexp_pattern_t)
+ 1 + length + size + 1);
if (nxt_slow_path(pattern == NULL)) {
+ njs_memory_error(vm);
return NULL;
}
@@ -434,9 +441,12 @@ njs_regexp_alloc(njs_vm_t *vm, njs_regex
regexp->object.extensible = 1;
regexp->last_index = 0;
regexp->pattern = pattern;
+ return regexp;
}
- return regexp;
+ njs_memory_error(vm);
+
+ return NULL;
}
@@ -655,6 +665,7 @@ njs_regexp_prototype_exec(njs_vm_t *vm,
match_data = nxt_regex_match_data(&pattern->regex[type],
vm->regex_context);
if (nxt_slow_path(match_data == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -744,6 +755,7 @@ njs_regexp_exec_result(njs_vm_t *vm, njs
ret = nxt_lvlhsh_insert(&array->object.hash, &lhq);
if (nxt_slow_path(ret != NXT_OK)) {
+ njs_internal_error(vm, NULL);
goto fail;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_string.c
--- a/njs/njs_string.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_string.c Thu Aug 02 19:39:55 2018 +0300
@@ -147,6 +147,7 @@ njs_string_create(njs_vm_t *vm, njs_valu
string = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_string_t));
if (nxt_slow_path(string == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -174,8 +175,6 @@ njs_string_new(njs_vm_t *vm, njs_value_t
return NXT_OK;
}
- njs_memory_error(vm);
-
return NXT_ERROR;
}
@@ -293,8 +292,6 @@ njs_string_hex(njs_vm_t *vm, njs_value_t
return NXT_OK;
}
- njs_memory_error(vm);
-
return NXT_ERROR;
}
@@ -386,7 +383,6 @@ njs_string_base64(njs_vm_t *vm, njs_valu
dst.start = njs_string_alloc(vm, &vm->retval, dst.length, dst.length);
if (nxt_slow_path(dst.start == NULL)) {
- njs_memory_error(vm);
return NXT_ERROR;
}
@@ -418,7 +414,6 @@ njs_string_base64url(njs_vm_t *vm, njs_v
dst.start = njs_string_alloc(vm, &vm->retval, dst.length, dst.length);
if (nxt_slow_path(dst.start == NULL)) {
- njs_memory_error(vm);
return NXT_ERROR;
}
@@ -491,6 +486,7 @@ njs_string_validate(njs_vm_t *vm, njs_st
start = nxt_mem_cache_alloc(vm->mem_cache_pool, new_size);
if (nxt_slow_path(start == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_time.c
--- a/njs/njs_time.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_time.c Thu Aug 02 19:39:55 2018 +0300
@@ -71,6 +71,7 @@ njs_set_timeout(njs_vm_t *vm, njs_value_
memory_error:
njs_memory_error(vm);
+
return NJS_ERROR;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_variable.c
--- a/njs/njs_variable.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_variable.c Thu Aug 02 19:39:55 2018 +0300
@@ -83,6 +83,7 @@ njs_builtin_add(njs_vm_t *vm, njs_parser
ret = nxt_lvlhsh_insert(&scope->variables, &lhq);
if (nxt_fast_path(ret == NXT_OK)) {
+ njs_internal_error(vm, NULL);
return var;
}
@@ -397,6 +398,7 @@ njs_variable_get(njs_vm_t *vm, njs_parse
value = nxt_mem_cache_align(vm->mem_cache_pool, sizeof(njs_value_t),
sizeof(njs_value_t));
if (nxt_slow_path(value == NULL)) {
+ njs_memory_error(vm);
return NULL;
}
@@ -503,6 +505,7 @@ njs_variable_alloc(njs_vm_t *vm, nxt_str
var = nxt_mem_cache_zalloc(vm->mem_cache_pool, sizeof(njs_variable_t));
if (nxt_slow_path(var == NULL)) {
+ njs_memory_error(vm);
return NULL;
}
@@ -533,6 +536,8 @@ njs_name_copy(njs_vm_t *vm, nxt_str_t *d
return NXT_OK;
}
+ njs_memory_error(vm);
+
return NXT_ERROR;
}
diff -r 4586ae051111 -r d2cbea77122c njs/njs_vm.c
--- a/njs/njs_vm.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/njs_vm.c Thu Aug 02 19:39:55 2018 +0300
@@ -383,6 +383,7 @@ njs_vmcode_function(njs_vm_t *vm, njs_va
function = nxt_mem_cache_zalloc(vm->mem_cache_pool, size);
if (nxt_slow_path(function == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -935,6 +936,7 @@ njs_method_private_copy(njs_vm_t *vm, nj
prop = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_object_prop_t));
if (nxt_slow_path(prop == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -968,6 +970,7 @@ njs_vmcode_property_foreach(njs_vm_t *vm
next = nxt_mem_cache_alloc(vm->mem_cache_pool,
sizeof(njs_property_next_t));
if (nxt_slow_path(next == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -2721,6 +2724,7 @@ njs_vmcode_try_start(njs_vm_t *vm, njs_v
if (vm->top_frame->exception.catch != NULL) {
e = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_exception_t));
if (nxt_slow_path(e == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -3298,6 +3302,7 @@ again:
if (size != NJS_STRING_LONG) {
start = nxt_mem_cache_alloc(vm->mem_cache_pool, size);
if (nxt_slow_path(start == NULL)) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
@@ -3332,6 +3337,7 @@ again:
p = nxt_mem_cache_alloc(vm->mem_cache_pool, len);
if (p == NULL) {
+ njs_memory_error(vm);
return NXT_ERROR;
}
diff -r 4586ae051111 -r d2cbea77122c njs/test/njs_unit_test.c
--- a/njs/test/njs_unit_test.c Wed Aug 01 18:37:15 2018 +0300
+++ b/njs/test/njs_unit_test.c Thu Aug 02 19:39:55 2018 +0300
@@ -5566,6 +5566,9 @@ static njs_unit_test_t njs_test[] =
{ nxt_string("var r = new RegExp('abc', 'i'); r.test('00ABC11')"),
nxt_string("true") },
+ { nxt_string("new RegExp('', 'x')"),
+ nxt_string("SyntaxError: Invalid RegExp flags \"x\"") },
+
{ nxt_string("[0].map(RegExp().toString)"),
nxt_string("TypeError: 'this' argument is not a regexp") },
@@ -6081,16 +6084,19 @@ static njs_unit_test_t njs_test[] =
nxt_string("1,two,3") },
{ nxt_string("var a = Array(-1)"),
- nxt_string("RangeError") },
+ nxt_string("RangeError: Invalid array length") },
{ nxt_string("var a = Array(2.5)"),
- nxt_string("RangeError") },
+ nxt_string("RangeError: Invalid array length") },
{ nxt_string("var a = Array(NaN)"),
- nxt_string("RangeError") },
+ nxt_string("RangeError: Invalid array length") },
{ nxt_string("var a = Array(Infinity)"),
- nxt_string("RangeError") },
+ nxt_string("RangeError: Invalid array length") },
+
+ { nxt_string("var a = Array(1111111111)"),
+ nxt_string("MemoryError") },
{ nxt_string("var a = new Array(3); a"),
nxt_string(",,") },
@@ -7533,6 +7539,9 @@ static njs_unit_test_t njs_test[] =
{ nxt_string("var d = new Date(); d.__proto__ === Date.prototype"),
nxt_string("true") },
+ { nxt_string("new Date(NaN)"),
+ nxt_string("Invalid Date") },
+
{ nxt_string("[0].map(new Date().getDate)"),
nxt_string("TypeError: cannot convert void to date") },
From dnj0496 at gmail.com Thu Aug 2 19:29:14 2018
From: dnj0496 at gmail.com (Dk Jack)
Date: Thu, 2 Aug 2018 12:29:14 -0700
Subject: post body
Message-ID:
Hi,
In my module, I'd like to look into the post body to decide if the request
should be allowed to proceed to origin server or not. Based on the examples
I could find I've coded this in my module. After some trial and error it
seems to be working. I am not sure if my implementation is 100% correct. I
would appreciate it if someone can go over my code below and let me know if
my implementation is correct. Especially, the body_complete_handler. Thanks.
Regards,
Dk.
static ngx_int_t mod_setup_body_handler(ngx_http_request_t* r);
static void mod_body_complete_handler(ngx_http_request_t* r);
static ngx_int_t mod_extract_post_body(ngx_http_request_t *r, char *data,
size_t *dsize);
static ngx_int_t
ngx_http_request_handler(ngx_http_request_t *r)
{
....
if ((NGX_HTTP_POST == r->method) || (NGX_HTTP_PUT == r->method)) {
return mod_setup_body_handler(r);
}
...
}
ngx_int_t
mod_setup_body_handler(ngx_http_request_t* r)
{
ngx_int_t rc;
r->request_body_in_single_buf = 1;
r->request_body_in_persistent_file = 1;
r->request_body_in_clean_file = 1;
rc = ngx_http_read_client_request_body(r, mod_body_complete_handler);
if (rc == NGX_ERROR) {
ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "read req. body
error");
return rc;
}
if (rc == NGX_ERROR || rc >= NGX_HTTP_SPECIAL_RESPONSE) {
ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "read req. body
error. special response");
return rc;
}
if (rc == NGX_AGAIN) {
/*
* nginx will call the body_received when needed. Returning
* NGX_DONE will prevent nginx from calling ngx_http_finalize_request
* (which we will call in body_received)
*/
return NGX_DONE;
}
return NGX_DECLINED;
}
void
mod_body_complete_handler(ngx_http_request_t* r)
{
ngx_http_request_t* r = mNginxRequest;
if(r->request_body == NULL) {
// callback was called but body is empty. Setup handler again.
mod_setup_body_handler(bh_ptr);
return;
} else if (r->request_body->rest) {
// we don't have the complete body. Complete callback will be called
again.
return;
}
if (NULL == mNginxRequest->request_body->temp_file) {
size_t dsize = 64*1024;
char data[dsize]
if (mod_extract_post_body(r, data, dsize) == NGX_OK) {
// we have the complete body.
int rc = mod_allow_request(body, dsize); // returns NGX_OK or 4XX.
if (rc != NGX_OK) {
// allow req. has already sent a custom response if rc != NGX_OK.
ngx_http_finalize_request(r, NGX_OK);
return;
}
}
}
// ???? not having this cause request to hang.
ngx_http_core_run_phases(r);
}
ngx_int_t
mod_extract_post_body(ngx_http_request_t *r, char *data, size_t *dsize)
{
size_t buf_size = *dsize;
if (NULL == r->request_body->temp_file) {
// we have body in buffer chain.
ngx_buf_t *buf;
ngx_chain_t *cl;
size_t data_start_pos = 0;
cl = r->request_body->bufs;
// copy the body into our temporary buffer.
for (;NULL != cl; cl = cl->next) {
buf = cl->buf;
size_t bsize = buf->last - buf->pos;
if ((data_start_pos+bsize) > buf_size) {
// data is bigger than the input buffer size, abort
*dsize = data_start_pos;
*(data+*dsize) = '\0';
return NGX_ERROR;
}
ngx_memcpy((data+data_start_pos), buf->pos, bsize);
data_start_pos += bsize;
}
*dsize = data_start_pos;
*(data+*dsize) = '\0';
return NGX_OK;
} else {
// body in file not implemented yet.
}
return NGX_ERROR;
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From pluknet at nginx.com Mon Aug 6 12:46:18 2018
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Mon, 06 Aug 2018 12:46:18 +0000
Subject: [nginx] Dav: removed dead store after 8e7a5de61664.
Message-ID:
details: http://hg.nginx.org/nginx/rev/bb40db0e380d
branches:
changeset: 7331:bb40db0e380d
user: Sergey Kandaurov
date: Thu Aug 02 13:19:48 2018 +0300
description:
Dav: removed dead store after 8e7a5de61664.
Found by Clang Static Analyzer.
diffstat:
src/http/modules/ngx_http_dav_module.c | 2 --
1 files changed, 0 insertions(+), 2 deletions(-)
diffs (12 lines):
diff -r 8e7a5de61664 -r bb40db0e380d src/http/modules/ngx_http_dav_module.c
--- a/src/http/modules/ngx_http_dav_module.c Wed Aug 01 02:12:21 2018 +0300
+++ b/src/http/modules/ngx_http_dav_module.c Thu Aug 02 13:19:48 2018 +0300
@@ -841,8 +841,6 @@ overwrite_done:
return NGX_HTTP_INTERNAL_SERVER_ERROR;
}
- dlcf = ngx_http_get_module_loc_conf(r, ngx_http_dav_module);
-
cf.size = ngx_file_size(&fi);
cf.buf_size = 0;
cf.access = ngx_file_access(&fi);
From mdounin at mdounin.ru Tue Aug 7 12:58:25 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 07 Aug 2018 12:58:25 +0000
Subject: [nginx] SSL: enabled TLSv1.3 with BoringSSL.
Message-ID:
details: http://hg.nginx.org/nginx/rev/7ad0f4ace359
branches:
changeset: 7332:7ad0f4ace359
user: Maxim Dounin
date: Tue Aug 07 02:15:28 2018 +0300
description:
SSL: enabled TLSv1.3 with BoringSSL.
BoringSSL currently requires SSL_CTX_set_max_proto_version(TLS1_3_VERSION)
to be able to enable TLS 1.3. This is because by default max protocol
version is set to TLS 1.2, and the SSL_OP_NO_* options are merely used
as a blacklist within the version range specified using the
SSL_CTX_set_min_proto_version() and SSL_CTX_set_max_proto_version()
functions.
With this change, we now call SSL_CTX_set_max_proto_version() with an
explicit maximum version set. This enables TLS 1.3 with BoringSSL.
As a side effect, this change also limits maximum protocol version to
the newest protocol we know about, TLS 1.3. This seems to be a good
change, as enabling unknown protocols might have unexpected results.
Additionally, we now explicitly call SSL_CTX_set_min_proto_version()
with 0. This is expected to help with Debian system-wide default
of MinProtocol set to TLSv1.2, see
http://mailman.nginx.org/pipermail/nginx-ru/2017-October/060411.html.
Note that there is no SSL_CTX_set_min_proto_version macro in BoringSSL,
so we call SSL_CTX_set_min_proto_version() and SSL_CTX_set_max_proto_version()
as long as the TLS1_3_VERSION macro is defined.
diffstat:
src/event/ngx_event_openssl.c | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
diffs (15 lines):
diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -331,6 +331,11 @@ ngx_ssl_create(ngx_ssl_t *ssl, ngx_uint_
}
#endif
+#ifdef TLS1_3_VERSION
+ SSL_CTX_set_min_proto_version(ssl->ctx, 0);
+ SSL_CTX_set_max_proto_version(ssl->ctx, TLS1_3_VERSION);
+#endif
+
#ifdef SSL_OP_NO_COMPRESSION
SSL_CTX_set_options(ssl->ctx, SSL_OP_NO_COMPRESSION);
#endif
From mdounin at mdounin.ru Tue Aug 7 12:58:27 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 07 Aug 2018 12:58:27 +0000
Subject: [nginx] SSL: support for TLSv1.3 early data with BoringSSL.
Message-ID:
details: http://hg.nginx.org/nginx/rev/ba971deb4b44
branches:
changeset: 7333:ba971deb4b44
user: Maxim Dounin
date: Tue Aug 07 02:16:07 2018 +0300
description:
SSL: support for TLSv1.3 early data with BoringSSL.
Early data AKA 0-RTT mode is enabled as long as "ssl_early_data on" is
specified in the configuration (default is off).
The $ssl_early_data variable evaluates to "1" if the SSL handshake
isn't yet completed, and can be used to set the Early-Data header as
per draft-ietf-httpbis-replay-04.
diffstat:
src/event/ngx_event_openssl.c | 38 ++++++++++++++++++++++++++++++++++
src/event/ngx_event_openssl.h | 4 +++
src/http/modules/ngx_http_ssl_module.c | 18 ++++++++++++++++
src/http/modules/ngx_http_ssl_module.h | 1 +
4 files changed, 61 insertions(+), 0 deletions(-)
diffs (143 lines):
diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -1169,6 +1169,29 @@ ngx_ssl_ecdh_curve(ngx_conf_t *cf, ngx_s
ngx_int_t
+ngx_ssl_early_data(ngx_conf_t *cf, ngx_ssl_t *ssl, ngx_uint_t enable)
+{
+ if (!enable) {
+ return NGX_OK;
+ }
+
+#ifdef SSL_ERROR_EARLY_DATA_REJECTED
+
+ /* BoringSSL */
+
+ SSL_CTX_set_early_data_enabled(ssl->ctx, 1);
+
+#else
+ ngx_log_error(NGX_LOG_WARN, ssl->log, 0,
+ "\"ssl_early_data\" is not supported on this platform, "
+ "ignored");
+#endif
+
+ return NGX_OK;
+}
+
+
+ngx_int_t
ngx_ssl_client_session_cache(ngx_conf_t *cf, ngx_ssl_t *ssl, ngx_uint_t enable)
{
if (!enable) {
@@ -3624,6 +3647,21 @@ ngx_ssl_get_session_reused(ngx_connectio
ngx_int_t
+ngx_ssl_get_early_data(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s)
+{
+ s->len = 0;
+
+#ifdef SSL_ERROR_EARLY_DATA_REJECTED
+ if (SSL_in_early_data(c->ssl->connection)) {
+ ngx_str_set(s, "1");
+ }
+#endif
+
+ return NGX_OK;
+}
+
+
+ngx_int_t
ngx_ssl_get_server_name(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s)
{
#ifdef SSL_CTRL_SET_TLSEXT_HOSTNAME
diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h
--- a/src/event/ngx_event_openssl.h
+++ b/src/event/ngx_event_openssl.h
@@ -171,6 +171,8 @@ RSA *ngx_ssl_rsa512_key_callback(ngx_ssl
ngx_array_t *ngx_ssl_read_password_file(ngx_conf_t *cf, ngx_str_t *file);
ngx_int_t ngx_ssl_dhparam(ngx_conf_t *cf, ngx_ssl_t *ssl, ngx_str_t *file);
ngx_int_t ngx_ssl_ecdh_curve(ngx_conf_t *cf, ngx_ssl_t *ssl, ngx_str_t *name);
+ngx_int_t ngx_ssl_early_data(ngx_conf_t *cf, ngx_ssl_t *ssl,
+ ngx_uint_t enable);
ngx_int_t ngx_ssl_client_session_cache(ngx_conf_t *cf, ngx_ssl_t *ssl,
ngx_uint_t enable);
ngx_int_t ngx_ssl_session_cache(ngx_ssl_t *ssl, ngx_str_t *sess_ctx,
@@ -213,6 +215,8 @@ ngx_int_t ngx_ssl_get_session_id(ngx_con
ngx_str_t *s);
ngx_int_t ngx_ssl_get_session_reused(ngx_connection_t *c, ngx_pool_t *pool,
ngx_str_t *s);
+ngx_int_t ngx_ssl_get_early_data(ngx_connection_t *c, ngx_pool_t *pool,
+ ngx_str_t *s);
ngx_int_t ngx_ssl_get_server_name(ngx_connection_t *c, ngx_pool_t *pool,
ngx_str_t *s);
ngx_int_t ngx_ssl_get_raw_certificate(ngx_connection_t *c, ngx_pool_t *pool,
diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c
--- a/src/http/modules/ngx_http_ssl_module.c
+++ b/src/http/modules/ngx_http_ssl_module.c
@@ -239,6 +239,13 @@ static ngx_command_t ngx_http_ssl_comma
offsetof(ngx_http_ssl_srv_conf_t, stapling_verify),
NULL },
+ { ngx_string("ssl_early_data"),
+ NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG,
+ ngx_conf_set_flag_slot,
+ NGX_HTTP_SRV_CONF_OFFSET,
+ offsetof(ngx_http_ssl_srv_conf_t, early_data),
+ NULL },
+
ngx_null_command
};
@@ -294,6 +301,10 @@ static ngx_http_variable_t ngx_http_ssl
{ ngx_string("ssl_session_reused"), NULL, ngx_http_ssl_variable,
(uintptr_t) ngx_ssl_get_session_reused, NGX_HTTP_VAR_CHANGEABLE, 0 },
+ { ngx_string("ssl_early_data"), NULL, ngx_http_ssl_variable,
+ (uintptr_t) ngx_ssl_get_early_data,
+ NGX_HTTP_VAR_CHANGEABLE|NGX_HTTP_VAR_NOCACHEABLE, 0 },
+
{ ngx_string("ssl_server_name"), NULL, ngx_http_ssl_variable,
(uintptr_t) ngx_ssl_get_server_name, NGX_HTTP_VAR_CHANGEABLE, 0 },
@@ -552,6 +563,7 @@ ngx_http_ssl_create_srv_conf(ngx_conf_t
sscf->enable = NGX_CONF_UNSET;
sscf->prefer_server_ciphers = NGX_CONF_UNSET;
+ sscf->early_data = NGX_CONF_UNSET;
sscf->buffer_size = NGX_CONF_UNSET_SIZE;
sscf->verify = NGX_CONF_UNSET_UINT;
sscf->verify_depth = NGX_CONF_UNSET_UINT;
@@ -594,6 +606,8 @@ ngx_http_ssl_merge_srv_conf(ngx_conf_t *
ngx_conf_merge_value(conf->prefer_server_ciphers,
prev->prefer_server_ciphers, 0);
+ ngx_conf_merge_value(conf->early_data, prev->early_data, 0);
+
ngx_conf_merge_bitmask_value(conf->protocols, prev->protocols,
(NGX_CONF_BITMASK_SET|NGX_SSL_TLSv1
|NGX_SSL_TLSv1_1|NGX_SSL_TLSv1_2));
@@ -809,6 +823,10 @@ ngx_http_ssl_merge_srv_conf(ngx_conf_t *
}
+ if (ngx_ssl_early_data(cf, &conf->ssl, conf->early_data) != NGX_OK) {
+ return NGX_CONF_ERROR;
+ }
+
return NGX_CONF_OK;
}
diff --git a/src/http/modules/ngx_http_ssl_module.h b/src/http/modules/ngx_http_ssl_module.h
--- a/src/http/modules/ngx_http_ssl_module.h
+++ b/src/http/modules/ngx_http_ssl_module.h
@@ -20,6 +20,7 @@ typedef struct {
ngx_ssl_t ssl;
ngx_flag_t prefer_server_ciphers;
+ ngx_flag_t early_data;
ngx_uint_t protocols;
From kahing at cloudflare.com Tue Aug 7 21:26:01 2018
From: kahing at cloudflare.com (Ka-Hing Cheung)
Date: Tue, 7 Aug 2018 14:26:01 -0700
Subject: cache: move open to thread pool
Message-ID:
commit 0bcfefa040abdff674047c15701d237ff16a5f2b
Author: Ka-Hing Cheung
Date: Fri Aug 3 13:37:58 2018 -0700
move open to thread pool
At cloudflare we found that open() can block a long time, especially
at p99 and p999. Moving it to thread pool improved overall p99 by 6x
or so during peak time.
This has the side effect of disabling nginx's open_file_cache when
"aio threads" is turned on. For us we found that to have no impact
(probably because we have way too many files for open file cache to
be effective anyway).
thread pool does increase CPU usage somewhat but we have not measured
difference in CPU usage for stock "aio threads" compared to after this
patch.
Only the cache hit open() is moved to thread pool.
We wrote more about it here:
https://blog.cloudflare.com/how-we-scaled-nginx-and-saved-the-world-54-years-every-day/
diff --git src/core/ngx_open_file_cache.c src/core/ngx_open_file_cache.c
index b23ee78d..9177ebfd 100644
--- src/core/ngx_open_file_cache.c
+++ src/core/ngx_open_file_cache.c
@@ -35,8 +35,6 @@ static ngx_fd_t ngx_open_file_wrapper(ngx_str_t *name,
ngx_int_t access, ngx_log_t *log);
static ngx_int_t ngx_file_info_wrapper(ngx_str_t *name,
ngx_open_file_info_t *of, ngx_file_info_t *fi, ngx_log_t *log);
-static ngx_int_t ngx_open_and_stat_file(ngx_str_t *name,
- ngx_open_file_info_t *of, ngx_log_t *log);
static void ngx_open_file_add_event(ngx_open_file_cache_t *cache,
ngx_cached_open_file_t *file, ngx_open_file_info_t *of, ngx_log_t *log);
static void ngx_open_file_cleanup(void *data);
@@ -836,7 +834,7 @@ ngx_file_info_wrapper(ngx_str_t *name,
ngx_open_file_info_t *of,
}
-static ngx_int_t
+ngx_int_t
ngx_open_and_stat_file(ngx_str_t *name, ngx_open_file_info_t *of,
ngx_log_t *log)
{
diff --git src/core/ngx_open_file_cache.h src/core/ngx_open_file_cache.h
index d119c129..94b4d552 100644
--- src/core/ngx_open_file_cache.h
+++ src/core/ngx_open_file_cache.h
@@ -124,6 +124,8 @@ ngx_open_file_cache_t
*ngx_open_file_cache_init(ngx_pool_t *pool,
ngx_uint_t max, time_t inactive);
ngx_int_t ngx_open_cached_file(ngx_open_file_cache_t *cache, ngx_str_t *name,
ngx_open_file_info_t *of, ngx_pool_t *pool);
+ngx_int_t ngx_open_and_stat_file(ngx_str_t *name,
+ ngx_open_file_info_t *of, ngx_log_t *log);
#endif /* _NGX_OPEN_FILE_CACHE_H_INCLUDED_ */
diff --git src/http/ngx_http_cache.h src/http/ngx_http_cache.h
index f9e96640..2910f6a1 100644
--- src/http/ngx_http_cache.h
+++ src/http/ngx_http_cache.h
@@ -114,6 +114,7 @@ struct ngx_http_cache_s {
unsigned exists:1;
unsigned temp_file:1;
unsigned purged:1;
+ unsigned opening:1;
unsigned reading:1;
unsigned secondary:1;
unsigned background:1;
diff --git src/http/ngx_http_file_cache.c src/http/ngx_http_file_cache.c
index 56866fa4..9cb110bf 100644
--- src/http/ngx_http_file_cache.c
+++ src/http/ngx_http_file_cache.c
@@ -18,6 +18,10 @@ static void
ngx_http_file_cache_lock_wait(ngx_http_request_t *r,
ngx_http_cache_t *c);
static ngx_int_t ngx_http_file_cache_read(ngx_http_request_t *r,
ngx_http_cache_t *c);
+static ngx_int_t ngx_http_file_cache_aio_open(ngx_http_request_t *r,
+ ngx_http_cache_t *c, ngx_int_t *rv);
+static ngx_int_t ngx_http_file_cache_do_aio_open(ngx_http_request_t *r,
+ ngx_http_cache_t *c, ngx_open_file_info_t *of, ngx_int_t *rv);
static ssize_t ngx_http_file_cache_aio_read(ngx_http_request_t *r,
ngx_http_cache_t *c);
#if (NGX_HAVE_FILE_AIO)
@@ -268,9 +272,7 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
ngx_uint_t test;
ngx_http_cache_t *c;
ngx_pool_cleanup_t *cln;
- ngx_open_file_info_t of;
ngx_http_file_cache_t *cache;
- ngx_http_core_loc_conf_t *clcf;
c = r->cache;
@@ -278,6 +280,10 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
return NGX_AGAIN;
}
+ if (c->opening) {
+ return ngx_http_file_cache_aio_open(r, c, &rv);
+ }
+
if (c->reading) {
return ngx_http_file_cache_read(r, c);
}
@@ -343,23 +349,33 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
goto done;
}
- clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
+ return ngx_http_file_cache_aio_open(r, c, &rv);
- ngx_memzero(&of, sizeof(ngx_open_file_info_t));
+done:
- of.uniq = c->uniq;
- of.valid = clcf->open_file_cache_valid;
- of.min_uses = clcf->open_file_cache_min_uses;
- of.events = clcf->open_file_cache_events;
- of.directio = NGX_OPEN_FILE_DIRECTIO_OFF;
- of.read_ahead = clcf->read_ahead;
+ if (rv == NGX_DECLINED) {
+ return ngx_http_file_cache_lock(r, c);
+ }
- if (ngx_open_cached_file(clcf->open_file_cache, &c->file.name,
&of, r->pool)
- != NGX_OK)
- {
- switch (of.err) {
+ return rv;
+}
+
+static ngx_int_t
+ngx_http_file_cache_aio_open(ngx_http_request_t *r, ngx_http_cache_t *c,
+ ngx_int_t *rv)
+{
+ ngx_int_t rc;
+ ngx_open_file_info_t of;
+ ngx_http_file_cache_t *cache = c->file_cache;
- case 0:
+ rc = ngx_http_file_cache_do_aio_open(r, c, &of, rv);
+ if (rc == NGX_AGAIN) {
+ return rc;
+ }
+
+ if (rc != NGX_OK) {
+ switch (of.err) {
+ case NGX_OK:
return NGX_ERROR;
case NGX_ENOENT:
@@ -391,14 +407,13 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
done:
- if (rv == NGX_DECLINED) {
+ if (*rv == NGX_DECLINED) {
return ngx_http_file_cache_lock(r, c);
}
- return rv;
+ return *rv;
}
-
static ngx_int_t
ngx_http_file_cache_lock(ngx_http_request_t *r, ngx_http_cache_t *c)
{
@@ -663,6 +678,127 @@ ngx_http_file_cache_read(ngx_http_request_t *r,
ngx_http_cache_t *c)
}
+#if (NGX_THREADS)
+typedef struct {
+ ngx_str_t name;
+ ngx_pool_t *pool;
+ ngx_open_file_info_t of;
+ ngx_int_t rv;
+} ngx_thread_file_open_ctx_t;
+
+static void
+ngx_http_file_cache_thread_open_handler(void *data, ngx_log_t *log)
+{
+ ngx_thread_file_open_ctx_t *ctx = data;
+ ngx_pool_t *pool = ctx->pool;
+ ngx_open_file_info_t *of = &ctx->of;
+ ngx_pool_cleanup_t *cln;
+ ngx_pool_cleanup_file_t *clnf;
+ ngx_int_t rc;
+
+ ngx_log_debug0(NGX_LOG_DEBUG_CORE, log, 0, "thread read handler");
+
+ cln = ngx_pool_cleanup_add(pool, sizeof(ngx_pool_cleanup_file_t));
+ if (cln == NULL) {
+ ctx->of.err = NGX_ERROR;
+ return;
+ }
+
+ of->fd = NGX_INVALID_FILE;
+
+ rc = ngx_open_and_stat_file(&ctx->name, of, pool->log);
+ if (rc == NGX_OK && !of->is_dir) {
+ cln->handler = ngx_pool_cleanup_file;
+ clnf = cln->data;
+
+ clnf->fd = of->fd;
+ clnf->name = ctx->name.data;
+ clnf->log = pool->log;
+ }
+}
+
+
+static ngx_int_t
+ngx_http_file_cache_thread_open(ngx_http_cache_t *c, ngx_open_file_info_t *of,
+ ngx_int_t *rv, ngx_pool_t *pool)
+{
+ ngx_thread_task_t *task;
+ ngx_thread_file_open_ctx_t *ctx;
+ ngx_file_t *file = &c->file;
+
+ task = file->thread_task;
+
+ if (task == NULL) {
+ task = ngx_thread_task_alloc(pool, sizeof(ngx_thread_file_open_ctx_t));
+ if (task == NULL) {
+ return NGX_ERROR;
+ }
+
+ file->thread_task = task;
+ }
+
+ ctx = task->ctx;
+
+ if (task->event.complete) {
+ task->event.complete = 0;
+
+ *of = ctx->of;
+ *rv = ctx->rv;
+ return of->err;
+ }
+
+ task->handler = ngx_http_file_cache_thread_open_handler;
+
+ ctx->pool = pool;
+ ctx->name = file->name;
+ ctx->of = *of;
+ ctx->rv = *rv;
+
+ if (file->thread_handler(task, file) != NGX_OK) {
+ return NGX_ERROR;
+ }
+
+ return NGX_AGAIN;
+}
+#endif
+
+static ngx_int_t
+ngx_http_file_cache_do_aio_open(ngx_http_request_t *r, ngx_http_cache_t *c,
+ ngx_open_file_info_t *of, ngx_int_t *rv)
+{
+ ngx_http_core_loc_conf_t *clcf;
+
+ clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
+
+ ngx_memzero(of, sizeof(ngx_open_file_info_t));
+
+ of->uniq = c->uniq;
+ of->valid = clcf->open_file_cache_valid;
+ of->min_uses = clcf->open_file_cache_min_uses;
+ of->events = clcf->open_file_cache_events;
+ of->directio = NGX_OPEN_FILE_DIRECTIO_OFF;
+ of->read_ahead = clcf->read_ahead;
+
+#if (NGX_THREADS)
+ if (clcf->aio == NGX_HTTP_AIO_THREADS) {
+ ngx_int_t rc;
+
+ c->file.thread_task = c->thread_task;
+ c->file.thread_handler = ngx_http_cache_thread_handler;
+ c->file.thread_ctx = r;
+
+ rc = ngx_http_file_cache_thread_open(c, of, rv, r->pool);
+
+ c->thread_task = c->file.thread_task;
+ c->opening = (rc == NGX_AGAIN);
+
+ return rc;
+ }
+#endif
+
+ return ngx_open_cached_file(clcf->open_file_cache, &c->file.name,
of, r->pool);
+}
+
static ssize_t
ngx_http_file_cache_aio_read(ngx_http_request_t *r, ngx_http_cache_t *c)
{
From dnj0496 at gmail.com Wed Aug 8 05:46:08 2018
From: dnj0496 at gmail.com (Dk Jack)
Date: Tue, 7 Aug 2018 22:46:08 -0700
Subject: post body
In-Reply-To:
References:
Message-ID:
Haven't received any response to inquiry. Would appreciate it if someone
can comment.
Thanks,
Dk.
On Thu, Aug 2, 2018 at 12:29 PM Dk Jack wrote:
> Hi,
> In my module, I'd like to look into the post body to decide if the request
> should be allowed to proceed to origin server or not. Based on the examples
> I could find I've coded this in my module. After some trial and error it
> seems to be working. I am not sure if my implementation is 100% correct. I
> would appreciate it if someone can go over my code below and let me know if
> my implementation is correct. Especially, the body_complete_handler. Thanks.
>
> Regards,
> Dk.
>
>
> static ngx_int_t mod_setup_body_handler(ngx_http_request_t* r);
> static void mod_body_complete_handler(ngx_http_request_t* r);
> static ngx_int_t mod_extract_post_body(ngx_http_request_t *r, char *data,
> size_t *dsize);
>
> static ngx_int_t
> ngx_http_request_handler(ngx_http_request_t *r)
> {
> ....
> if ((NGX_HTTP_POST == r->method) || (NGX_HTTP_PUT == r->method)) {
> return mod_setup_body_handler(r);
> }
>
> ...
> }
>
> ngx_int_t
> mod_setup_body_handler(ngx_http_request_t* r)
> {
> ngx_int_t rc;
>
> r->request_body_in_single_buf = 1;
> r->request_body_in_persistent_file = 1;
> r->request_body_in_clean_file = 1;
>
> rc = ngx_http_read_client_request_body(r, mod_body_complete_handler);
> if (rc == NGX_ERROR) {
> ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "read req. body
> error");
> return rc;
> }
>
> if (rc == NGX_ERROR || rc >= NGX_HTTP_SPECIAL_RESPONSE) {
> ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "read req. body
> error. special response");
> return rc;
> }
>
> if (rc == NGX_AGAIN) {
> /*
> * nginx will call the body_received when needed. Returning
> * NGX_DONE will prevent nginx from calling ngx_http_finalize_request
> * (which we will call in body_received)
> */
> return NGX_DONE;
> }
>
> return NGX_DECLINED;
> }
>
>
> void
> mod_body_complete_handler(ngx_http_request_t* r)
> {
> ngx_http_request_t* r = mNginxRequest;
>
> if(r->request_body == NULL) {
> // callback was called but body is empty. Setup handler again.
> mod_setup_body_handler(bh_ptr);
> return;
> } else if (r->request_body->rest) {
> // we don't have the complete body. Complete callback will be called
> again.
> return;
> }
>
> if (NULL == mNginxRequest->request_body->temp_file) {
> size_t dsize = 64*1024;
> char data[dsize]
>
> if (mod_extract_post_body(r, data, dsize) == NGX_OK) {
> // we have the complete body.
> int rc = mod_allow_request(body, dsize); // returns NGX_OK or 4XX.
>
> if (rc != NGX_OK) {
> // allow req. has already sent a custom response if rc != NGX_OK.
> ngx_http_finalize_request(r, NGX_OK);
> return;
> }
> }
> }
>
> // ???? not having this cause request to hang.
> ngx_http_core_run_phases(r);
> }
>
> ngx_int_t
> mod_extract_post_body(ngx_http_request_t *r, char *data, size_t *dsize)
> {
> size_t buf_size = *dsize;
>
> if (NULL == r->request_body->temp_file) {
> // we have body in buffer chain.
> ngx_buf_t *buf;
> ngx_chain_t *cl;
> size_t data_start_pos = 0;
>
> cl = r->request_body->bufs;
>
> // copy the body into our temporary buffer.
> for (;NULL != cl; cl = cl->next) {
> buf = cl->buf;
> size_t bsize = buf->last - buf->pos;
>
> if ((data_start_pos+bsize) > buf_size) {
> // data is bigger than the input buffer size, abort
> *dsize = data_start_pos;
> *(data+*dsize) = '\0';
> return NGX_ERROR;
> }
>
> ngx_memcpy((data+data_start_pos), buf->pos, bsize);
> data_start_pos += bsize;
> }
>
> *dsize = data_start_pos;
> *(data+*dsize) = '\0';
> return NGX_OK;
> } else {
> // body in file not implemented yet.
> }
>
> return NGX_ERROR;
> }
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From mdounin at mdounin.ru Wed Aug 8 18:16:53 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 8 Aug 2018 21:16:53 +0300
Subject: cache: move open to thread pool
In-Reply-To:
References:
Message-ID: <20180808181653.GX56558@mdounin.ru>
Hello!
On Tue, Aug 07, 2018 at 02:26:01PM -0700, Ka-Hing Cheung via nginx-devel wrote:
> commit 0bcfefa040abdff674047c15701d237ff16a5f2b
> Author: Ka-Hing Cheung
> Date: Fri Aug 3 13:37:58 2018 -0700
>
> move open to thread pool
>
> At cloudflare we found that open() can block a long time, especially
> at p99 and p999. Moving it to thread pool improved overall p99 by 6x
> or so during peak time.
>
> This has the side effect of disabling nginx's open_file_cache when
> "aio threads" is turned on. For us we found that to have no impact
> (probably because we have way too many files for open file cache to
> be effective anyway).
>
> thread pool does increase CPU usage somewhat but we have not measured
> difference in CPU usage for stock "aio threads" compared to after this
> patch.
>
> Only the cache hit open() is moved to thread pool.
>
> We wrote more about it here:
> https://blog.cloudflare.com/how-we-scaled-nginx-and-saved-the-world-54-years-every-day/
First of all, thank you for your work. The numbers provided in
the article looks promising.
We've thought about offloading more operations to thread pools,
and that's basically why "aio_write" is a separate directive, so
more directives like "aio_" can be added in the
future. It's good to see this is finally happening.
Unfortunately, there are number problems with the patch, and it
needs more work before it can be committed. In particular:
- The open() calls are offloaded to thread pools unconditionally.
This may actually reduce performance in various typical
workloads where open() does not block. Instead, there should be
an "aio_open on|off" directive, similar to "aio_write".
- The code bypasses open file cache, and uses a direct call
in the http cache code instead. While it might be ok in your
setup, it looks like an incomplete solution from the generic point
of view. A better solution would be to introduce a generic
interface in ngx_open_cached_file() to allow use of thread
pools.
- The code calls ngx_open_and_stat_file() whithin a thread, which
is relatively complex function never designed to be thread safe.
While it looks like currently it is, a better solution would be to
introduce a separate simple function to execute within a thread,
similar to ngx_thread_read_handler().
- In the ngx_http_file_cache_thread_open() function you allocate
thread task of size sizeof(ngx_thread_file_open_ctx_t) and store
it in the file->thread_task. There is no guarantee that this
task will be compatible with other tasks stored in
file->thread_task. A better solution would be to extend
ngx_thread_file_ctx_t and use the same context for all
file-related threaded operations.
- A better place for the thread-specific code would be
src/os/unix/ngx_files.c, where ngx_thread_file_ctx_t and
existing threaded file operations are defined.
- The code uses memory pool operations wihin a thread,
specifically ngx_pool_cleanup_add(). Memory pools are not
thread safe, and the behaviour of this code is therefore
undefined.
- Also there are various style issues, including wrong
capitalization of the commit log summary line, strange commit
log indentation and header, missing empty lines. Please take a
look at http://nginx.org/en/docs/contributing_changes.html for
various hints. In particular, you may want to use Mercurial to
submit patches.
See also below for some in-line comments.
>
> diff --git src/core/ngx_open_file_cache.c src/core/ngx_open_file_cache.c
> index b23ee78d..9177ebfd 100644
> --- src/core/ngx_open_file_cache.c
> +++ src/core/ngx_open_file_cache.c
> @@ -35,8 +35,6 @@ static ngx_fd_t ngx_open_file_wrapper(ngx_str_t *name,
> ngx_int_t access, ngx_log_t *log);
> static ngx_int_t ngx_file_info_wrapper(ngx_str_t *name,
> ngx_open_file_info_t *of, ngx_file_info_t *fi, ngx_log_t *log);
> -static ngx_int_t ngx_open_and_stat_file(ngx_str_t *name,
> - ngx_open_file_info_t *of, ngx_log_t *log);
> static void ngx_open_file_add_event(ngx_open_file_cache_t *cache,
> ngx_cached_open_file_t *file, ngx_open_file_info_t *of, ngx_log_t *log);
> static void ngx_open_file_cleanup(void *data);
> @@ -836,7 +834,7 @@ ngx_file_info_wrapper(ngx_str_t *name,
> ngx_open_file_info_t *of,
> }
>
>
> -static ngx_int_t
> +ngx_int_t
> ngx_open_and_stat_file(ngx_str_t *name, ngx_open_file_info_t *of,
> ngx_log_t *log)
> {
> diff --git src/core/ngx_open_file_cache.h src/core/ngx_open_file_cache.h
> index d119c129..94b4d552 100644
> --- src/core/ngx_open_file_cache.h
> +++ src/core/ngx_open_file_cache.h
> @@ -124,6 +124,8 @@ ngx_open_file_cache_t
> *ngx_open_file_cache_init(ngx_pool_t *pool,
> ngx_uint_t max, time_t inactive);
> ngx_int_t ngx_open_cached_file(ngx_open_file_cache_t *cache, ngx_str_t *name,
> ngx_open_file_info_t *of, ngx_pool_t *pool);
> +ngx_int_t ngx_open_and_stat_file(ngx_str_t *name,
> + ngx_open_file_info_t *of, ngx_log_t *log);
>
>
> #endif /* _NGX_OPEN_FILE_CACHE_H_INCLUDED_ */
> diff --git src/http/ngx_http_cache.h src/http/ngx_http_cache.h
> index f9e96640..2910f6a1 100644
> --- src/http/ngx_http_cache.h
> +++ src/http/ngx_http_cache.h
> @@ -114,6 +114,7 @@ struct ngx_http_cache_s {
> unsigned exists:1;
> unsigned temp_file:1;
> unsigned purged:1;
> + unsigned opening:1;
> unsigned reading:1;
> unsigned secondary:1;
> unsigned background:1;
> diff --git src/http/ngx_http_file_cache.c src/http/ngx_http_file_cache.c
> index 56866fa4..9cb110bf 100644
> --- src/http/ngx_http_file_cache.c
> +++ src/http/ngx_http_file_cache.c
> @@ -18,6 +18,10 @@ static void
> ngx_http_file_cache_lock_wait(ngx_http_request_t *r,
> ngx_http_cache_t *c);
> static ngx_int_t ngx_http_file_cache_read(ngx_http_request_t *r,
> ngx_http_cache_t *c);
> +static ngx_int_t ngx_http_file_cache_aio_open(ngx_http_request_t *r,
> + ngx_http_cache_t *c, ngx_int_t *rv);
> +static ngx_int_t ngx_http_file_cache_do_aio_open(ngx_http_request_t *r,
> + ngx_http_cache_t *c, ngx_open_file_info_t *of, ngx_int_t *rv);
> static ssize_t ngx_http_file_cache_aio_read(ngx_http_request_t *r,
> ngx_http_cache_t *c);
> #if (NGX_HAVE_FILE_AIO)
> @@ -268,9 +272,7 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
> ngx_uint_t test;
> ngx_http_cache_t *c;
> ngx_pool_cleanup_t *cln;
> - ngx_open_file_info_t of;
> ngx_http_file_cache_t *cache;
> - ngx_http_core_loc_conf_t *clcf;
>
> c = r->cache;
>
> @@ -278,6 +280,10 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
> return NGX_AGAIN;
> }
>
> + if (c->opening) {
> + return ngx_http_file_cache_aio_open(r, c, &rv);
> + }
> +
The value of rv is uninitialized here. While it is not strictly a
bug, but this code indicate bad design. Also, duplicate handling
of "rv == NGX_DECLINED" below and in the
ngx_http_file_cache_aio_open() function contributes to the
problem, as well as passing the rv value to the threaded operation
where it is not needed. It might be a good idea to re-think the
design.
> if (c->reading) {
> return ngx_http_file_cache_read(r, c);
> }
> @@ -343,23 +349,33 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
> goto done;
> }
>
> - clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
> + return ngx_http_file_cache_aio_open(r, c, &rv);
>
> - ngx_memzero(&of, sizeof(ngx_open_file_info_t));
> +done:
>
> - of.uniq = c->uniq;
> - of.valid = clcf->open_file_cache_valid;
> - of.min_uses = clcf->open_file_cache_min_uses;
> - of.events = clcf->open_file_cache_events;
> - of.directio = NGX_OPEN_FILE_DIRECTIO_OFF;
> - of.read_ahead = clcf->read_ahead;
> + if (rv == NGX_DECLINED) {
> + return ngx_http_file_cache_lock(r, c);
> + }
>
> - if (ngx_open_cached_file(clcf->open_file_cache, &c->file.name,
> &of, r->pool)
> - != NGX_OK)
> - {
> - switch (of.err) {
> + return rv;
> +}
> +
> +static ngx_int_t
Style: there should be two empty lines between functions.
> +ngx_http_file_cache_aio_open(ngx_http_request_t *r, ngx_http_cache_t *c,
> + ngx_int_t *rv)
Style: function arguments should be indented with 4 spaces.
> +{
> + ngx_int_t rc;
> + ngx_open_file_info_t of;
> + ngx_http_file_cache_t *cache = c->file_cache;
>
> - case 0:
> + rc = ngx_http_file_cache_do_aio_open(r, c, &of, rv);
> + if (rc == NGX_AGAIN) {
> + return rc;
> + }
> +
> + if (rc != NGX_OK) {
> + switch (of.err) {
> + case NGX_OK:
> return NGX_ERROR;
>
> case NGX_ENOENT:
> @@ -391,14 +407,13 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
>
> done:
>
> - if (rv == NGX_DECLINED) {
> + if (*rv == NGX_DECLINED) {
> return ngx_http_file_cache_lock(r, c);
> }
>
> - return rv;
> + return *rv;
> }
>
> -
> static ngx_int_t
Style: there should be two empty lines between functions:
> ngx_http_file_cache_lock(ngx_http_request_t *r, ngx_http_cache_t *c)
> {
> @@ -663,6 +678,127 @@ ngx_http_file_cache_read(ngx_http_request_t *r,
> ngx_http_cache_t *c)
> }
>
>
> +#if (NGX_THREADS)
> +typedef struct {
Style: as long as #if contains something complex, it should be
separated with empty lines.
> + ngx_str_t name;
> + ngx_pool_t *pool;
> + ngx_open_file_info_t of;
> + ngx_int_t rv;
> +} ngx_thread_file_open_ctx_t;
> +
> +static void
> +ngx_http_file_cache_thread_open_handler(void *data, ngx_log_t *log)
> +{
> + ngx_thread_file_open_ctx_t *ctx = data;
> + ngx_pool_t *pool = ctx->pool;
> + ngx_open_file_info_t *of = &ctx->of;
> + ngx_pool_cleanup_t *cln;
> + ngx_pool_cleanup_file_t *clnf;
> + ngx_int_t rc;
Style: variable types should be sorted from shortest to longest;
variable names should be separated with two spaces from the
longest type; assignments should be written separately, except
some specific cases where one assignment is allowed in a separate
variables block.
> +
> + ngx_log_debug0(NGX_LOG_DEBUG_CORE, log, 0, "thread read handler");
The "read" here is incorrect, should be "open".
> +
> + cln = ngx_pool_cleanup_add(pool, sizeof(ngx_pool_cleanup_file_t));
> + if (cln == NULL) {
> + ctx->of.err = NGX_ERROR;
> + return;
> + }
The code is executed in a thread, and using request pool here is
not allowed.
> +
> + of->fd = NGX_INVALID_FILE;
> +
> + rc = ngx_open_and_stat_file(&ctx->name, of, pool->log);
> + if (rc == NGX_OK && !of->is_dir) {
> + cln->handler = ngx_pool_cleanup_file;
> + clnf = cln->data;
> +
> + clnf->fd = of->fd;
> + clnf->name = ctx->name.data;
> + clnf->log = pool->log;
> + }
> +}
> +
> +
> +static ngx_int_t
> +ngx_http_file_cache_thread_open(ngx_http_cache_t *c, ngx_open_file_info_t *of,
> + ngx_int_t *rv, ngx_pool_t *pool)
> +{
> + ngx_thread_task_t *task;
> + ngx_thread_file_open_ctx_t *ctx;
> + ngx_file_t *file = &c->file;
Style, see above.
> +
> + task = file->thread_task;
> +
> + if (task == NULL) {
> + task = ngx_thread_task_alloc(pool, sizeof(ngx_thread_file_open_ctx_t));
> + if (task == NULL) {
> + return NGX_ERROR;
> + }
> +
> + file->thread_task = task;
> + }
See above, this needs some guarantee that what's stored in
file->thread_task is compatible with all operations.
> +
> + ctx = task->ctx;
> +
> + if (task->event.complete) {
> + task->event.complete = 0;
> +
> + *of = ctx->of;
> + *rv = ctx->rv;
> + return of->err;
> + }
> +
> + task->handler = ngx_http_file_cache_thread_open_handler;
> +
> + ctx->pool = pool;
> + ctx->name = file->name;
> + ctx->of = *of;
> + ctx->rv = *rv;
> +
> + if (file->thread_handler(task, file) != NGX_OK) {
> + return NGX_ERROR;
> + }
> +
> + return NGX_AGAIN;
> +}
> +#endif
> +
> +static ngx_int_t
Style: there should be an empty line before "#endif", and two
empty lines between "#endif" and the next function.
> +ngx_http_file_cache_do_aio_open(ngx_http_request_t *r, ngx_http_cache_t *c,
> + ngx_open_file_info_t *of, ngx_int_t *rv)
> +{
> + ngx_http_core_loc_conf_t *clcf;
> +
> + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
> +
> + ngx_memzero(of, sizeof(ngx_open_file_info_t));
> +
> + of->uniq = c->uniq;
> + of->valid = clcf->open_file_cache_valid;
> + of->min_uses = clcf->open_file_cache_min_uses;
> + of->events = clcf->open_file_cache_events;
> + of->directio = NGX_OPEN_FILE_DIRECTIO_OFF;
> + of->read_ahead = clcf->read_ahead;
> +
> +#if (NGX_THREADS)
> + if (clcf->aio == NGX_HTTP_AIO_THREADS) {
> + ngx_int_t rc;
Style: there should be two spaces between a type and a variable
name; there should be an empty line after #if (and before #endif).
> +
> + c->file.thread_task = c->thread_task;
> + c->file.thread_handler = ngx_http_cache_thread_handler;
> + c->file.thread_ctx = r;
> +
> + rc = ngx_http_file_cache_thread_open(c, of, rv, r->pool);
> +
> + c->thread_task = c->file.thread_task;
> + c->opening = (rc == NGX_AGAIN);
> +
> + return rc;
> + }
> +#endif
> +
> + return ngx_open_cached_file(clcf->open_file_cache, &c->file.name,
> of, r->pool);
Style: maximum text width is 80 characters.
> +}
> +
> static ssize_t
> ngx_http_file_cache_aio_read(ngx_http_request_t *r, ngx_http_cache_t *c)
> {
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
--
Maxim Dounin
http://mdounin.ru/
From mdounin at mdounin.ru Wed Aug 8 18:34:05 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 8 Aug 2018 21:34:05 +0300
Subject: post body
In-Reply-To:
References:
Message-ID: <20180808183405.GY56558@mdounin.ru>
Hello!
On Tue, Aug 07, 2018 at 10:46:08PM -0700, Dk Jack wrote:
> Haven't received any response to inquiry. Would appreciate it if someone
> can comment.
>
> > In my module, I'd like to look into the post body to decide if the request
> > should be allowed to proceed to origin server or not. Based on the examples
> > I could find I've coded this in my module. After some trial and error it
> > seems to be working. I am not sure if my implementation is 100% correct. I
> > would appreciate it if someone can go over my code below and let me know if
> > my implementation is correct. Especially, the body_complete_handler. Thanks.
You may want to read general request body reading guidelines here:
http://nginx.org/en/docs/dev/development_guide.html#http_request_body
Your code suggests you've never seen these guidelines, so consider
re-writing it after reading the above link.
Note that the above guidelines cover content phase handling. In
other request processing phases reqding the request body might be
tricky and will require additional handling (if at all possible).
In particular, you'll have to restore request phases processing
after the call, and you might have to call
ngx_http_finalize_request() yourself. Consider looking into
src/http/modules/ngx_http_mirror_module.c for an example. Note
well that required handling might depend on the exact request
processing phase you are trying to use.
Note well that if you are trying to inspec the request body,
writing a request body filter might be a better solution. See
here for a trivial example:
http://mdounin.ru/hg/ngx_http_catch_body_filter_module/file/tip/ngx_http_catch_body_filter_module.c
--
Maxim Dounin
http://mdounin.ru/
From dnj0496 at gmail.com Wed Aug 8 18:57:33 2018
From: dnj0496 at gmail.com (Dk Jack)
Date: Wed, 8 Aug 2018 11:57:33 -0700
Subject: post body
In-Reply-To: <20180808183405.GY56558@mdounin.ru>
References:
<20180808183405.GY56558@mdounin.ru>
Message-ID:
Thank you. Will looking it and get back. Since I need to inspect, I'll
follow your suggestion
and see if I can use the body filter option.
regards,
Dk
On Wed, Aug 8, 2018 at 11:34 AM Maxim Dounin wrote:
> Hello!
>
> On Tue, Aug 07, 2018 at 10:46:08PM -0700, Dk Jack wrote:
>
> > Haven't received any response to inquiry. Would appreciate it if someone
> > can comment.
> >
> > > In my module, I'd like to look into the post body to decide if the
> request
> > > should be allowed to proceed to origin server or not. Based on the
> examples
> > > I could find I've coded this in my module. After some trial and error
> it
> > > seems to be working. I am not sure if my implementation is 100%
> correct. I
> > > would appreciate it if someone can go over my code below and let me
> know if
> > > my implementation is correct. Especially, the body_complete_handler.
> Thanks.
>
> You may want to read general request body reading guidelines here:
>
> http://nginx.org/en/docs/dev/development_guide.html#http_request_body
>
> Your code suggests you've never seen these guidelines, so consider
> re-writing it after reading the above link.
>
> Note that the above guidelines cover content phase handling. In
> other request processing phases reqding the request body might be
> tricky and will require additional handling (if at all possible).
> In particular, you'll have to restore request phases processing
> after the call, and you might have to call
> ngx_http_finalize_request() yourself. Consider looking into
> src/http/modules/ngx_http_mirror_module.c for an example. Note
> well that required handling might depend on the exact request
> processing phase you are trying to use.
>
> Note well that if you are trying to inspec the request body,
> writing a request body filter might be a better solution. See
> here for a trivial example:
>
>
> http://mdounin.ru/hg/ngx_http_catch_body_filter_module/file/tip/ngx_http_catch_body_filter_module.c
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From eran.kornblau at kaltura.com Wed Aug 8 19:44:20 2018
From: eran.kornblau at kaltura.com (Eran Kornblau)
Date: Wed, 8 Aug 2018 19:44:20 +0000
Subject: cache: move open to thread pool
In-Reply-To: <20180808181653.GX56558@mdounin.ru>
References:
<20180808181653.GX56558@mdounin.ru>
Message-ID:
Hi,
>
> - The code bypasses open file cache, and uses a direct call
> in the http cache code instead. While it might be ok in your
> setup, it looks like an incomplete solution from the generic point
> of view. A better solution would be to introduce a generic
> interface in ngx_open_cached_file() to allow use of thread
> pools.
>
A small comment on this - I wrote such an implementation a while ago
in my module. We've been using it on production for the last ~3 years.
Code is here -
https://github.com/kaltura/nginx-vod-module/blob/master/ngx_async_open_file_cache.c
(There's a lot of code there that was copied from ngx_open_file_cache.c,
I did that since those functions are static, and didn't want to change nginx core.
If implemented as part of nginx core, all this duplication can be avoided...)
In my implementation, I added a function similar to ngx_open_cached_file -
(ngx_async_open_cached_file), that gets a few extra params -
1. The thread pool
2. The thread task (optional) - this was done in order to reuse the task
when opening multiple files in a single request
3. Callback + context - invoked once the async open completes
This function first checks the cache, if there's a hit, it returns synchronously
(NGX_OK).
Otherwise, it posts a task that does ngx_open_and_stat_file (NGX_AGAIN).
When the task completes, the main nginx thread updates the cache, and invokes
the user callback.
Hope this helps...
Eran
From mdounin at mdounin.ru Thu Aug 9 09:26:37 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 09 Aug 2018 09:26:37 +0000
Subject: [nginx] Skipping spaces in configuration files (ticket #1557).
Message-ID:
details: http://hg.nginx.org/nginx/rev/f17e313009b0
branches:
changeset: 7334:f17e313009b0
user: Maxim Dounin
date: Thu Aug 09 12:15:42 2018 +0300
description:
Skipping spaces in configuration files (ticket #1557).
Previously, a chunk of spaces larger than NGX_CONF_BUFFER (4096 bytes)
resulted in the "too long parameter" error during parsing such a
configuration. This was because the code only set start and start_line
on non-whitespace characters, and hence adjacent whitespace characters
were preserved when reading additional data from the configuration file.
Fix is to always move start and start_line if the last character was
a space.
diffstat:
src/core/ngx_conf_file.c | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
diffs (21 lines):
diff --git a/src/core/ngx_conf_file.c b/src/core/ngx_conf_file.c
--- a/src/core/ngx_conf_file.c
+++ b/src/core/ngx_conf_file.c
@@ -656,13 +656,14 @@ ngx_conf_read_token(ngx_conf_t *cf)
}
if (last_space) {
+
+ start = b->pos - 1;
+ start_line = cf->conf_file->line;
+
if (ch == ' ' || ch == '\t' || ch == CR || ch == LF) {
continue;
}
- start = b->pos - 1;
- start_line = cf->conf_file->line;
-
switch (ch) {
case ';':
From mdounin at mdounin.ru Thu Aug 9 17:46:33 2018
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 09 Aug 2018 17:46:33 +0000
Subject: [nginx] HTTP/2: workaround for clients which fail on table size
updates.
Message-ID:
details: http://hg.nginx.org/nginx/rev/fbb683496705
branches:
changeset: 7335:fbb683496705
user: Maxim Dounin
date: Thu Aug 09 20:12:17 2018 +0300
description:
HTTP/2: workaround for clients which fail on table size updates.
There are clients which cannot handle HPACK's dynamic table size updates
as added in 12cadc4669a7 (1.13.6). Notably, old versions of OkHttp library
are known to fail on it (ticket #1397).
This change makes it possible to work with such clients by only sending
dynamic table size updates in response to SETTINGS_HEADER_TABLE_SIZE. As
a downside, clients which do not use SETTINGS_HEADER_TABLE_SIZE will
continue to maintain default 4k table.
diffstat:
src/http/v2/ngx_http_v2.c | 7 +++++--
1 files changed, 5 insertions(+), 2 deletions(-)
diffs (24 lines):
diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c
+++ b/src/http/v2/ngx_http_v2.c
@@ -270,8 +270,6 @@ ngx_http_v2_init(ngx_event_t *rev)
h2c->frame_size = NGX_HTTP_V2_DEFAULT_FRAME_SIZE;
- h2c->table_update = 1;
-
h2scf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_v2_module);
h2c->concurrent_pushes = h2scf->concurrent_pushes;
@@ -2075,6 +2073,11 @@ ngx_http_v2_state_settings_params(ngx_ht
h2c->concurrent_pushes = ngx_min(value, h2scf->concurrent_pushes);
break;
+ case NGX_HTTP_V2_HEADER_TABLE_SIZE_SETTING:
+
+ h2c->table_update = 1;
+ break;
+
default:
break;
}
From kahing at cloudflare.com Thu Aug 9 21:43:19 2018
From: kahing at cloudflare.com (Ka-Hing Cheung)
Date: Thu, 9 Aug 2018 14:43:19 -0700
Subject: cache: move open to thread pool
In-Reply-To: <20180808181653.GX56558@mdounin.ru>
References:
<20180808181653.GX56558@mdounin.ru>
Message-ID:
Very good feedback, we will look into implementing some of them at least.
On Wed, Aug 8, 2018 at 11:16 AM, Maxim Dounin wrote:
>
> We've thought about offloading more operations to thread pools,
> and that's basically why "aio_write" is a separate directive, so
> more directives like "aio_" can be added in the
> future. It's good to see this is finally happening.
>
> Unfortunately, there are number problems with the patch, and it
> needs more work before it can be committed. In particular:
>
> - The open() calls are offloaded to thread pools unconditionally.
> This may actually reduce performance in various typical
> workloads where open() does not block. Instead, there should be
> an "aio_open on|off" directive, similar to "aio_write".
>
> - The code bypasses open file cache, and uses a direct call
> in the http cache code instead. While it might be ok in your
> setup, it looks like an incomplete solution from the generic point
> of view. A better solution would be to introduce a generic
> interface in ngx_open_cached_file() to allow use of thread
> pools.
Agreed that not everyone wants threaded open() with aio "threads" (if
low CPU overhead is more important than low latency for example). The
semantic of "aio_open on" is uncleared though when aio is on but not
"threads". Having open file cache working with threaded open is nice
to have (although people who need "aio_open" probably also have so
much cached assets that they won't need open file cache).
>
> - The code calls ngx_open_and_stat_file() whithin a thread, which
> is relatively complex function never designed to be thread safe.
> While it looks like currently it is, a better solution would be to
> introduce a separate simple function to execute within a thread,
> similar to ngx_thread_read_handler().
agreed.
>
> - In the ngx_http_file_cache_thread_open() function you allocate
> thread task of size sizeof(ngx_thread_file_open_ctx_t) and store
> it in the file->thread_task. There is no guarantee that this
> task will be compatible with other tasks stored in
> file->thread_task. A better solution would be to extend
> ngx_thread_file_ctx_t and use the same context for all
> file-related threaded operations.
does this matter? the different thread_task won't overlap in their
usage (you can't read a file until after its opened) so there's no
reason they need to be compatible. Isn't that why the task ctx is void
* and ngx_thread_task_alloc takes a generic size?
>
> - A better place for the thread-specific code would be
> src/os/unix/ngx_files.c, where ngx_thread_file_ctx_t and
> existing threaded file operations are defined.
sure
>
> - The code uses memory pool operations wihin a thread,
> specifically ngx_pool_cleanup_add(). Memory pools are not
> thread safe, and the behaviour of this code is therefore
> undefined.
Agreed that it may not be very clean but is this a problem in
practice? The pool is tied to the request and shouldn't shared with
other threads. Asking mostly to clarify my understandings with nginx
memory pools.
>> @@ -278,6 +280,10 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
>> return NGX_AGAIN;
>> }
>>
>> + if (c->opening) {
>> + return ngx_http_file_cache_aio_open(r, c, &rv);
>> + }
>> +
>
> The value of rv is uninitialized here. While it is not strictly a
> bug, but this code indicate bad design. Also, duplicate handling
> of "rv == NGX_DECLINED" below and in the
> ngx_http_file_cache_aio_open() function contributes to the
> problem, as well as passing the rv value to the threaded operation
> where it is not needed. It might be a good idea to re-think the
> design.
Some of this is because we have other internal changes here. Will try
to clean this up better.
The rest of the styling suggestions are all reasonable.
- Ka-Hing
From maxim at nginx.com Fri Aug 10 11:15:20 2018
From: maxim at nginx.com (Maxim Konovalov)
Date: Fri, 10 Aug 2018 11:15:20 +0000
Subject: [nginx] A link to the error_log directive on nginx.org removed.
Message-ID:
details: http://hg.nginx.org/nginx/rev/1cd63ae46243
branches:
changeset: 7336:1cd63ae46243
user: Maxim Konovalov
date: Fri Aug 10 14:15:05 2018 +0300
description:
A link to the error_log directive on nginx.org removed.
It makes more harm than good for users and nginx.org
infrastructure.
diffstat:
docs/html/50x.html | 2 +-
1 ?????? ????????, 1 ???????(+), 1 ????????(-)
???????? (12 ?????):
diff -r fbb683496705 -r 1cd63ae46243 docs/html/50x.html
--- a/docs/html/50x.html Thu Aug 09 20:12:17 2018 +0300
+++ b/docs/html/50x.html Fri Aug 10 14:15:05 2018 +0300
@@ -15,7 +15,7 @@
Sorry, the page you are looking for is currently unavailable.
Please try again later.
If you are the system administrator of this resource then you should check
-the error log for details.
+the error log for details.
Faithfully yours, nginx.