From otto at seravo.fi  Mon Jan  2 14:06:15 2017
From: otto at seravo.fi (=?UTF-8?B?T3R0byBLZWvDpGzDpGluZW4=?=)
Date: Mon, 2 Jan 2017 16:06:15 +0200
Subject: Include typical font files like ttf in the default mime.types
Message-ID: <CAHj_TLA=wCV-AUjf6B7GtKTsJAoCGM8dKRy4WdwcTsGy66UENQ@mail.gmail.com>

Hello!

I noticed the default mime.types in Nginx does not include some very
common assets, like ttf font files:
https://github.com/nginx/nginx/blob/master/conf/mime.types

I suggest the mime.types should be extended to include at least these:

    application/x-font-ttf                ttf;
    application/vnd.ms-opentype           otf;


A fairly extensive and well tested list combiled by multiple people
with HB5P authority can be viewed at:
https://github.com/h5bp/server-configs-nginx/blob/master/mime.types

I'd recommend adopting the full listing above to fix the font issues
and probably many other issues in one go.

From mdounin at mdounin.ru  Mon Jan  2 23:49:35 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 3 Jan 2017 02:49:35 +0300
Subject: Include typical font files like ttf in the default mime.types
In-Reply-To: <CAHj_TLA=wCV-AUjf6B7GtKTsJAoCGM8dKRy4WdwcTsGy66UENQ@mail.gmail.com>
References: <CAHj_TLA=wCV-AUjf6B7GtKTsJAoCGM8dKRy4WdwcTsGy66UENQ@mail.gmail.com>
Message-ID: <20170102234935.GA93993@mdounin.ru>

Hello!

On Mon, Jan 02, 2017 at 04:06:15PM +0200, Otto Kek?l?inen wrote:

> I noticed the default mime.types in Nginx does not include some very
> common assets, like ttf font files:
> https://github.com/nginx/nginx/blob/master/conf/mime.types
> 
> I suggest the mime.types should be extended to include at least these:
> 
>     application/x-font-ttf                ttf;
>     application/vnd.ms-opentype           otf;

As far as I can tell, both .ttf and .otf are handled fine when 
served as application/octet-stream.  Do you have any specific 
reasons to add these types?

Note that nginx doesn't try to maintain all the mime types known, 
but rather tries to supply types generally needed for a web 
server.

> A fairly extensive and well tested list combiled by multiple people
> with HB5P authority can be viewed at:
> https://github.com/h5bp/server-configs-nginx/blob/master/mime.types

Note the link suggested uses "font/otf" for OpenType fonts instead.

> I'd recommend adopting the full listing above to fix the font issues
> and probably many other issues in one go.

No, thanks.

-- 
Maxim Dounin
http://nginx.org/

From josh at joshuaspence.com  Tue Jan  3 06:24:59 2017
From: josh at joshuaspence.com (josh at joshuaspence.com)
Date: Tue, 03 Jan 2017 17:24:59 +1100
Subject: [PATCH] Escape strings produced by the `%V` format specification
Message-ID: <8e734d01f026c4346b46.1483424699@deagle>

# HG changeset patch
# User Joshua Spence <josh at joshuaspence.com>
# Date 1483420178 -39600
#      Tue Jan 03 16:09:38 2017 +1100
# Node ID 8e734d01f026c4346b46006ee9b558031439be61
# Parent  a42afc225e98afe1f7c3b428c81c8af7eba362c0
Escape strings produced by the `%V` format specification

Currently nginx error logs loosely adhere to the following pattern:

```
%{TIMESTAMP} [%{SEVERITY}] %{PROCESS_ID}#%{THREAD_ID}:( *%{CONNECTION_ID})?%{MESSAGE}(, %{KEY}: "%{VALUE}")*
```

In our environment we are using Filebeat to ship nginx error logs into Logstash. Our Logstash filter for parsing nginx error logs is as follows:

```
filter {
  grok {
    match => {
      "message" => "(?<timestamp>%{YEAR}/%{MONTHNUM2}/%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:process_id:int}#%{NONNEGINT:thread_id:int}:(?: \*%{NONNEGINT:connection_id:int})? (?<message>[^,]*)(?:, %{GREEDYDATA:nginx_keyvalue_data})?"
    }
    overwrite => ["message"]
  }

  date {
    match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
    remove_field => ["timestamp"]
  }

  kv {
    field_split => ", "
    remove_field => "nginx_keyvalue_data"
    source => "nginx_keyvalue_data"
    value_split => ": "
  }
}
```

The problem, however, is that it is difficult to unambigiously parse the key-value data from the nginx error log format because of a lack of data sanitization. Specifically, consider the following entry from our nginx error logs:

```
2016/11/15 03:45:51 [error] 23770#23770: *21535000 open() "/REDACTED/+"BauGutachter"++41+2016+.ch&tbo=1" failed (2: No such file or directory), client: REDACTED, server: REDACTED, request: "GET /job/+%22BauGutachter%22++41+2016+.ch&tbo=1 HTTP/1.1", host: "REDACTED", referrer: "https://www.google.com.au/search?hl=en&q="BauGutachter" +41 2016 .ch&num=100&start=0"
```

Specifically, the problem in the above sample is the `referrer` field, the value of which contains double quotes. I think that it makes sense to escape `\` and `"` characters before they are written to the error log.

diff -r a42afc225e98 -r 8e734d01f026 src/core/ngx_string.c
--- a/src/core/ngx_string.c	Tue Dec 27 17:23:08 2016 +0300
+++ b/src/core/ngx_string.c	Tue Jan 03 16:09:38 2017 +1100
@@ -15,7 +15,7 @@
     const u_char *basis, ngx_uint_t padding);
 static ngx_int_t ngx_decode_base64_internal(ngx_str_t *dst, ngx_str_t *src,
     const u_char *basis);
-
+static uintptr_t ngx_escape_string(u_char *dst, u_char *src, size_t size);
 
 void
 ngx_strlow(u_char *dst, u_char *src, size_t n)
@@ -235,7 +235,7 @@
                 v = va_arg(args, ngx_str_t *);
 
                 len = ngx_min(((size_t) (last - buf)), v->len);
-                buf = ngx_cpymem(buf, v->data, len);
+                buf = (u_char *) ngx_escape_string(buf, v->data, len);
                 fmt++;
 
                 continue;
@@ -1844,6 +1844,43 @@
 }
 
 
+static uintptr_t
+ngx_escape_string(u_char *dst, u_char *src, size_t size)
+{
+    u_char      ch;
+    ngx_uint_t  len;
+
+    if (dst == NULL) {
+        len = 0;
+
+        while (size) {
+            ch = *src++;
+
+            if (ch == '\\' || ch == '"') {
+                len++;
+            }
+
+            size--;
+        }
+
+        return (uintptr_t) len;
+    }
+
+    while (size) {
+        ch = *src++;
+
+        if (ch == '\\' || ch == '"') {
+            *dst++ = '\\';
+        }
+
+        *dst++ = ch;
+        size--;
+    }
+
+    return (uintptr_t) dst;
+}
+
+
 void
 ngx_str_rbtree_insert_value(ngx_rbtree_node_t *temp,
     ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel)

From josh at joshuaspence.com  Tue Jan  3 06:38:02 2017
From: josh at joshuaspence.com (Joshua Spence)
Date: Tue, 3 Jan 2017 17:38:02 +1100
Subject: [PATCH] Escape strings produced by the `%V` format specification
In-Reply-To: <8e734d01f026c4346b46.1483424699@deagle>
References: <8e734d01f026c4346b46.1483424699@deagle>
Message-ID: <CADO4dzNaAE8sQco5bNzFKbc=o7DH9QwHTCY4c60QNXFHpL16hA@mail.gmail.com>

Hmm, I'm not sure if I submitted this correctly because it is showing up on
https://forum.nginx.org/read.php?29,271801 from "Anonymous user". I'll see
if I can figure out what I did wrong. Sorry, I haven't submitted patches
via a mailing list before.

*Joshua Spence*

*mobile:* +61 422 990 263
*email:* josh at joshuaspence.com
*web:* http://www.joshuaspence.com

On 3 January 2017 at 17:24, <josh at joshuaspence.com> wrote:

> # HG changeset patch
> # User Joshua Spence <josh at joshuaspence.com>
> # Date 1483420178 -39600
> #      Tue Jan 03 16:09:38 2017 +1100
> # Node ID 8e734d01f026c4346b46006ee9b558031439be61
> # Parent  a42afc225e98afe1f7c3b428c81c8af7eba362c0
> Escape strings produced by the `%V` format specification
>
> Currently nginx error logs loosely adhere to the following pattern:
>
> ```
> %{TIMESTAMP} [%{SEVERITY}] %{PROCESS_ID}#%{THREAD_ID}:(
> *%{CONNECTION_ID})?%{MESSAGE}(, %{KEY}: "%{VALUE}")*
> ```
>
> In our environment we are using Filebeat to ship nginx error logs into
> Logstash. Our Logstash filter for parsing nginx error logs is as follows:
>
> ```
> filter {
>   grok {
>     match => {
>       "message" => "(?<timestamp>%{YEAR}/%{MONTHNUM2}/%{MONTHDAY}
> %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:process_id:int}#%{NONNEGINT:thread_id:int}:(?:
> \*%{NONNEGINT:connection_id:int})? (?<message>[^,]*)(?:,
> %{GREEDYDATA:nginx_keyvalue_data})?"
>     }
>     overwrite => ["message"]
>   }
>
>   date {
>     match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
>     remove_field => ["timestamp"]
>   }
>
>   kv {
>     field_split => ", "
>     remove_field => "nginx_keyvalue_data"
>     source => "nginx_keyvalue_data"
>     value_split => ": "
>   }
> }
> ```
>
> The problem, however, is that it is difficult to unambigiously parse the
> key-value data from the nginx error log format because of a lack of data
> sanitization. Specifically, consider the following entry from our nginx
> error logs:
>
> ```
> 2016/11/15 03:45:51 [error] 23770#23770: *21535000 open()
> "/REDACTED/+"BauGutachter"++41+2016+.ch&tbo=1" failed (2: No such file or
> directory), client: REDACTED, server: REDACTED, request: "GET
> /job/+%22BauGutachter%22++41+2016+.ch&tbo=1 HTTP/1.1", host: "REDACTED",
> referrer: "https://www.google.com.au/search?hl=en&q="BauGutachter" +41
> 2016 .ch&num=100&start=0"
> ```
>
> Specifically, the problem in the above sample is the `referrer` field, the
> value of which contains double quotes. I think that it makes sense to
> escape `\` and `"` characters before they are written to the error log.
>
> diff -r a42afc225e98 -r 8e734d01f026 src/core/ngx_string.c
> --- a/src/core/ngx_string.c     Tue Dec 27 17:23:08 2016 +0300
> +++ b/src/core/ngx_string.c     Tue Jan 03 16:09:38 2017 +1100
> @@ -15,7 +15,7 @@
>      const u_char *basis, ngx_uint_t padding);
>  static ngx_int_t ngx_decode_base64_internal(ngx_str_t *dst, ngx_str_t
> *src,
>      const u_char *basis);
> -
> +static uintptr_t ngx_escape_string(u_char *dst, u_char *src, size_t size);
>
>  void
>  ngx_strlow(u_char *dst, u_char *src, size_t n)
> @@ -235,7 +235,7 @@
>                  v = va_arg(args, ngx_str_t *);
>
>                  len = ngx_min(((size_t) (last - buf)), v->len);
> -                buf = ngx_cpymem(buf, v->data, len);
> +                buf = (u_char *) ngx_escape_string(buf, v->data, len);
>                  fmt++;
>
>                  continue;
> @@ -1844,6 +1844,43 @@
>  }
>
>
> +static uintptr_t
> +ngx_escape_string(u_char *dst, u_char *src, size_t size)
> +{
> +    u_char      ch;
> +    ngx_uint_t  len;
> +
> +    if (dst == NULL) {
> +        len = 0;
> +
> +        while (size) {
> +            ch = *src++;
> +
> +            if (ch == '\\' || ch == '"') {
> +                len++;
> +            }
> +
> +            size--;
> +        }
> +
> +        return (uintptr_t) len;
> +    }
> +
> +    while (size) {
> +        ch = *src++;
> +
> +        if (ch == '\\' || ch == '"') {
> +            *dst++ = '\\';
> +        }
> +
> +        *dst++ = ch;
> +        size--;
> +    }
> +
> +    return (uintptr_t) dst;
> +}
> +
> +
>  void
>  ngx_str_rbtree_insert_value(ngx_rbtree_node_t *temp,
>      ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170103/9aecb2ba/attachment.html>

From josh at joshuaspence.com  Tue Jan  3 06:50:37 2017
From: josh at joshuaspence.com (Joshua Spence)
Date: Tue, 03 Jan 2017 17:50:37 +1100
Subject: [PATCH] Escape strings produced by the `%V` format specification
Message-ID: <30a9ce42aadfcf135f3a.1483426237@deagle>

# HG changeset patch
# User Joshua Spence <josh at joshuaspence.com>
# Date 1483420178 -39600
#      Tue Jan 03 16:09:38 2017 +1100
# Branch error_log_escape
# Node ID 30a9ce42aadfcf135f3ad95a296c95e7044d5f9d
# Parent  a42afc225e98afe1f7c3b428c81c8af7eba362c0
Escape strings produced by the `%V` format specification

Currently nginx error logs loosely adhere to the following pattern:

```
%{TIMESTAMP} [%{SEVERITY}] %{PROCESS_ID}#%{THREAD_ID}:( *%{CONNECTION_ID})?%{MESSAGE}(, %{KEY}: "%{VALUE}")*
```

In our environment we are using Filebeat to ship nginx error logs into Logstash. Our Logstash filter for parsing nginx error logs is as follows:

```
filter {
  grok {
    match => {
      "message" => "(?<timestamp>%{YEAR}/%{MONTHNUM2}/%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:process_id:int}#%{NONNEGINT:thread_id:int}:(?: \*%{NONNEGINT:connection_id:int})? (?<message>[^,]*)(?:, %{GREEDYDATA:nginx_keyvalue_data})?"
    }
    overwrite => ["message"]
  }

  date {
    match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
    remove_field => ["timestamp"]
  }

  kv {
    field_split => ", "
    remove_field => "nginx_keyvalue_data"
    source => "nginx_keyvalue_data"
    value_split => ": "
  }
}
```

The problem, however, is that it is difficult to unambigiously parse the key-value data from the nginx error log format because of a lack of data sanitization. Specifically, consider the following entry from our nginx error logs:

```
2016/11/15 03:45:51 [error] 23770#23770: *21535000 open() "/REDACTED/+"BauGutachter"++41+2016+.ch&tbo=1" failed (2: No such file or directory), client: REDACTED, server: REDACTED, request: "GET /job/+%22BauGutachter%22++41+2016+.ch&tbo=1 HTTP/1.1", host: "REDACTED", referrer: "https://www.google.com.au/search?hl=en&q="BauGutachter" +41 2016 .ch&num=100&start=0"
```

Specifically, the problem in the above sample is the `referrer` field, the value of which contains double quotes. I think that it makes sense to escape `\` and `"` characters before they are written to the error log.

diff -r a42afc225e98 -r 30a9ce42aadf src/core/ngx_string.c
--- a/src/core/ngx_string.c	Tue Dec 27 17:23:08 2016 +0300
+++ b/src/core/ngx_string.c	Tue Jan 03 16:09:38 2017 +1100
@@ -15,7 +15,7 @@
     const u_char *basis, ngx_uint_t padding);
 static ngx_int_t ngx_decode_base64_internal(ngx_str_t *dst, ngx_str_t *src,
     const u_char *basis);
-
+static uintptr_t ngx_escape_string(u_char *dst, u_char *src, size_t size);
 
 void
 ngx_strlow(u_char *dst, u_char *src, size_t n)
@@ -235,7 +235,7 @@
                 v = va_arg(args, ngx_str_t *);
 
                 len = ngx_min(((size_t) (last - buf)), v->len);
-                buf = ngx_cpymem(buf, v->data, len);
+                buf = (u_char *) ngx_escape_string(buf, v->data, len);
                 fmt++;
 
                 continue;
@@ -1844,6 +1844,43 @@
 }
 
 
+static uintptr_t
+ngx_escape_string(u_char *dst, u_char *src, size_t size)
+{
+    u_char      ch;
+    ngx_uint_t  len;
+
+    if (dst == NULL) {
+        len = 0;
+
+        while (size) {
+            ch = *src++;
+
+            if (ch == '\\' || ch == '"') {
+                len++;
+            }
+
+            size--;
+        }
+
+        return (uintptr_t) len;
+    }
+
+    while (size) {
+        ch = *src++;
+
+        if (ch == '\\' || ch == '"') {
+            *dst++ = '\\';
+        }
+
+        *dst++ = ch;
+        size--;
+    }
+
+    return (uintptr_t) dst;
+}
+
+
 void
 ngx_str_rbtree_insert_value(ngx_rbtree_node_t *temp,
     ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel)

From josh at joshuaspence.com  Tue Jan  3 11:14:44 2017
From: josh at joshuaspence.com (Joshua Spence)
Date: Tue, 3 Jan 2017 22:14:44 +1100
Subject: [PATCH] Escape strings produced by the `%V` format specification
In-Reply-To: <30a9ce42aadfcf135f3a.1483426237@deagle>
References: <30a9ce42aadfcf135f3a.1483426237@deagle>
Message-ID: <CADO4dzNgRytA6c_tTAq4WGvZJB7Cies3reyYvAS2M429bcCbDA@mail.gmail.com>

I tested this change with the following script:

```
#!/usr/bin/perl

###############################################################################

use warnings;
use strict;

use Test::More;

BEGIN { use FindBin; chdir($FindBin::Bin); }

use lib 'lib';
use Test::Nginx;

###############################################################################

select STDERR; $| = 1;
select STDOUT; $| = 1;

my $t = Test::Nginx->new()
    ->has(qw/http limit_req/)
    ->plan(2)
    ->write_file_expand('nginx.conf', <<'EOF');

%%TEST_GLOBALS%%

daemon off;

events {}

http {
    %%TEST_GLOBALS_HTTP%%

    limit_req_zone $binary_remote_addr zone=one:1m rate=1r/m;
    limit_req zone=one;

    server {
        listen       127.0.0.1:8080;
        server_name  localhost;

        location / {
            error_log stderr;
        }
    }
}

EOF

open OLDERR, '>&', \*STDERR;
open STDERR, '>', $t->testdir() . '/stderr' or die "Can't reopen STDERR:
$!";
open my $stderr, '<', $t->testdir() . '/stderr' or die "Can't open stderr
file: $!";

$t->run();

open STDERR, '>&', \*OLDERR;

###############################################################################

# charge limit_req
http_get('/');

http_get('/"foo bar"');
is(lines($t, 'request: "GET /\"foo bar\" HTTP/1.0"'), 1);

http_get_with_referrer('/',
'https://www.google.com.au/search?hl=en&q="BauGutachter"
+41 2016 .ch&num=100&start=0');
is(lines($t, 'referrer:
"https://www.google.com.au/search?hl=en&q=\"BauGutachter\"
+41 2016 .ch&num=100&start=0"'), 1);

###############################################################################

sub http_get_with_referrer {
    my ($url, $referer) = @_;
    return http(<<EOF);
GET $url HTTP/1.0
Host: localhost
Referer: $referer

EOF
}

sub lines {
    my ($t, $pattern) = @_;
    return map { $_ =~ /\Q$pattern\E/ } (<$stderr>);
}

###############################################################################
```

*Joshua Spence*

*mobile:* +61 422 990 263
*email:* josh at joshuaspence.com
*web:* http://www.joshuaspence.com

On 3 January 2017 at 17:50, Joshua Spence <josh at joshuaspence.com> wrote:

> # HG changeset patch
> # User Joshua Spence <josh at joshuaspence.com>
> # Date 1483420178 -39600
> #      Tue Jan 03 16:09:38 2017 +1100
> # Branch error_log_escape
> # Node ID 30a9ce42aadfcf135f3ad95a296c95e7044d5f9d
> # Parent  a42afc225e98afe1f7c3b428c81c8af7eba362c0
> Escape strings produced by the `%V` format specification
>
> Currently nginx error logs loosely adhere to the following pattern:
>
> ```
> %{TIMESTAMP} [%{SEVERITY}] %{PROCESS_ID}#%{THREAD_ID}:(
> *%{CONNECTION_ID})?%{MESSAGE}(, %{KEY}: "%{VALUE}")*
> ```
>
> In our environment we are using Filebeat to ship nginx error logs into
> Logstash. Our Logstash filter for parsing nginx error logs is as follows:
>
> ```
> filter {
>   grok {
>     match => {
>       "message" => "(?<timestamp>%{YEAR}/%{MONTHNUM2}/%{MONTHDAY}
> %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:process_id:int}#%{NONNEGINT:thread_id:int}:(?:
> \*%{NONNEGINT:connection_id:int})? (?<message>[^,]*)(?:,
> %{GREEDYDATA:nginx_keyvalue_data})?"
>     }
>     overwrite => ["message"]
>   }
>
>   date {
>     match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
>     remove_field => ["timestamp"]
>   }
>
>   kv {
>     field_split => ", "
>     remove_field => "nginx_keyvalue_data"
>     source => "nginx_keyvalue_data"
>     value_split => ": "
>   }
> }
> ```
>
> The problem, however, is that it is difficult to unambigiously parse the
> key-value data from the nginx error log format because of a lack of data
> sanitization. Specifically, consider the following entry from our nginx
> error logs:
>
> ```
> 2016/11/15 03:45:51 [error] 23770#23770: *21535000 open()
> "/REDACTED/+"BauGutachter"++41+2016+.ch&tbo=1" failed (2: No such file or
> directory), client: REDACTED, server: REDACTED, request: "GET
> /job/+%22BauGutachter%22++41+2016+.ch&tbo=1 HTTP/1.1", host: "REDACTED",
> referrer: "https://www.google.com.au/search?hl=en&q="BauGutachter" +41
> 2016 .ch&num=100&start=0"
> ```
>
> Specifically, the problem in the above sample is the `referrer` field, the
> value of which contains double quotes. I think that it makes sense to
> escape `\` and `"` characters before they are written to the error log.
>
> diff -r a42afc225e98 -r 30a9ce42aadf src/core/ngx_string.c
> --- a/src/core/ngx_string.c     Tue Dec 27 17:23:08 2016 +0300
> +++ b/src/core/ngx_string.c     Tue Jan 03 16:09:38 2017 +1100
> @@ -15,7 +15,7 @@
>      const u_char *basis, ngx_uint_t padding);
>  static ngx_int_t ngx_decode_base64_internal(ngx_str_t *dst, ngx_str_t
> *src,
>      const u_char *basis);
> -
> +static uintptr_t ngx_escape_string(u_char *dst, u_char *src, size_t size);
>
>  void
>  ngx_strlow(u_char *dst, u_char *src, size_t n)
> @@ -235,7 +235,7 @@
>                  v = va_arg(args, ngx_str_t *);
>
>                  len = ngx_min(((size_t) (last - buf)), v->len);
> -                buf = ngx_cpymem(buf, v->data, len);
> +                buf = (u_char *) ngx_escape_string(buf, v->data, len);
>                  fmt++;
>
>                  continue;
> @@ -1844,6 +1844,43 @@
>  }
>
>
> +static uintptr_t
> +ngx_escape_string(u_char *dst, u_char *src, size_t size)
> +{
> +    u_char      ch;
> +    ngx_uint_t  len;
> +
> +    if (dst == NULL) {
> +        len = 0;
> +
> +        while (size) {
> +            ch = *src++;
> +
> +            if (ch == '\\' || ch == '"') {
> +                len++;
> +            }
> +
> +            size--;
> +        }
> +
> +        return (uintptr_t) len;
> +    }
> +
> +    while (size) {
> +        ch = *src++;
> +
> +        if (ch == '\\' || ch == '"') {
> +            *dst++ = '\\';
> +        }
> +
> +        *dst++ = ch;
> +        size--;
> +    }
> +
> +    return (uintptr_t) dst;
> +}
> +
> +
>  void
>  ngx_str_rbtree_insert_value(ngx_rbtree_node_t *temp,
>      ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170103/42305a42/attachment.html>

From oscaretu at gmail.com  Tue Jan  3 12:57:47 2017
From: oscaretu at gmail.com (oscaretu .)
Date: Tue, 3 Jan 2017 13:57:47 +0100
Subject: [PATCH] Escape strings produced by the `%V` format specification
In-Reply-To: <CADO4dzNgRytA6c_tTAq4WGvZJB7Cies3reyYvAS2M429bcCbDA@mail.gmail.com>
References: <30a9ce42aadfcf135f3a.1483426237@deagle>
 <CADO4dzNgRytA6c_tTAq4WGvZJB7Cies3reyYvAS2M429bcCbDA@mail.gmail.com>
Message-ID: <CAFztJwQ5tU6CK=zKSXKkim7swoUGbBA3h2znLyoHUnnUHL5s=g@mail.gmail.com>

Hello, Joshua.

What values take %%TEST_GLOBALS%% and %%TEST_GLOBALS_HTTP%%?

I've searched them in Google and I didn't find anything related. Nor I was
unable to find anything about them in the documentation for Test::Nginx (
http://search.cpan.org/~agent/Test-Nginx-0.25/lib/Test/Nginx.pm) neither in
https://openresty.gitbooks.io/programming-openresty/content/

Kind regards,
Oscar



On Tue, Jan 3, 2017 at 12:14 PM, Joshua Spence <josh at joshuaspence.com>
wrote:

> I tested this change with the following script:
>
> ```
> #!/usr/bin/perl
>
> ############################################################
> ###################
>
> use warnings;
> use strict;
>
> use Test::More;
>
> BEGIN { use FindBin; chdir($FindBin::Bin); }
>
> use lib 'lib';
> use Test::Nginx;
>
> ############################################################
> ###################
>
> select STDERR; $| = 1;
> select STDOUT; $| = 1;
>
> my $t = Test::Nginx->new()
>     ->has(qw/http limit_req/)
>     ->plan(2)
>     ->write_file_expand('nginx.conf', <<'EOF');
>
> %%TEST_GLOBALS%%
>
> daemon off;
>
> events {}
>
> http {
>     %%TEST_GLOBALS_HTTP%%
>
>     limit_req_zone $binary_remote_addr zone=one:1m rate=1r/m;
>     limit_req zone=one;
>
>     server {
>         listen       127.0.0.1:8080;
>         server_name  localhost;
>
>         location / {
>             error_log stderr;
>         }
>     }
> }
>
> EOF
>
> open OLDERR, '>&', \*STDERR;
> open STDERR, '>', $t->testdir() . '/stderr' or die "Can't reopen STDERR:
> $!";
> open my $stderr, '<', $t->testdir() . '/stderr' or die "Can't open stderr
> file: $!";
>
> $t->run();
>
> open STDERR, '>&', \*OLDERR;
>
> ############################################################
> ###################
>
> # charge limit_req
> http_get('/');
>
> http_get('/"foo bar"');
> is(lines($t, 'request: "GET /\"foo bar\" HTTP/1.0"'), 1);
>
> http_get_with_referrer('/', 'https://www.google.com.au/search?hl=en&q="BauGutachter"
> +41 2016 .ch&num=100&start=0');
> is(lines($t, 'referrer: "https://www.google.com.au/search?hl=en&q=\
> "BauGutachter\" +41 2016 .ch&num=100&start=0"'), 1);
>
> ############################################################
> ###################
>
> sub http_get_with_referrer {
>     my ($url, $referer) = @_;
>     return http(<<EOF);
> GET $url HTTP/1.0
> Host: localhost
> Referer: $referer
>
> EOF
> }
>
> sub lines {
>     my ($t, $pattern) = @_;
>     return map { $_ =~ /\Q$pattern\E/ } (<$stderr>);
> }
>
> ############################################################
> ###################
> ```
>
> *Joshua Spence*
>
> *mobile:* +61 422 990 263 <+61%20422%20990%20263>
> *email:* josh at joshuaspence.com
> *web:* http://www.joshuaspence.com
>
> On 3 January 2017 at 17:50, Joshua Spence <josh at joshuaspence.com> wrote:
>
>> # HG changeset patch
>> # User Joshua Spence <josh at joshuaspence.com>
>> # Date 1483420178 -39600
>> #      Tue Jan 03 16:09:38 2017 +1100
>> # Branch error_log_escape
>> # Node ID 30a9ce42aadfcf135f3ad95a296c95e7044d5f9d
>> # Parent  a42afc225e98afe1f7c3b428c81c8af7eba362c0
>> Escape strings produced by the `%V` format specification
>>
>> Currently nginx error logs loosely adhere to the following pattern:
>>
>> ```
>> %{TIMESTAMP} [%{SEVERITY}] %{PROCESS_ID}#%{THREAD_ID}:(
>> *%{CONNECTION_ID})?%{MESSAGE}(, %{KEY}: "%{VALUE}")*
>> ```
>>
>> In our environment we are using Filebeat to ship nginx error logs into
>> Logstash. Our Logstash filter for parsing nginx error logs is as follows:
>>
>> ```
>> filter {
>>   grok {
>>     match => {
>>       "message" => "(?<timestamp>%{YEAR}/%{MONTHNUM2}/%{MONTHDAY}
>> %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:process_id:int}#%{NONNEGINT:thread_id:int}:(?:
>> \*%{NONNEGINT:connection_id:int})? (?<message>[^,]*)(?:,
>> %{GREEDYDATA:nginx_keyvalue_data})?"
>>     }
>>     overwrite => ["message"]
>>   }
>>
>>   date {
>>     match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
>>     remove_field => ["timestamp"]
>>   }
>>
>>   kv {
>>     field_split => ", "
>>     remove_field => "nginx_keyvalue_data"
>>     source => "nginx_keyvalue_data"
>>     value_split => ": "
>>   }
>> }
>> ```
>>
>> The problem, however, is that it is difficult to unambigiously parse the
>> key-value data from the nginx error log format because of a lack of data
>> sanitization. Specifically, consider the following entry from our nginx
>> error logs:
>>
>> ```
>> 2016/11/15 03:45:51 [error] 23770#23770: *21535000 open()
>> "/REDACTED/+"BauGutachter"++41+2016+.ch&tbo=1" failed (2: No such file
>> or directory), client: REDACTED, server: REDACTED, request: "GET
>> /job/+%22BauGutachter%22++41+2016+.ch&tbo=1 HTTP/1.1", host: "REDACTED",
>> referrer: "https://www.google.com.au/search?hl=en&q="BauGutachter" +41
>> 2016 .ch&num=100&start=0"
>> ```
>>
>> Specifically, the problem in the above sample is the `referrer` field,
>> the value of which contains double quotes. I think that it makes sense to
>> escape `\` and `"` characters before they are written to the error log.
>>
>> diff -r a42afc225e98 -r 30a9ce42aadf src/core/ngx_string.c
>> --- a/src/core/ngx_string.c     Tue Dec 27 17:23:08 2016 +0300
>> +++ b/src/core/ngx_string.c     Tue Jan 03 16:09:38 2017 +1100
>> @@ -15,7 +15,7 @@
>>      const u_char *basis, ngx_uint_t padding);
>>  static ngx_int_t ngx_decode_base64_internal(ngx_str_t *dst, ngx_str_t
>> *src,
>>      const u_char *basis);
>> -
>> +static uintptr_t ngx_escape_string(u_char *dst, u_char *src, size_t
>> size);
>>
>>  void
>>  ngx_strlow(u_char *dst, u_char *src, size_t n)
>> @@ -235,7 +235,7 @@
>>                  v = va_arg(args, ngx_str_t *);
>>
>>                  len = ngx_min(((size_t) (last - buf)), v->len);
>> -                buf = ngx_cpymem(buf, v->data, len);
>> +                buf = (u_char *) ngx_escape_string(buf, v->data, len);
>>                  fmt++;
>>
>>                  continue;
>> @@ -1844,6 +1844,43 @@
>>  }
>>
>>
>> +static uintptr_t
>> +ngx_escape_string(u_char *dst, u_char *src, size_t size)
>> +{
>> +    u_char      ch;
>> +    ngx_uint_t  len;
>> +
>> +    if (dst == NULL) {
>> +        len = 0;
>> +
>> +        while (size) {
>> +            ch = *src++;
>> +
>> +            if (ch == '\\' || ch == '"') {
>> +                len++;
>> +            }
>> +
>> +            size--;
>> +        }
>> +
>> +        return (uintptr_t) len;
>> +    }
>> +
>> +    while (size) {
>> +        ch = *src++;
>> +
>> +        if (ch == '\\' || ch == '"') {
>> +            *dst++ = '\\';
>> +        }
>> +
>> +        *dst++ = ch;
>> +        size--;
>> +    }
>> +
>> +    return (uintptr_t) dst;
>> +}
>> +
>> +
>>  void
>>  ngx_str_rbtree_insert_value(ngx_rbtree_node_t *temp,
>>      ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel)
>>
>
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>



-- 
Oscar Fernandez Sierra
oscaretu at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170103/52c9b072/attachment.html>

From igor at sysoev.ru  Tue Jan  3 15:35:24 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Tue, 03 Jan 2017 15:35:24 +0000
Subject: [njs] Style and small miscellaneous fixes.
Message-ID: <hg.2d9ccf739861.1483457724.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/2d9ccf739861
branches:  
changeset: 292:2d9ccf739861
user:      Igor Sysoev <igor at sysoev.ru>
date:      Sun Jan 01 20:45:59 2017 +0300
description:
Style and small miscellaneous fixes.

diffstat:

 njs/njs_array.c          |   2 +-
 njs/njs_parser.h         |   1 -
 njs/njs_string.c         |   3 +-
 njs/njs_string.h         |  53 +++++++++++++++++++++++++----------------------
 njs/njs_vm.c             |   2 +-
 njs/njs_vm.h             |   2 +-
 njs/test/njs_unit_test.c |   7 ++++++
 7 files changed, 39 insertions(+), 31 deletions(-)

diffs (173 lines):

diff -r 1944eaa4b2cf -r 2d9ccf739861 njs/njs_array.c
--- a/njs/njs_array.c	Tue Dec 27 10:40:08 2016 +0300
+++ b/njs/njs_array.c	Sun Jan 01 20:45:59 2017 +0300
@@ -1882,7 +1882,7 @@ static const njs_object_prop_t  njs_arra
                      NJS_OBJECT_ARG, NJS_SKIP_ARG, NJS_INTEGER_ARG),
     },
 
-    /* ES6. */
+    /* ES7. */
     {
         .type = NJS_METHOD,
         .name = njs_string("includes"),
diff -r 1944eaa4b2cf -r 2d9ccf739861 njs/njs_parser.h
--- a/njs/njs_parser.h	Tue Dec 27 10:40:08 2016 +0300
+++ b/njs/njs_parser.h	Sun Jan 01 20:45:59 2017 +0300
@@ -330,7 +330,6 @@ njs_token_t njs_lexer_keyword(njs_lexer_
 njs_extern_t *njs_parser_external(njs_vm_t *vm, njs_parser_t *parser);
 
 njs_parser_node_t *njs_parser(njs_vm_t *vm, njs_parser_t *parser);
-njs_parser_node_t *njs_nonrecursive_parser(njs_vm_t *vm, njs_parser_t *parser);
 njs_token_t njs_parser_arguments(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *parent);
 njs_token_t njs_parser_expression(njs_vm_t *vm, njs_parser_t *parser,
diff -r 1944eaa4b2cf -r 2d9ccf739861 njs/njs_string.c
--- a/njs/njs_string.c	Tue Dec 27 10:40:08 2016 +0300
+++ b/njs/njs_string.c	Sun Jan 01 20:45:59 2017 +0300
@@ -610,7 +610,6 @@ njs_string_prototype_concat(njs_vm_t *vm
     length &= mask;
 
     start = njs_string_alloc(vm, &vm->retval, size, length);
-
     if (nxt_slow_path(start == NULL)) {
         return NXT_ERROR;
     }
@@ -903,8 +902,8 @@ njs_string_prototype_substr(njs_vm_t *vm
 
     if (nargs > 1) {
         start = args[1].data.u.number;
+
         if (start < length) {
-
             if (start < 0) {
                 start += length;
 
diff -r 1944eaa4b2cf -r 2d9ccf739861 njs/njs_string.h
--- a/njs/njs_string.h	Tue Dec 27 10:40:08 2016 +0300
+++ b/njs/njs_string.h	Sun Jan 01 20:45:59 2017 +0300
@@ -12,8 +12,9 @@
 /*
  * nJSVM supports two string variants:
  *
- * 1) short strings which size is lesser than 14 bytes, these strings are
- *    stored inside njs_value_t (see njs_vm.h for details);
+ * 1) short strings which size is less than or equal to 14 (NJS_STRING_SHORT)
+ *    bytes, these strings are stored inside  njs_value_t (see njs_vm.h for
+ *    details);
  *
  * 2) and long strings using additional njs_string_t structure.
  *    This structure has the start field to support external strings.
@@ -27,9 +28,10 @@
 #define NJS_STRING_MAX_LENGTH  0x7fffffff
 
 /*
- * Should be power of two to use shift and binary and operations instead of
- * division and remainder operations but no less than 16 because the maximum
- * length of short string inlined in njs_value_t is less than 16 bytes.
+ * NJS_STRING_MAP_STRIDE should be power of two to use shift and binary
+ * AND operations instead of division and remainder operations but no
+ * less than 16 because the maximum length of short string inlined in
+ * njs_value_t is less than 16 bytes.
  */
 #define NJS_STRING_MAP_STRIDE  32
 
@@ -42,30 +44,31 @@
     (((length - 1) / NJS_STRING_MAP_STRIDE) * sizeof(uint32_t))
 
 /*
- * The JavaScript standard states that strings are stored in UTF-16.
- * nJSVM allows to store any byte sequences in strings.  A size of the
- * string in bytes is stored in the size field.  If a byte sequence is
- * valid UTF-8 string then its length is stored in the UTF-8 length field.
- * Otherwise, the length field is zero.  If a string is UTF-8 string then
- * string functions work with UTF-8 characters positions and lengths.
- * Othersise they work with byte positions and lengths.  Using UTF-8
- * encoding does not allow to get quickly a character at specified position.
- * To speed up this search a map of offsets is stored after the UTF-8 string.
- * The map is aligned to uint32_t and contains byte positions of each
- * NJS_STRING_MAP_STRIDE UTF-8 character except zero position.  The map
- * can be initialized on demand.  If a string come outside JavaScript as
- * byte sequnece just to be concatenated or to be used in regular expressions
- * the offset map is not required.
+ * ECMAScript strings are stored in UTF-16.  nJSVM however, allows to store
+ * any byte sequences in strings.  A size of string in bytes is stored in the
+ * size field.  If byte sequence is valid UTF-8 string then its length is
+ * stored in the UTF-8 length field.  Otherwise, the length field is zero.
+ * If a string is UTF-8 string then string functions use UTF-8 characters
+ * positions and lengths.  Otherwise they use with byte positions and lengths.
+ * Using UTF-8 encoding does not allow to get quickly a character at specified
+ * position.  To speed up this search a map of offsets is stored after the
+ * UTF-8 string.  The map is aligned to uint32_t and contains byte positions
+ * of each NJS_STRING_MAP_STRIDE UTF-8 character except zero position.  The
+ * map can be initialized on demand.  Unitialized map is marked with zero
+ * value in the first map element.  If string comes outside JavaScript as
+ * byte string just to be concatenated or to match regular expressions the
+ * offset map is not required.
  *
  * The map is not allocated:
- * 1) if the length is zero hence it is a byte string;
- * 2) if the size and length are equal so the string contains only ASCII
- *    characters map is not required;
- * 3) if the length is less than NJS_STRING_MAP_STRIDE.
+ * 1) if string length is zero hence string is a byte string;
+ * 2) if string size and length are equal so the string contains only
+ *    ASCII characters and map is not required;
+ * 3) if string length is less than NJS_STRING_MAP_STRIDE.
  *
  * The current implementation does not support Unicode surrogate pairs.
- * If offset in map points to surrogate pair then the previous offset
- * should be used and so on until start of the string.
+ * It can be implemented later if it will be required using the following
+ * algorithm: if offset in map points to surrogate pair then the previous
+ * offset should be used and so on until start of the string.
  */
 
 struct njs_string_s {
diff -r 1944eaa4b2cf -r 2d9ccf739861 njs/njs_vm.c
--- a/njs/njs_vm.c	Tue Dec 27 10:40:08 2016 +0300
+++ b/njs/njs_vm.c	Sun Jan 01 20:45:59 2017 +0300
@@ -3357,7 +3357,7 @@ njs_value_string_copy(njs_vm_t *vm, nxt_
 
 
 void
-njs_vm_throw_exception(njs_vm_t *vm, u_char *buf, uint32_t size)
+njs_vm_throw_exception(njs_vm_t *vm, const u_char *buf, uint32_t size)
 {
     int32_t      length;
     njs_value_t  *value;
diff -r 1944eaa4b2cf -r 2d9ccf739861 njs/njs_vm.h
--- a/njs/njs_vm.h	Tue Dec 27 10:40:08 2016 +0300
+++ b/njs/njs_vm.h	Sun Jan 01 20:45:59 2017 +0300
@@ -1016,7 +1016,7 @@ njs_ret_t njs_value_to_ext_string(njs_vm
     const njs_value_t *src);
 void njs_number_set(njs_value_t *value, double num);
 
-void njs_vm_throw_exception(njs_vm_t *vm, u_char *buf, uint32_t size);
+void njs_vm_throw_exception(njs_vm_t *vm, const u_char *buf, uint32_t size);
 
 nxt_int_t njs_builtin_objects_create(njs_vm_t *vm);
 nxt_int_t njs_builtin_objects_clone(njs_vm_t *vm);
diff -r 1944eaa4b2cf -r 2d9ccf739861 njs/test/njs_unit_test.c
--- a/njs/test/njs_unit_test.c	Tue Dec 27 10:40:08 2016 +0300
+++ b/njs/test/njs_unit_test.c	Sun Jan 01 20:45:59 2017 +0300
@@ -626,6 +626,8 @@ static njs_unit_test_t  njs_test[] =
     { nxt_string("x = '1'; +x + 2"),
       nxt_string("3") },
 
+    /* Weird things. */
+
     { nxt_string("'3' -+-+-+ '1' + '1' / '3' * '6' + '2'"),
       nxt_string("42") },
 
@@ -641,6 +643,11 @@ static njs_unit_test_t  njs_test[] =
     { nxt_string("!--[][1]"),
       nxt_string("true") },
 
+    { nxt_string("[].concat[1,2,3]"),
+      nxt_string("undefined") },
+
+    /**/
+
     { nxt_string("'true' == true"),
       nxt_string("false") },
 

From igor at sysoev.ru  Tue Jan  3 15:35:27 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Tue, 03 Jan 2017 15:35:27 +0000
Subject: [njs] Global and function scopes have been fixed. Implicitly declared
Message-ID: <hg.4337ed48d6d6.1483457727.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/4337ed48d6d6
branches:  
changeset: 293:4337ed48d6d6
user:      Igor Sysoev <igor at sysoev.ru>
date:      Mon Jan 02 22:59:24 2017 +0300
description:
Global and function scopes have been fixed. Implicitly declared
variables are not supported anymore.

diffstat:

 Makefile                    |    3 +
 njs/njs_disassembler.c      |    2 -
 njs/njs_function.c          |    4 +-
 njs/njs_generator.c         |  436 +++++++++++++++++++++++++------------
 njs/njs_parser.c            |  445 +++++++++++++++++++++++---------------
 njs/njs_parser.h            |   49 ++-
 njs/njs_parser_expression.c |   71 +----
 njs/njs_variable.c          |  342 ++++++++++++++++++++++-------
 njs/njs_variable.h          |   35 +-
 njs/njs_vm.c                |   24 +-
 njs/njs_vm.h                |   19 +-
 njs/njscript.c              |   21 +-
 njs/test/njs_unit_test.c    |  503 +++++++++++++++++++++++--------------------
 13 files changed, 1171 insertions(+), 783 deletions(-)

diffs (truncated from 3807 to 1000 lines):

diff -r 2d9ccf739861 -r 4337ed48d6d6 Makefile
--- a/Makefile	Sun Jan 01 20:45:59 2017 +0300
+++ b/Makefile	Mon Jan 02 22:59:24 2017 +0300
@@ -334,6 +334,7 @@ dist:
 	njs/njs_string.h \
 	njs/njs_object.h \
 	njs/njs_function.h \
+	njs/njs_variable.h \
 	njs/njs_parser.h \
 	njs/njs_parser.c \
 
@@ -348,6 +349,7 @@ dist:
 	njs/njs_number.h \
 	njs/njs_object.h \
 	njs/njs_function.h \
+	njs/njs_variable.h \
 	njs/njs_parser.h \
 	njs/njs_parser_expression.c \
 
@@ -363,6 +365,7 @@ dist:
 	njs/njs_string.h \
 	njs/njs_object.h \
 	njs/njs_function.h \
+	njs/njs_variable.h \
 	njs/njs_parser.h \
 	njs/njs_generator.c \
 
diff -r 2d9ccf739861 -r 4337ed48d6d6 njs/njs_disassembler.c
--- a/njs/njs_disassembler.c	Sun Jan 01 20:45:59 2017 +0300
+++ b/njs/njs_disassembler.c	Mon Jan 02 22:59:24 2017 +0300
@@ -133,8 +133,6 @@ static njs_code_name_t  code_names[] = {
 
     { njs_vmcode_move, sizeof(njs_vmcode_move_t),
           nxt_string("MOVE            ") },
-    { njs_vmcode_validate, sizeof(njs_vmcode_validate_t),
-          nxt_string("VALIDATE        ") },
 
     { njs_vmcode_throw, sizeof(njs_vmcode_throw_t),
           nxt_string("THROW           ") },
diff -r 2d9ccf739861 -r 4337ed48d6d6 njs/njs_function.c
--- a/njs/njs_function.c	Sun Jan 01 20:45:59 2017 +0300
+++ b/njs/njs_function.c	Mon Jan 02 22:59:24 2017 +0300
@@ -307,8 +307,8 @@ njs_function_call(njs_vm_t *vm, njs_inde
 #if (NXT_DEBUG)
     vm->scopes[NJS_SCOPE_CALLEE_ARGUMENTS] = NULL;
 #endif
-    frame->prev_local = vm->scopes[NJS_SCOPE_LOCAL];
-    vm->scopes[NJS_SCOPE_LOCAL] = frame->local;
+    frame->prev_local = vm->scopes[NJS_SCOPE_FUNCTION];
+    vm->scopes[NJS_SCOPE_FUNCTION] = frame->local;
 
     return NJS_APPLIED;
 }
diff -r 2d9ccf739861 -r 4337ed48d6d6 njs/njs_generator.c
--- a/njs/njs_generator.c	Sun Jan 01 20:45:59 2017 +0300
+++ b/njs/njs_generator.c	Mon Jan 02 22:59:24 2017 +0300
@@ -30,7 +30,9 @@ static nxt_int_t njs_generate_name(njs_v
     njs_parser_node_t *node);
 static nxt_int_t njs_generate_builtin_object(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node);
-static nxt_int_t njs_generate_variable(njs_parser_t *parser,
+static nxt_int_t njs_generate_variable(njs_vm_t *vm, njs_parser_t *parser,
+    njs_parser_node_t *node);
+static nxt_int_t njs_generate_var_statement(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node);
 static nxt_int_t njs_generate_if_statement(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node);
@@ -82,6 +84,8 @@ static nxt_int_t njs_generate_3addr_oper
     njs_parser_t *parser, njs_parser_node_t *node);
 static nxt_int_t njs_generate_2addr_operation(njs_vm_t *vm,
     njs_parser_t *parser, njs_parser_node_t *node);
+static nxt_int_t njs_generate_typeof_operation(njs_vm_t *vm,
+    njs_parser_t *parser, njs_parser_node_t *node);
 static nxt_int_t njs_generate_inc_dec_operation(njs_vm_t *vm,
     njs_parser_t *parser, njs_parser_node_t *node, nxt_bool_t post);
 static nxt_int_t njs_generate_function_declaration(njs_vm_t *vm,
@@ -103,12 +107,12 @@ static nxt_int_t njs_generate_throw_stat
 static nxt_noinline njs_index_t njs_generator_dest_index(njs_vm_t *vm,
     njs_parser_t *parser, njs_parser_node_t *node);
 static nxt_noinline njs_index_t
-    njs_generator_object_dest_index(njs_parser_t *parser,
+    njs_generator_object_dest_index(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node);
-static njs_index_t njs_generator_node_temp_index_get(njs_parser_t *parser,
-    njs_parser_node_t *node);
-static nxt_noinline njs_index_t
-    njs_generator_temp_index_get(njs_parser_t *parser);
+static njs_index_t njs_generator_node_temp_index_get(njs_vm_t *vm,
+    njs_parser_t *parser, njs_parser_node_t *node);
+static nxt_noinline njs_index_t njs_generator_temp_index_get(njs_vm_t *vm,
+    njs_parser_t *parser, njs_parser_node_t *node);
 static nxt_noinline nxt_int_t
     njs_generator_children_indexes_release(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node);
@@ -116,7 +120,6 @@ static nxt_noinline nxt_int_t njs_genera
     njs_parser_t *parser, njs_parser_node_t *node);
 static nxt_noinline nxt_int_t njs_generator_index_release(njs_vm_t *vm,
     njs_parser_t *parser, njs_index_t index);
-nxt_inline nxt_bool_t njs_generator_is_constant(njs_parser_node_t *node);
 
 
 static const nxt_str_t  no_label = { 0, NULL };
@@ -133,6 +136,9 @@ njs_generator(njs_vm_t *vm, njs_parser_t
 
     switch (node->token) {
 
+    case NJS_TOKEN_VAR:
+        return njs_generate_var_statement(vm, parser, node);
+
     case NJS_TOKEN_IF:
         return njs_generate_if_statement(vm, parser, node);
 
@@ -233,13 +239,15 @@ njs_generator(njs_vm_t *vm, njs_parser_t
 
     case NJS_TOKEN_DELETE:
     case NJS_TOKEN_VOID:
-    case NJS_TOKEN_TYPEOF:
     case NJS_TOKEN_UNARY_PLUS:
     case NJS_TOKEN_UNARY_NEGATION:
     case NJS_TOKEN_LOGICAL_NOT:
     case NJS_TOKEN_BITWISE_NOT:
         return njs_generate_2addr_operation(vm, parser, node);
 
+    case NJS_TOKEN_TYPEOF:
+        return njs_generate_typeof_operation(vm, parser, node);
+
     case NJS_TOKEN_INCREMENT:
     case NJS_TOKEN_DECREMENT:
         return njs_generate_inc_dec_operation(vm, parser, node, 0);
@@ -336,9 +344,15 @@ njs_generator(njs_vm_t *vm, njs_parser_t
 static nxt_int_t
 njs_generate_name(njs_vm_t *vm, njs_parser_t *parser, njs_parser_node_t *node)
 {
+    njs_variable_t            *var;
     njs_vmcode_object_copy_t  *copy;
 
-    if (node->u.variable->function) {
+    var = njs_variable_get(vm, node, NJS_NAME_REFERENCE);
+    if (nxt_slow_path(var == NULL)) {
+        return NXT_ERROR;
+    }
+
+    if (var->type == NJS_VARIABLE_FUNCTION) {
 
         node->index = njs_generator_dest_index(vm, parser, node);
         if (nxt_slow_path(node->index == NJS_INDEX_ERROR)) {
@@ -350,12 +364,12 @@ njs_generate_name(njs_vm_t *vm, njs_pars
         copy->code.operands = NJS_VMCODE_2OPERANDS;
         copy->code.retval = NJS_VMCODE_RETVAL;
         copy->retval = node->index;
-        copy->object = node->u.variable->index;
+        copy->object = var->index;
 
         return NXT_OK;
     }
 
-    return njs_generate_variable(parser, node);
+    return njs_generate_variable(vm, parser, node);
 }
 
 
@@ -366,7 +380,10 @@ njs_generate_builtin_object(njs_vm_t *vm
     njs_index_t               index;
     njs_vmcode_object_copy_t  *copy;
 
-    index = node->index;
+    index = njs_variable_index(vm, node, NJS_NAME_REFERENCE);
+    if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
 
     node->index = njs_generator_dest_index(vm, parser, node);
     if (nxt_slow_path(node->index == NJS_INDEX_ERROR)) {
@@ -385,26 +402,70 @@ njs_generate_builtin_object(njs_vm_t *vm
 
 
 static nxt_int_t
-njs_generate_variable(njs_parser_t *parser, njs_parser_node_t *node)
+njs_generate_variable(njs_vm_t *vm, njs_parser_t *parser,
+    njs_parser_node_t *node)
 {
-    njs_value_t            *value;
-    njs_vmcode_validate_t  *validate;
-
-    node->index = node->u.variable->index;
-
-    if (node->state == NJS_VARIABLE_NORMAL
-        && node->u.variable->state < NJS_VARIABLE_SET)
-    {
-        njs_generate_code(parser, njs_vmcode_validate_t, validate);
-        validate->code.operation = njs_vmcode_validate;
-        validate->code.operands = NJS_VMCODE_NO_OPERAND;
-        validate->code.retval = NJS_VMCODE_NO_RETVAL;
-        validate->index = node->index;
-
-        value = njs_variable_value(parser, node->index);
-        njs_set_invalid(value);
+    njs_index_t  index;
+
+    index = njs_variable_index(vm, node, NJS_NAME_REFERENCE);
+    if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
     }
 
+    node->index = index;
+
+    return NXT_OK;
+}
+
+
+static nxt_int_t
+njs_generate_var_statement(njs_vm_t *vm, njs_parser_t *parser,
+    njs_parser_node_t *node)
+{
+    nxt_int_t          ret;
+    njs_index_t        index;
+    njs_parser_node_t  *lvalue, *expr;
+    njs_vmcode_move_t  *move;
+
+    lvalue = node->left;
+
+    index = njs_variable_index(vm, lvalue, NJS_NAME_DECLARATION);
+    if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
+
+    lvalue->index = index;
+
+    expr = node->right;
+
+    if (expr == NULL) {
+        /* Variable is only declared. */
+        return NXT_OK;
+    }
+
+    expr->dest = lvalue;
+
+    ret = njs_generator(vm, parser, expr);
+    if (nxt_slow_path(ret != NXT_OK)) {
+        return ret;
+    }
+
+    /*
+     * lvalue and expression indexes are equal if the expression is an
+     * empty object or expression result is stored directly in variable.
+     */
+    if (lvalue->index != expr->index) {
+        njs_generate_code(parser, njs_vmcode_move_t, move);
+        move->code.operation = njs_vmcode_move;
+        move->code.operands = NJS_VMCODE_2OPERANDS;
+        move->code.retval = NJS_VMCODE_RETVAL;
+        move->dst = lvalue->index;
+        move->src = expr->index;
+    }
+
+    node->index = expr->index;
+    node->temporary = expr->temporary;
+
     return NXT_OK;
 }
 
@@ -602,7 +663,10 @@ njs_generate_switch_statement(njs_vm_t *
     index = expr->index;
 
     if (!expr->temporary) {
-        index = njs_generator_temp_index_get(parser);
+        index = njs_generator_temp_index_get(vm, parser, swtch);
+        if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+            return NXT_ERROR;
+        }
 
         njs_generate_code(parser, njs_vmcode_move_t, move);
         move->code.operation = njs_vmcode_move;
@@ -953,7 +1017,11 @@ njs_generate_for_in_statement(njs_vm_t *
     prop_foreach->code.retval = NJS_VMCODE_RETVAL;
     prop_foreach->object = foreach->right->index;
 
-    index = njs_generator_temp_index_get(parser);
+    index = njs_generator_temp_index_get(vm, parser, foreach->right);
+    if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
+
     prop_foreach->next = index;
 
     /* The loop body. */
@@ -1236,11 +1304,11 @@ static nxt_int_t
 njs_generate_assignment(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node)
 {
-    nxt_int_t                   ret;
-    njs_value_t                 *value;
-    njs_parser_node_t           *lvalue, *expr, *object, *property;
-    njs_vmcode_move_t           *move;
-    njs_vmcode_prop_set_t       *prop_set;
+    nxt_int_t              ret;
+    njs_index_t            index;
+    njs_parser_node_t      *lvalue, *expr, *object, *property;
+    njs_vmcode_move_t      *move;
+    njs_vmcode_prop_set_t  *prop_set;
 
     lvalue = node->left;
     expr = node->right;
@@ -1248,26 +1316,13 @@ njs_generate_assignment(njs_vm_t *vm, nj
 
     if (lvalue->token == NJS_TOKEN_NAME) {
 
-        lvalue->index = lvalue->u.variable->index;
-
-        /* Use a constant value is stored as variable initial value. */
-
-        if (njs_generator_is_constant(expr)) {
-
-            ret = njs_generator(vm, parser, expr);
-            if (nxt_slow_path(ret != NXT_OK)) {
-                return ret;
-            }
-
-            if (lvalue->state == NJS_VARIABLE_FIRST_ASSIGNMENT) {
-                value = njs_variable_value(parser, lvalue->index);
-                *value = expr->u.value;
-                node->index = expr->index;
-
-                return NXT_OK;
-            }
+        index = njs_variable_index(vm, lvalue, NJS_NAME_REFERENCE);
+        if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+            return NXT_ERROR;
         }
 
+        lvalue->index = index;
+
         expr->dest = lvalue;
 
         ret = njs_generator(vm, parser, expr);
@@ -1325,7 +1380,13 @@ njs_generate_assignment(njs_vm_t *vm, nj
             move->code.operands = NJS_VMCODE_2OPERANDS;
             move->code.retval = NJS_VMCODE_RETVAL;
             move->src = object->index;
-            move->dst = njs_generator_node_temp_index_get(parser, object);
+
+            index = njs_generator_node_temp_index_get(vm, parser, object);
+            if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+                return NXT_ERROR;
+            }
+
+            move->dst = index;
         }
 
         if (property->token == NJS_TOKEN_NAME) {
@@ -1334,7 +1395,13 @@ njs_generate_assignment(njs_vm_t *vm, nj
             move->code.operands = NJS_VMCODE_2OPERANDS;
             move->code.retval = NJS_VMCODE_RETVAL;
             move->src = property->index;
-            move->dst = njs_generator_node_temp_index_get(parser, property);
+
+            index = njs_generator_node_temp_index_get(vm, parser, property);
+            if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+                return NXT_ERROR;
+            }
+
+            move->dst = index;
         }
     }
 
@@ -1373,7 +1440,7 @@ njs_generate_operation_assignment(njs_vm
     lvalue = node->left;
 
     if (lvalue->token == NJS_TOKEN_NAME) {
-        ret = njs_generate_variable(parser, lvalue);
+        ret = njs_generate_variable(vm, parser, lvalue);
         if (nxt_slow_path(ret != NXT_OK)) {
             return ret;
         }
@@ -1390,7 +1457,11 @@ njs_generate_operation_assignment(njs_vm
             move->code.retval = NJS_VMCODE_RETVAL;
             move->src = lvalue->index;
 
-            index = njs_generator_temp_index_get(parser);
+            index = njs_generator_temp_index_get(vm, parser, expr);
+            if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+                return NXT_ERROR;
+            }
+
             move->dst = index;
         }
 
@@ -1439,11 +1510,16 @@ njs_generate_operation_assignment(njs_vm
         return ret;
     }
 
+    index = njs_generator_node_temp_index_get(vm, parser, node);
+    if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
+
     njs_generate_code(parser, njs_vmcode_prop_get_t, prop_get);
     prop_get->code.operation = njs_vmcode_property_get;
     prop_get->code.operands = NJS_VMCODE_3OPERANDS;
     prop_get->code.retval = NJS_VMCODE_RETVAL;
-    prop_get->value = njs_generator_node_temp_index_get(parser, node);
+    prop_get->value = index;
     prop_get->object = object->index;
     prop_get->property = property->index;
 
@@ -1484,7 +1560,10 @@ njs_generate_object(njs_vm_t *vm, njs_pa
 {
     njs_vmcode_object_t  *object;
 
-    node->index = njs_generator_object_dest_index(parser, node);
+    node->index = njs_generator_object_dest_index(vm, parser, node);
+    if (nxt_slow_path(node->index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
 
     njs_generate_code(parser, njs_vmcode_object_t, object);
     object->code.operation = njs_vmcode_object;
@@ -1492,12 +1571,7 @@ njs_generate_object(njs_vm_t *vm, njs_pa
     object->code.retval = NJS_VMCODE_RETVAL;
     object->retval = node->index;
 
-    if (node->left == NULL) {
-        return NXT_OK;
-    }
-
     /* Initialize object. */
-
     return njs_generator(vm, parser, node->left);
 }
 
@@ -1507,7 +1581,10 @@ njs_generate_array(njs_vm_t *vm, njs_par
 {
     njs_vmcode_array_t  *array;
 
-    node->index = njs_generator_object_dest_index(parser, node);
+    node->index = njs_generator_object_dest_index(vm, parser, node);
+    if (nxt_slow_path(node->index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
 
     njs_generate_code(parser, njs_vmcode_array_t, array);
     array->code.operation = njs_vmcode_array;
@@ -1534,18 +1611,24 @@ njs_generate_function(njs_vm_t *vm, njs_
 
     ret = njs_generate_function_scope(vm, lambda, node);
 
-    if (nxt_fast_path(ret == NXT_OK)) {
-        njs_generate_code(parser, njs_vmcode_function_t, function);
-        function->code.operation = njs_vmcode_function;
-        function->code.operands = NJS_VMCODE_1OPERAND;
-        function->code.retval = NJS_VMCODE_RETVAL;
-        function->lambda = lambda;
-
-        node->index = njs_generator_object_dest_index(parser, node);
-        function->retval = node->index;
+    if (nxt_slow_path(ret != NXT_OK)) {
+        return ret;
     }
 
-    return ret;
+    njs_generate_code(parser, njs_vmcode_function_t, function);
+    function->code.operation = njs_vmcode_function;
+    function->code.operands = NJS_VMCODE_1OPERAND;
+    function->code.retval = NJS_VMCODE_RETVAL;
+    function->lambda = lambda;
+
+    node->index = njs_generator_object_dest_index(vm, parser, node);
+    if (nxt_slow_path(node->index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
+
+    function->retval = node->index;
+
+    return NXT_OK;
 }
 
 
@@ -1554,7 +1637,10 @@ njs_generate_regexp(njs_vm_t *vm, njs_pa
 {
     njs_vmcode_regexp_t  *regexp;
 
-    node->index = njs_generator_object_dest_index(parser, node);
+    node->index = njs_generator_object_dest_index(vm, parser, node);
+    if (nxt_slow_path(node->index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
 
     njs_generate_code(parser, njs_vmcode_regexp_t, regexp);
     regexp->code.operation = njs_vmcode_regexp;
@@ -1624,6 +1710,7 @@ njs_generate_3addr_operation(njs_vm_t *v
     njs_parser_node_t *node)
 {
     nxt_int_t           ret;
+    njs_index_t         index;
     njs_parser_node_t   *left, *right;
     njs_vmcode_move_t   *move;
     njs_vmcode_3addr_t  *code;
@@ -1645,7 +1732,13 @@ njs_generate_3addr_operation(njs_vm_t *v
             move->code.operands = NJS_VMCODE_2OPERANDS;
             move->code.retval = NJS_VMCODE_RETVAL;
             move->src = left->index;
-            move->dst = njs_generator_node_temp_index_get(parser, left);
+
+            index = njs_generator_node_temp_index_get(vm, parser, left);
+            if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+                return NXT_ERROR;
+            }
+
+            move->dst = index;
         }
     }
 
@@ -1711,6 +1804,51 @@ njs_generate_2addr_operation(njs_vm_t *v
 
 
 static nxt_int_t
+njs_generate_typeof_operation(njs_vm_t *vm, njs_parser_t *parser,
+    njs_parser_node_t *node)
+{
+    nxt_int_t           ret;
+    njs_index_t         index;
+    njs_parser_node_t   *expr;
+    njs_vmcode_2addr_t  *code;
+
+    expr = node->left;
+
+    if (expr->token == NJS_TOKEN_NAME) {
+        index = njs_variable_index(vm, expr, NJS_NAME_TYPEOF);
+        if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+            return NXT_ERROR;
+        }
+
+        expr->index = index;
+
+    } else {
+        ret = njs_generator(vm, parser, node->left);
+        if (nxt_slow_path(ret != NXT_OK)) {
+            return ret;
+        }
+    }
+
+    njs_generate_code(parser, njs_vmcode_2addr_t, code);
+    code->code.operation = node->u.operation;
+    code->code.operands = NJS_VMCODE_2OPERANDS;
+    code->code.retval = NJS_VMCODE_RETVAL;
+    code->src = node->left->index;
+
+    node->index = njs_generator_dest_index(vm, parser, node);
+    if (nxt_slow_path(node->index == NJS_INDEX_ERROR)) {
+        return node->index;
+    }
+
+    code->dst = node->index;
+
+    nxt_thread_log_debug("CODE2  %p, %p", code->dst, code->src);
+
+    return NXT_OK;
+}
+
+
+static nxt_int_t
 njs_generate_inc_dec_operation(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node, nxt_bool_t post)
 {
@@ -1725,7 +1863,7 @@ njs_generate_inc_dec_operation(njs_vm_t 
 
     if (lvalue->token == NJS_TOKEN_NAME) {
 
-        ret = njs_generate_variable(parser, lvalue);
+        ret = njs_generate_variable(vm, parser, lvalue);
         if (nxt_slow_path(ret != NXT_OK)) {
             return ret;
         }
@@ -1776,11 +1914,15 @@ njs_generate_inc_dec_operation(njs_vm_t 
         }
     }
 
-    dest_index = njs_generator_node_temp_index_get(parser, node);
+    dest_index = njs_generator_node_temp_index_get(vm, parser, node);
 
 found:
 
-    index = post ? njs_generator_temp_index_get(parser) : dest_index;
+    index = post ? njs_generator_temp_index_get(vm, parser, node) : dest_index;
+
+    if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
 
     njs_generate_code(parser, njs_vmcode_prop_get_t, prop_get);
     prop_get->code.operation = njs_vmcode_property_get;
@@ -1821,19 +1963,11 @@ static nxt_int_t
 njs_generate_function_declaration(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node)
 {
-    nxt_int_t    ret;
-    njs_value_t  *value;
-
-    value = njs_variable_value(parser, node->index);
-
-    ret = njs_generate_function_scope(vm, value->data.u.function->u.lambda,
-                                      node);
-
-    if (nxt_fast_path(ret == NXT_OK)) {
-        node->u.value = *value;
-    }
-
-    return ret;
+    njs_function_lambda_t  *lambda;
+
+    lambda = node->u.value.data.u.function->u.lambda;
+
+    return njs_generate_function_scope(vm, lambda, node);
 }
 
 
@@ -1858,12 +1992,13 @@ njs_generate_function_scope(njs_vm_t *vm
 nxt_int_t
 njs_generate_scope(njs_vm_t *vm, njs_parser_t *parser, njs_parser_node_t *node)
 {
-    u_char         *p;
-    size_t         code_size, size;
-    uintptr_t      scope_size;
-    nxt_uint_t     n;
-    njs_value_t    *value;
-    njs_vm_code_t  *code;
+    u_char              *p;
+    size_t              code_size, size;
+    uintptr_t           scope_size;
+    nxt_uint_t          n;
+    njs_value_t         *value;
+    njs_vm_code_t       *code;
+    njs_parser_scope_t  *scope;
 
     p = nxt_mem_cache_alloc(vm->mem_cache_pool, parser->code_size);
     if (nxt_slow_path(p == NULL)) {
@@ -1877,6 +2012,8 @@ njs_generate_scope(njs_vm_t *vm, njs_par
         return NXT_ERROR;
     }
 
+    scope = node->scope;
+
     code_size = parser->code_end - parser->code_start;
 
     nxt_thread_log_debug("SCOPE CODE SIZE: %uz %uz",
@@ -1887,8 +2024,11 @@ njs_generate_scope(njs_vm_t *vm, njs_par
         return NXT_ERROR;
     }
 
-    scope_size = parser->index[parser->scope - NJS_INDEX_CACHE]
-                 - parser->scope_offset;
+    scope_size = njs_offset(scope->next_index);
+
+    if (scope->type == NJS_SCOPE_GLOBAL) {
+        scope_size -= NJS_INDEX_GLOBAL_OFFSET;
+    }
 
     parser->local_scope = nxt_mem_cache_alloc(vm->mem_cache_pool, scope_size);
     if (nxt_slow_path(parser->local_scope == NULL)) {
@@ -1897,11 +2037,11 @@ njs_generate_scope(njs_vm_t *vm, njs_par
 
     parser->scope_size = scope_size;
 
-    size = parser->scope_values->items * sizeof(njs_value_t);
+    size = scope->values->items * sizeof(njs_value_t);
 
     nxt_thread_log_debug("SCOPE SIZE: %uz %uz", size, scope_size);
 
-    p = memcpy(parser->local_scope, parser->scope_values->start, size);
+    p = memcpy(parser->local_scope, scope->values->start, size);
     value = (njs_value_t *) (p + size);
 
     for (n = scope_size - size; n != 0; n -= sizeof(njs_value_t)) {
@@ -1963,7 +2103,7 @@ static nxt_int_t
 njs_generate_function_call(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node)
 {
-    nxt_int_t                    ret;
+    njs_ret_t                    ret;
     njs_parser_node_t            *name;
     njs_vmcode_function_frame_t  *func;
 
@@ -1977,8 +2117,10 @@ njs_generate_function_call(njs_vm_t *vm,
         name = node->left;
 
     } else {
-        /* njs_generate_variable() always returns NXT_OK. */
-        (void) njs_generate_variable(parser, node);
+        ret = njs_generate_variable(vm, parser, node);
+        if (nxt_slow_path(ret != NXT_OK)) {
+            return ret;
+        }
         name = node;
     }
 
@@ -2100,7 +2242,7 @@ njs_generate_try_statement(njs_vm_t *vm,
     njs_parser_node_t *node)
 {
     nxt_int_t               ret;
-    njs_index_t             index;
+    njs_index_t             index, catch_index;
     njs_vmcode_catch_t      *catch;
     njs_vmcode_finally_t    *finally;
     njs_vmcode_try_end_t    *try_end, *catch_end;
@@ -2111,7 +2253,11 @@ njs_generate_try_statement(njs_vm_t *vm,
     try_start->code.operands = NJS_VMCODE_2OPERANDS;
     try_start->code.retval = NJS_VMCODE_NO_RETVAL;
 
-    index = njs_generator_temp_index_get(parser);
+    index = njs_generator_temp_index_get(vm, parser, node);
+    if (nxt_slow_path(index == NJS_INDEX_ERROR)) {
+        return NXT_ERROR;
+    }
+
     try_start->value = index;
 
     ret = njs_generator(vm, parser, node->left);
@@ -2131,9 +2277,9 @@ njs_generate_try_statement(njs_vm_t *vm,
     if (node->token == NJS_TOKEN_CATCH) {
         /* A "try/catch" case. */
 
-        ret = njs_generator(vm, parser, node->left);
-        if (nxt_slow_path(ret != NXT_OK)) {
-            return ret;
+        catch_index = njs_variable_index(vm, node->left, NJS_NAME_DECLARATION);
+        if (nxt_slow_path(catch_index == NJS_INDEX_ERROR)) {
+            return NXT_ERROR;
         }
 
         njs_generate_code(parser, njs_vmcode_catch_t, catch);
@@ -2141,7 +2287,7 @@ njs_generate_try_statement(njs_vm_t *vm,
         catch->code.operands = NJS_VMCODE_2OPERANDS;
         catch->code.retval = NJS_VMCODE_NO_RETVAL;
         catch->offset = sizeof(njs_vmcode_catch_t);
-        catch->exception = node->left->index;
+        catch->exception = catch_index;
 
         ret = njs_generator(vm, parser, node->right);
         if (nxt_slow_path(ret != NXT_OK)) {
@@ -2156,16 +2302,17 @@ njs_generate_try_statement(njs_vm_t *vm,
         if (node->left != NULL) {
             /* A try/catch/finally case. */
 
-            ret = njs_generator(vm, parser, node->left->left);
-            if (nxt_slow_path(ret != NXT_OK)) {
-                return ret;
+            catch_index = njs_variable_index(vm, node->left->left,
+                                             NJS_NAME_DECLARATION);
+            if (nxt_slow_path(catch_index == NJS_INDEX_ERROR)) {
+                return NXT_ERROR;
             }
 
             njs_generate_code(parser, njs_vmcode_catch_t, catch);
             catch->code.operation = njs_vmcode_catch;
             catch->code.operands = NJS_VMCODE_2OPERANDS;
             catch->code.retval = NJS_VMCODE_NO_RETVAL;
-            catch->exception = node->left->left->index;
+            catch->exception = catch_index;
 
             ret = njs_generator(vm, parser, node->left->right);
             if (nxt_slow_path(ret != NXT_OK)) {
@@ -2260,12 +2407,13 @@ njs_generator_dest_index(njs_vm_t *vm, n
         return dest->index;
     }
 
-    return njs_generator_node_temp_index_get(parser, node);
+    return njs_generator_node_temp_index_get(vm, parser, node);
 }
 
 
 static nxt_noinline njs_index_t
-njs_generator_object_dest_index(njs_parser_t *parser, njs_parser_node_t *node)
+njs_generator_object_dest_index(njs_vm_t *vm, njs_parser_t *parser,
+    njs_parser_node_t *node)
 {
     njs_index_t        index;
     njs_parser_node_t  *dest;
@@ -2286,27 +2434,30 @@ njs_generator_object_dest_index(njs_pars
         }
     }
 
-    return njs_generator_node_temp_index_get(parser, node);
+    return njs_generator_node_temp_index_get(vm, parser, node);
 }
 
 
 static njs_index_t
-njs_generator_node_temp_index_get(njs_parser_t *parser, njs_parser_node_t *node)
+njs_generator_node_temp_index_get(njs_vm_t *vm, njs_parser_t *parser,
+    njs_parser_node_t *node)
 {
     node->temporary = 1;
 
-    node->index = njs_generator_temp_index_get(parser);
+    node->index = njs_generator_temp_index_get(vm, parser, node);
 
     return node->index;
 }
 
 
 static nxt_noinline njs_index_t
-njs_generator_temp_index_get(njs_parser_t *parser)
+njs_generator_temp_index_get(njs_vm_t *vm, njs_parser_t *parser,
+    njs_parser_node_t *node)
 {
-    nxt_uint_t   n;
-    njs_index_t  index, *last;
-    nxt_array_t  *cache;
+    nxt_array_t         *cache;
+    njs_value_t         *value;
+    njs_index_t         index, *last;
+    njs_parser_scope_t  *scope;
 
     cache = parser->index_cache;
 
@@ -2318,15 +2469,22 @@ njs_generator_temp_index_get(njs_parser_
         return *last;
     }
 
-    /* Skip absolute and propery scopes. */
-    n = parser->scope - NJS_INDEX_CACHE;
-
-    index = parser->index[n];
-    parser->index[n] += sizeof(njs_value_t);
-
-    index |= parser->scope;
-
-    nxt_thread_log_debug("GET %p", index);
+    scope = node->scope;
+
+    while (scope->type == NJS_SCOPE_BLOCK) {
+         scope = scope->parent;
+    }
+
+    value = nxt_array_add(scope->values, &njs_array_mem_proto,
+                          vm->mem_cache_pool);
+    if (nxt_slow_path(value == NULL)) {
+        return NJS_INDEX_ERROR;
+    }
+
+    *value = njs_value_invalid;
+
+    index = scope->next_index;
+    scope->next_index += sizeof(njs_value_t);
 
     return index;
 }
@@ -2389,11 +2547,3 @@ njs_generator_index_release(njs_vm_t *vm
 
     return NXT_ERROR;
 }
-
-
-nxt_inline nxt_bool_t
-njs_generator_is_constant(njs_parser_node_t *node)
-{
-    return (node->token >= NJS_TOKEN_FIRST_CONST
-            && node->token <= NJS_TOKEN_LAST_CONST);
-}
diff -r 2d9ccf739861 -r 4337ed48d6d6 njs/njs_parser.c
--- a/njs/njs_parser.c	Sun Jan 01 20:45:59 2017 +0300
+++ b/njs/njs_parser.c	Mon Jan 02 22:59:24 2017 +0300
@@ -38,11 +38,15 @@
  * is treated as a single expiression.
  */
 
+static njs_ret_t njs_parser_scope_begin(njs_vm_t *vm, njs_parser_t *parser,
+    njs_scope_t type);
+static void njs_parser_scope_end(njs_vm_t *vm, njs_parser_t *parser);
 static njs_token_t njs_parser_statement_chain(njs_vm_t *vm,
     njs_parser_t *parser, njs_token_t token);
 static njs_token_t njs_parser_statement(njs_vm_t *vm, njs_parser_t *parser,
     njs_token_t token);
-static njs_token_t njs_parser_block(njs_vm_t *vm, njs_parser_t *parser);
+static njs_token_t njs_parser_block_statement(njs_vm_t *vm,
+    njs_parser_t *parser);
 static njs_token_t njs_parser_function_declaration(njs_vm_t *vm,
     njs_parser_t *parser);
 static njs_parser_t *njs_parser_function_create(njs_vm_t *vm,
@@ -89,9 +93,15 @@ static njs_token_t njs_parser_unexpected
 njs_parser_node_t *
 njs_parser(njs_vm_t *vm, njs_parser_t *parser)
 {
+    njs_ret_t          ret;
     njs_token_t        token;
     njs_parser_node_t  *node;
 
+    ret = njs_parser_scope_begin(vm, parser, NJS_SCOPE_GLOBAL);
+    if (nxt_slow_path(ret != NXT_OK)) {
+        return NULL;
+    }
+
     token = njs_parser_token(parser);
 
     while (token != NJS_TOKEN_END) {
@@ -112,6 +122,7 @@ njs_parser(njs_vm_t *vm, njs_parser_t *p
     if (node != NULL && node->right != NULL) {
         if (node->right->token == NJS_TOKEN_FUNCTION) {
             node->token = NJS_TOKEN_CALL;
+            node->scope = parser->scope;
             return node;
         }
 
@@ -125,11 +136,83 @@ njs_parser(njs_vm_t *vm, njs_parser_t *p
     }
 
     node->token = NJS_TOKEN_END;
+    node->scope = parser->scope;
 
     return node;
 }
 
 
+static njs_ret_t
+njs_parser_scope_begin(njs_vm_t *vm, njs_parser_t *parser, njs_scope_t type)
+{
+    nxt_array_t         *values;
+    njs_parser_scope_t  *scope, *parent;
+
+    scope = nxt_mem_cache_alloc(vm->mem_cache_pool, sizeof(njs_parser_scope_t));
+    if (nxt_slow_path(scope == NULL)) {
+        return NXT_ERROR;
+    }
+
+    scope->type = type;
+
+    if (type == NJS_SCOPE_GLOBAL) {
+        type += NJS_INDEX_GLOBAL_OFFSET;
+    }
+
+    scope->next_index = type;
+
+    scope->inclusive = 0;
+    nxt_lvlhsh_init(&scope->variables);
+
+    values = NULL;
+
+    if (scope->type < NJS_SCOPE_BLOCK) {
+        values = nxt_array_create(4, sizeof(njs_value_t), &njs_array_mem_proto,
+                                  vm->mem_cache_pool);
+        if (nxt_slow_path(values == NULL)) {
+            return NXT_ERROR;
+        }
+    }
+
+    scope->values = values;
+
+    parent = parser->scope;
+
+    if (parent != NULL) {
+        parent->inclusive++;
+    }
+
+    scope->parent = parent;
+    parser->scope = scope;
+
+    return NXT_OK;
+}
+
+
+static void
+njs_parser_scope_end(njs_vm_t *vm, njs_parser_t *parser)
+{
+    njs_parser_scope_t  *scope, *parent;
+
+    scope = parser->scope;
+
+    parent = scope->parent;
+
+#if 0
+    if (scope->inclusive == 0
+        && scope->type == NJS_SCOPE_BLOCK
+        && nxt_lvlhsh_is_empty(&scope->variables))
+    {
+        parent->inclusive--;
+
+        nxt_mem_cache_free(vm->mem_cache_pool, scope);
+    }
+#endif
+
+    parser->scope = parent;
+}
+
+
 static njs_token_t
 njs_parser_statement_chain(njs_vm_t *vm, njs_parser_t *parser,
     njs_token_t token)
@@ -213,7 +296,7 @@ njs_parser_statement(njs_vm_t *vm, njs_p
         return njs_parser_token(parser);
 
     case NJS_TOKEN_OPEN_BRACE:
-        return njs_parser_block(vm, parser);
+        return njs_parser_block_statement(vm, parser);
 
     case NJS_TOKEN_CLOSE_BRACE:
         parser->node = NULL;
@@ -251,8 +334,9 @@ njs_parser_statement(njs_vm_t *vm, njs_p
 
 
 static njs_token_t

From igor at sysoev.ru  Tue Jan  3 15:35:29 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Tue, 03 Jan 2017 15:35:29 +0000
Subject: [njs] For and for-in loops support variable declaration.
Message-ID: <hg.6c5bebb914ef.1483457729.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/6c5bebb914ef
branches:  
changeset: 294:6c5bebb914ef
user:      Igor Sysoev <igor at sysoev.ru>
date:      Mon Jan 02 22:59:29 2017 +0300
description:
For and for-in loops support variable declaration.

diffstat:

 njs/njs_parser.c         |  218 +++++++++++++++++++++++++++++++++++++++++-----
 njs/test/njs_unit_test.c |   13 ++
 2 files changed, 205 insertions(+), 26 deletions(-)

diffs (272 lines):

diff -r 4337ed48d6d6 -r 6c5bebb914ef njs/njs_parser.c
--- a/njs/njs_parser.c	Mon Jan 02 22:59:24 2017 +0300
+++ b/njs/njs_parser.c	Mon Jan 02 22:59:29 2017 +0300
@@ -64,6 +64,10 @@ static njs_token_t njs_parser_while_stat
 static njs_token_t njs_parser_do_while_statement(njs_vm_t *vm,
     njs_parser_t *parser);
 static njs_token_t njs_parser_for_statement(njs_vm_t *vm, njs_parser_t *parser);
+static njs_token_t njs_parser_for_var_statement(njs_vm_t *vm,
+    njs_parser_t *parser);
+static njs_token_t njs_parser_for_var_in_statement(njs_vm_t *vm,
+    njs_parser_t *parser, njs_parser_node_t *name);
 static njs_token_t njs_parser_for_in_statement(njs_vm_t *vm,
     njs_parser_t *parser, nxt_str_t *name, njs_token_t token);
 static njs_token_t njs_parser_continue_statement(njs_vm_t *vm,
@@ -1122,17 +1126,34 @@ njs_parser_for_statement(njs_vm_t *vm, n
     }
 
     if (token != NJS_TOKEN_SEMICOLON) {
-        name = parser->lexer->text;
-
-        token = njs_parser_expression(vm, parser, token);
-        if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
-            return token;
-        }
-
-        init = parser->node;
-
-        if (init->token == NJS_TOKEN_IN) {
-            return njs_parser_for_in_statement(vm, parser, &name, token);
+
+        if (token == NJS_TOKEN_VAR) {
+
+            token = njs_parser_for_var_statement(vm, parser);
+            if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+                return token;
+            }
+
+            init = parser->node;
+
+            if (init->token == NJS_TOKEN_FOR_IN) {
+                return token;
+            }
+
+        } else {
+
+            name = parser->lexer->text;
+
+            token = njs_parser_expression(vm, parser, token);
+            if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+                return token;
+            }
+
+            init = parser->node;
+
+            if (init->token == NJS_TOKEN_IN) {
+                return njs_parser_for_in_statement(vm, parser, &name, token);
+            }
         }
     }
 
@@ -1148,21 +1169,21 @@ njs_parser_for_statement(njs_vm_t *vm, n
             return token;
         }
 
-        condition = parser->node;
-    }
-
-    token = njs_parser_match(vm, parser, token, NJS_TOKEN_SEMICOLON);
-    if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
-        return token;
-    }
-
-    if (token != NJS_TOKEN_CLOSE_PARENTHESIS) {
-
-        token = njs_parser_expression(vm, parser, token);
-        if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
-            return token;
-        }
-
+        condition = parser->node;
+    }
+
+    token = njs_parser_match(vm, parser, token, NJS_TOKEN_SEMICOLON);
+    if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+        return token;
+    }
+
+    if (token != NJS_TOKEN_CLOSE_PARENTHESIS) {
+
+        token = njs_parser_expression(vm, parser, token);
+        if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+            return token;
+        }
+
         update = parser->node;
     }
 
@@ -1209,6 +1230,151 @@ njs_parser_for_statement(njs_vm_t *vm, n
 
 
 static njs_token_t
+njs_parser_for_var_statement(njs_vm_t *vm, njs_parser_t *parser)
+{
+    njs_ret_t          ret;
+    njs_token_t        token;
+    njs_variable_t     *var;
+    njs_parser_node_t  *left, *stmt, *name, *assign, *expr;
+
+    parser->node = NULL;
+    left = NULL;
+
+    do {
+        token = njs_parser_token(parser);
+        if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+            return token;
+        }
+
+        if (token != NJS_TOKEN_NAME) {
+            return NJS_TOKEN_ILLEGAL;
+        }
+
+        var = njs_variable_add(vm, parser, NJS_VARIABLE_VAR);
+        if (nxt_slow_path(var == NULL)) {
+            return NJS_TOKEN_ERROR;
+        }
+
+        name = njs_parser_node_alloc(vm);
+        if (nxt_slow_path(name == NULL)) {
+            return NJS_TOKEN_ERROR;
+        }
+
+        name->token = NJS_TOKEN_NAME;
+
+        ret = njs_variable_reference(vm, parser, name);
+        if (nxt_slow_path(ret != NXT_OK)) {
+            return NJS_TOKEN_ERROR;
+        }
+
+        token = njs_parser_token(parser);
+        if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+            return token;
+        }
+
+        if (token == NJS_TOKEN_IN) {
+            return njs_parser_for_var_in_statement(vm, parser, name);
+        }
+
+        expr = NULL;
+
+        if (token == NJS_TOKEN_ASSIGNMENT) {
+
+            token = njs_parser_token(parser);
+            if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+                return token;
+            }
+
+            token = njs_parser_var_expression(vm, parser, token);
+            if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+                return token;
+            }
+
+            expr = parser->node;
+        }
+
+        assign = njs_parser_node_alloc(vm);
+        if (nxt_slow_path(assign == NULL)) {
+            return NJS_TOKEN_ERROR;
+        }
+
+        assign->token = NJS_TOKEN_VAR;
+        assign->u.operation = njs_vmcode_move;
+        assign->left = name;
+        assign->right = expr;
+
+        stmt = njs_parser_node_alloc(vm);
+        if (nxt_slow_path(stmt == NULL)) {
+            return NJS_TOKEN_ERROR;
+        }
+
+        stmt->token = NJS_TOKEN_STATEMENT;
+        stmt->left = left;
+        stmt->right = assign;
+        parser->node = stmt;
+        parser->code_size += sizeof(njs_vmcode_2addr_t);
+
+        left = stmt;
+
+    } while (token == NJS_TOKEN_COMMA);
+
+    return token;
+}
+
+
+static njs_token_t
+njs_parser_for_var_in_statement(njs_vm_t *vm, njs_parser_t *parser,
+    njs_parser_node_t *name)
+{
+    njs_token_t        token;
+    njs_parser_node_t  *node, *foreach;
+
+    token = njs_parser_token(parser);
+    if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+        return token;
+    }
+
+    token = njs_parser_expression(vm, parser, token);
+    if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+        return token;
+    }
+
+    node = njs_parser_node_alloc(vm);
+    if (nxt_slow_path(node == NULL)) {
+        return NJS_TOKEN_ERROR;
+    }
+
+    node->token = NJS_TOKEN_IN;
+    node->left = name;
+    node->right = parser->node;
+
+    token = njs_parser_match(vm, parser, token, NJS_TOKEN_CLOSE_PARENTHESIS);
+    if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+        return token;
+    }
+
+    token = njs_parser_statement(vm, parser, token);
+    if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+        return token;
+    }
+
+    foreach = njs_parser_node_alloc(vm);
+    if (nxt_slow_path(foreach == NULL)) {
+        return NJS_TOKEN_ERROR;
+    }
+
+    foreach->token = NJS_TOKEN_FOR_IN;
+    foreach->left = node;
+    foreach->right = parser->node;
+
+    parser->node = foreach;
+    parser->code_size += sizeof(njs_vmcode_prop_foreach_t)
+                         + sizeof(njs_vmcode_prop_next_t);
+    return token;
+}
+
+
+static njs_token_t
 njs_parser_for_in_statement(njs_vm_t *vm, njs_parser_t *parser, nxt_str_t *name,
     njs_token_t token)
 {
diff -r 4337ed48d6d6 -r 6c5bebb914ef njs/test/njs_unit_test.c
--- a/njs/test/njs_unit_test.c	Mon Jan 02 22:59:24 2017 +0300
+++ b/njs/test/njs_unit_test.c	Mon Jan 02 22:59:29 2017 +0300
@@ -4506,6 +4506,19 @@ static njs_unit_test_t  njs_test[] =
     { nxt_string("function f(a) { return a } var a = '2'; a + f(5)"),
       nxt_string("25") },
 
+    { nxt_string("for (var i = 0; i < 5; i++); i"),
+      nxt_string("5") },
+
+    { nxt_string("for (var i = 0, j; i < 5; i++); j"),
+      nxt_string("undefined") },
+
+    { nxt_string("for (var i = 0, j, k = 3; i < 5; i++); k"),
+      nxt_string("3") },
+
+    { nxt_string("var o = { a: 1, b: 2, c: 3 }, s = ''; "
+                 "for (var i in o) { s += i }; s"),
+      nxt_string("abc") },
+
     /* RegExp. */
 
     { nxt_string("/./x"),

From igor at sysoev.ru  Tue Jan  3 15:35:30 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Tue, 03 Jan 2017 15:35:30 +0000
Subject: [njs] For-in loop does not discard the last value of property
 variable.
Message-ID: <hg.8e1de8ab59e6.1483457730.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/8e1de8ab59e6
branches:  
changeset: 295:8e1de8ab59e6
user:      Igor Sysoev <igor at sysoev.ru>
date:      Mon Jan 02 22:59:31 2017 +0300
description:
For-in loop does not discard the last value of property variable.

diffstat:

 njs/njs_generator.c      |   2 +-
 njs/njs_vm.c             |  10 +++++-----
 njs/test/njs_unit_test.c |   9 +++++++++
 3 files changed, 15 insertions(+), 6 deletions(-)

diffs (82 lines):

diff -r 6c5bebb914ef -r 8e1de8ab59e6 njs/njs_generator.c
--- a/njs/njs_generator.c	Mon Jan 02 22:59:29 2017 +0300
+++ b/njs/njs_generator.c	Mon Jan 02 22:59:31 2017 +0300
@@ -1047,7 +1047,7 @@ njs_generate_for_in_statement(njs_vm_t *
     njs_generate_code(parser, njs_vmcode_prop_next_t, prop_next);
     prop_next->code.operation = njs_vmcode_property_next;
     prop_next->code.operands = NJS_VMCODE_3OPERANDS;
-    prop_next->code.retval = NJS_VMCODE_RETVAL;
+    prop_next->code.retval = NJS_VMCODE_NO_RETVAL;
     prop_next->retval = foreach->left->index;
     prop_next->object = foreach->right->index;
     prop_next->next = index;
diff -r 6c5bebb914ef -r 8e1de8ab59e6 njs/njs_vm.c
--- a/njs/njs_vm.c	Mon Jan 02 22:59:29 2017 +0300
+++ b/njs/njs_vm.c	Mon Jan 02 22:59:31 2017 +0300
@@ -1227,6 +1227,7 @@ njs_vmcode_property_next(njs_vm_t *vm, n
 {
     njs_ret_t               ret;
     nxt_uint_t              n;
+    njs_value_t             *retval;
     njs_array_t             *array;
     njs_extern_t            *ext;
     njs_object_prop_t       *prop;
@@ -1234,6 +1235,7 @@ njs_vmcode_property_next(njs_vm_t *vm, n
     njs_vmcode_prop_next_t  *code;
 
     code = (njs_vmcode_prop_next_t *) vm->current;
+    retval = njs_vmcode_operand(vm, code->retval);
 
     if (njs_is_object(object)) {
         next = value->data.u.next;
@@ -1245,7 +1247,7 @@ njs_vmcode_property_next(njs_vm_t *vm, n
                 n = next->index++;
 
                 if (njs_is_valid(&array->start[n])) {
-                    njs_number_set(&vm->retval, n);
+                    njs_number_set(retval, n);
 
                     return code->offset;
                 }
@@ -1257,20 +1259,18 @@ njs_vmcode_property_next(njs_vm_t *vm, n
         prop = nxt_lvlhsh_each(&object->data.u.object->hash, &next->lhe);
 
         if (prop != NULL) {
-            vm->retval = prop->name;
+            *retval = prop->name;
 
             return code->offset;
         }
 
         nxt_mem_cache_free(vm->mem_cache_pool, next);
 
-        vm->retval = njs_value_void;
-
     } else if (njs_is_external(object)) {
         ext = object->data.u.external;
 
         if (ext->next != NULL) {
-            ret = ext->next(vm, &vm->retval, vm->external[ext->object], value);
+            ret = ext->next(vm, retval, vm->external[ext->object], value);
 
             if (ret == NXT_OK) {
                 return code->offset;
diff -r 6c5bebb914ef -r 8e1de8ab59e6 njs/test/njs_unit_test.c
--- a/njs/test/njs_unit_test.c	Mon Jan 02 22:59:29 2017 +0300
+++ b/njs/test/njs_unit_test.c	Mon Jan 02 22:59:31 2017 +0300
@@ -4519,6 +4519,15 @@ static njs_unit_test_t  njs_test[] =
                  "for (var i in o) { s += i }; s"),
       nxt_string("abc") },
 
+    { nxt_string("var o = { a: 1, b: 2, c: 3 }; for (var i in o); i"),
+      nxt_string("c") },
+
+    { nxt_string("var o = {}; i = 7; for (var i in o); i"),
+      nxt_string("7") },
+
+    { nxt_string("var a = [1,2,3]; for (var i in a); i"),
+      nxt_string("2") },
+
     /* RegExp. */
 
     { nxt_string("/./x"),

From igor at sysoev.ru  Tue Jan  3 15:35:31 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Tue, 03 Jan 2017 15:35:31 +0000
Subject: [njs] Fixed code generation for "in" operation with side effect.
Message-ID: <hg.9672a3c3aaae.1483457731.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/9672a3c3aaae
branches:  
changeset: 296:9672a3c3aaae
user:      Igor Sysoev <igor at sysoev.ru>
date:      Mon Jan 02 22:59:33 2017 +0300
description:
Fixed code generation for "in" operation with side effect.

diffstat:

 njs/njs_generator.c      |  41 +++++++++++++++++++++--------------------
 njs/test/njs_unit_test.c |   3 +++
 2 files changed, 24 insertions(+), 20 deletions(-)

diffs (99 lines):

diff -r 8e1de8ab59e6 -r 9672a3c3aaae njs/njs_generator.c
--- a/njs/njs_generator.c	Mon Jan 02 22:59:31 2017 +0300
+++ b/njs/njs_generator.c	Mon Jan 02 22:59:33 2017 +0300
@@ -81,7 +81,7 @@ static nxt_int_t njs_generate_regexp(njs
 static nxt_int_t njs_generate_test_jump_expression(njs_vm_t *vm,
     njs_parser_t *parser, njs_parser_node_t *node);
 static nxt_int_t njs_generate_3addr_operation(njs_vm_t *vm,
-    njs_parser_t *parser, njs_parser_node_t *node);
+    njs_parser_t *parser, njs_parser_node_t *node, nxt_bool_t swap);
 static nxt_int_t njs_generate_2addr_operation(njs_vm_t *vm,
     njs_parser_t *parser, njs_parser_node_t *node);
 static nxt_int_t njs_generate_typeof_operation(njs_vm_t *vm,
@@ -128,8 +128,6 @@ static const nxt_str_t  no_label = { 0, 
 static nxt_int_t
 njs_generator(njs_vm_t *vm, njs_parser_t *parser, njs_parser_node_t *node)
 {
-    njs_parser_node_t  *left;
-
     if (node == NULL) {
         return NXT_OK;
     }
@@ -195,19 +193,6 @@ njs_generator(njs_vm_t *vm, njs_parser_t
     case NJS_TOKEN_REMAINDER_ASSIGNMENT:
         return njs_generate_operation_assignment(vm, parser, node);
 
-    case NJS_TOKEN_IN:
-        /*
-         * An "in" operation is parsed as standard binary expression
-         * by njs_parser_binary_expression().  However, its operands
-         * should be swapped to be uniform with other property operations
-         * (get/set and delete) to use the property trap.
-         */
-        left = node->left;
-        node->left = node->right;
-        node->right = left;
-
-        /* Fall through. */
-
     case NJS_TOKEN_BITWISE_OR:
     case NJS_TOKEN_BITWISE_XOR:
     case NJS_TOKEN_BITWISE_AND:
@@ -231,7 +216,16 @@ njs_generator(njs_vm_t *vm, njs_parser_t
     case NJS_TOKEN_REMAINDER:
     case NJS_TOKEN_PROPERTY_DELETE:
     case NJS_TOKEN_PROPERTY:
-        return njs_generate_3addr_operation(vm, parser, node);
+        return njs_generate_3addr_operation(vm, parser, node, 0);
+
+    case NJS_TOKEN_IN:
+        /*
+         * An "in" operation is parsed as standard binary expression
+         * by njs_parser_binary_expression().  However, its operands
+         * should be swapped to be uniform with other property operations
+         * (get/set and delete) to use the property trap.
+         */
+        return njs_generate_3addr_operation(vm, parser, node, 1);
 
     case NJS_TOKEN_LOGICAL_AND:
     case NJS_TOKEN_LOGICAL_OR:
@@ -1707,7 +1701,7 @@ njs_generate_test_jump_expression(njs_vm
 
 static nxt_int_t
 njs_generate_3addr_operation(njs_vm_t *vm, njs_parser_t *parser,
-    njs_parser_node_t *node)
+    njs_parser_node_t *node, nxt_bool_t swap)
 {
     nxt_int_t           ret;
     njs_index_t         index;
@@ -1751,8 +1745,15 @@ njs_generate_3addr_operation(njs_vm_t *v
     code->code.operation = node->u.operation;
     code->code.operands = NJS_VMCODE_3OPERANDS;
     code->code.retval = NJS_VMCODE_RETVAL;
-    code->src1 = left->index;
-    code->src2 = right->index;
+
+    if (!swap) {
+        code->src1 = left->index;
+        code->src2 = right->index;
+
+    } else {
+        code->src1 = right->index;
+        code->src2 = left->index;
+    }
 
     /*
      * The temporary index of MOVE destination
diff -r 8e1de8ab59e6 -r 9672a3c3aaae njs/test/njs_unit_test.c
--- a/njs/test/njs_unit_test.c	Mon Jan 02 22:59:31 2017 +0300
+++ b/njs/test/njs_unit_test.c	Mon Jan 02 22:59:33 2017 +0300
@@ -224,6 +224,9 @@ static njs_unit_test_t  njs_test[] =
     { nxt_string("var a = 1; function f(x) { a = x; return 2 }; a += f(5)"),
       nxt_string("3") },
 
+    { nxt_string("var x; x in (x = 1, [1, 2, 3])"),
+      nxt_string("false") },
+
     /* Exponentiation. */
 
     { nxt_string("2 ** 3 ** 2"),

From igor at sysoev.ru  Tue Jan  3 16:43:53 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Tue, 03 Jan 2017 16:43:53 +0000
Subject: [njs] Logical "or" and "and" operations should store result in a
Message-ID: <hg.fe8027493a08.1483461833.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/fe8027493a08
branches:  
changeset: 297:fe8027493a08
user:      Igor Sysoev <igor at sysoev.ru>
date:      Tue Jan 03 19:38:17 2017 +0300
description:
Logical "or" and "and" operations should store result in a
temporary destination because they are actually branch operations.

diffstat:

 njs/njs_generator.c      |  4 ++--
 njs/test/njs_unit_test.c |  6 ++++++
 2 files changed, 8 insertions(+), 2 deletions(-)

diffs (37 lines):

diff -r 9672a3c3aaae -r fe8027493a08 njs/njs_generator.c
--- a/njs/njs_generator.c	Mon Jan 02 22:59:33 2017 +0300
+++ b/njs/njs_generator.c	Tue Jan 03 19:38:17 2017 +0300
@@ -1666,7 +1666,7 @@ njs_generate_test_jump_expression(njs_vm
     test_jump->code.retval = NJS_VMCODE_RETVAL;
     test_jump->value = node->left->index;
 
-    node->index = njs_generator_dest_index(vm, parser, node);
+    node->index = njs_generator_node_temp_index_get(vm, parser, node);
     if (nxt_slow_path(node->index == NJS_INDEX_ERROR)) {
         return node->index;
     }
@@ -1695,7 +1695,7 @@ njs_generate_test_jump_expression(njs_vm
 
     test_jump->offset = parser->code_end - (u_char *) test_jump;
 
-    return njs_generator_node_index_release(vm, parser, node->right);
+    return njs_generator_children_indexes_release(vm, parser, node);
 }
 
 
diff -r 9672a3c3aaae -r fe8027493a08 njs/test/njs_unit_test.c
--- a/njs/test/njs_unit_test.c	Mon Jan 02 22:59:33 2017 +0300
+++ b/njs/test/njs_unit_test.c	Tue Jan 03 19:38:17 2017 +0300
@@ -568,6 +568,12 @@ static njs_unit_test_t  njs_test[] =
     { nxt_string("var a = 1; 1 || (a = 2); a"),
       nxt_string("1") },
 
+    { nxt_string("var x; x = 0 || x; x"),
+      nxt_string("undefined") },
+
+    { nxt_string("var x; x = 1 && x; x"),
+      nxt_string("undefined") },
+
     { nxt_string("1 || 2 || 3"),
       nxt_string("1") },
 

From josh at joshuaspence.com  Tue Jan  3 21:01:24 2017
From: josh at joshuaspence.com (Joshua Spence)
Date: Wed, 4 Jan 2017 08:01:24 +1100
Subject: [PATCH] Escape strings produced by the `%V` format specification
In-Reply-To: <CAFztJwQ5tU6CK=zKSXKkim7swoUGbBA3h2znLyoHUnnUHL5s=g@mail.gmail.com>
References: <30a9ce42aadfcf135f3a.1483426237@deagle>
 <CADO4dzNgRytA6c_tTAq4WGvZJB7Cies3reyYvAS2M429bcCbDA@mail.gmail.com>
 <CAFztJwQ5tU6CK=zKSXKkim7swoUGbBA3h2znLyoHUnnUHL5s=g@mail.gmail.com>
Message-ID: <CADO4dzPNn3ooipiTZJkYDd3d6M9vfFT3ZGyEhfboYieW5WQ8ng@mail.gmail.com>

%%TEST_GLOBALS%% and %%TEST_GLOBALS_HTTP%% are macros inserted by
Test::Nginx. I don't know if it is the same Test::Nginx from CPAN, it is
vendored in the nginx-tests repository.

*Joshua Spence*

*mobile:* +61 422 990 263
*email:* josh at joshuaspence.com
*web:* http://www.joshuaspence.com

On 3 January 2017 at 23:57, oscaretu . <oscaretu at gmail.com> wrote:

> Hello, Joshua.
>
> What values take %%TEST_GLOBALS%% and %%TEST_GLOBALS_HTTP%%?
>
> I've searched them in Google and I didn't find anything related. Nor I was
> unable to find anything about them in the documentation for Test::Nginx (
> http://search.cpan.org/~agent/Test-Nginx-0.25/lib/Test/Nginx.pm) neither
> in https://openresty.gitbooks.io/programming-openresty/content/
>
> Kind regards,
> Oscar
>
>
>
> On Tue, Jan 3, 2017 at 12:14 PM, Joshua Spence <josh at joshuaspence.com>
> wrote:
>
>> I tested this change with the following script:
>>
>> ```
>> #!/usr/bin/perl
>>
>> ############################################################
>> ###################
>>
>> use warnings;
>> use strict;
>>
>> use Test::More;
>>
>> BEGIN { use FindBin; chdir($FindBin::Bin); }
>>
>> use lib 'lib';
>> use Test::Nginx;
>>
>> ############################################################
>> ###################
>>
>> select STDERR; $| = 1;
>> select STDOUT; $| = 1;
>>
>> my $t = Test::Nginx->new()
>>     ->has(qw/http limit_req/)
>>     ->plan(2)
>>     ->write_file_expand('nginx.conf', <<'EOF');
>>
>> %%TEST_GLOBALS%%
>>
>> daemon off;
>>
>> events {}
>>
>> http {
>>     %%TEST_GLOBALS_HTTP%%
>>
>>     limit_req_zone $binary_remote_addr zone=one:1m rate=1r/m;
>>     limit_req zone=one;
>>
>>     server {
>>         listen       127.0.0.1:8080;
>>         server_name  localhost;
>>
>>         location / {
>>             error_log stderr;
>>         }
>>     }
>> }
>>
>> EOF
>>
>> open OLDERR, '>&', \*STDERR;
>> open STDERR, '>', $t->testdir() . '/stderr' or die "Can't reopen STDERR:
>> $!";
>> open my $stderr, '<', $t->testdir() . '/stderr' or die "Can't open stderr
>> file: $!";
>>
>> $t->run();
>>
>> open STDERR, '>&', \*OLDERR;
>>
>> ############################################################
>> ###################
>>
>> # charge limit_req
>> http_get('/');
>>
>> http_get('/"foo bar"');
>> is(lines($t, 'request: "GET /\"foo bar\" HTTP/1.0"'), 1);
>>
>> http_get_with_referrer('/', 'https://www.google.com.au/search?hl=en&q="BauGutachter"
>> +41 2016 .ch&num=100&start=0');
>> is(lines($t, 'referrer: "https://www.google.com.au/search?hl=en&q=\"BauGutachter\"
>> +41 2016 .ch&num=100&start=0"'), 1);
>>
>> ############################################################
>> ###################
>>
>> sub http_get_with_referrer {
>>     my ($url, $referer) = @_;
>>     return http(<<EOF);
>> GET $url HTTP/1.0
>> Host: localhost
>> Referer: $referer
>>
>> EOF
>> }
>>
>> sub lines {
>>     my ($t, $pattern) = @_;
>>     return map { $_ =~ /\Q$pattern\E/ } (<$stderr>);
>> }
>>
>> ############################################################
>> ###################
>> ```
>>
>> *Joshua Spence*
>>
>> *mobile:* +61 422 990 263 <+61%20422%20990%20263>
>> *email:* josh at joshuaspence.com
>> *web:* http://www.joshuaspence.com
>>
>> On 3 January 2017 at 17:50, Joshua Spence <josh at joshuaspence.com> wrote:
>>
>>> # HG changeset patch
>>> # User Joshua Spence <josh at joshuaspence.com>
>>> # Date 1483420178 -39600
>>> #      Tue Jan 03 16:09:38 2017 +1100
>>> # Branch error_log_escape
>>> # Node ID 30a9ce42aadfcf135f3ad95a296c95e7044d5f9d
>>> # Parent  a42afc225e98afe1f7c3b428c81c8af7eba362c0
>>> Escape strings produced by the `%V` format specification
>>>
>>> Currently nginx error logs loosely adhere to the following pattern:
>>>
>>> ```
>>> %{TIMESTAMP} [%{SEVERITY}] %{PROCESS_ID}#%{THREAD_ID}:(
>>> *%{CONNECTION_ID})?%{MESSAGE}(, %{KEY}: "%{VALUE}")*
>>> ```
>>>
>>> In our environment we are using Filebeat to ship nginx error logs into
>>> Logstash. Our Logstash filter for parsing nginx error logs is as follows:
>>>
>>> ```
>>> filter {
>>>   grok {
>>>     match => {
>>>       "message" => "(?<timestamp>%{YEAR}/%{MONTHNUM2}/%{MONTHDAY}
>>> %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:process_id:int}#%{NONNEGINT:thread_id:int}:(?:
>>> \*%{NONNEGINT:connection_id:int})? (?<message>[^,]*)(?:,
>>> %{GREEDYDATA:nginx_keyvalue_data})?"
>>>     }
>>>     overwrite => ["message"]
>>>   }
>>>
>>>   date {
>>>     match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
>>>     remove_field => ["timestamp"]
>>>   }
>>>
>>>   kv {
>>>     field_split => ", "
>>>     remove_field => "nginx_keyvalue_data"
>>>     source => "nginx_keyvalue_data"
>>>     value_split => ": "
>>>   }
>>> }
>>> ```
>>>
>>> The problem, however, is that it is difficult to unambigiously parse the
>>> key-value data from the nginx error log format because of a lack of data
>>> sanitization. Specifically, consider the following entry from our nginx
>>> error logs:
>>>
>>> ```
>>> 2016/11/15 03:45:51 [error] 23770#23770: *21535000 open()
>>> "/REDACTED/+"BauGutachter"++41+2016+.ch&tbo=1" failed (2: No such file
>>> or directory), client: REDACTED, server: REDACTED, request: "GET
>>> /job/+%22BauGutachter%22++41+2016+.ch&tbo=1 HTTP/1.1", host:
>>> "REDACTED", referrer: "https://www.google.com.au/search?hl=en&q="BauGutachter"
>>> +41 2016 .ch&num=100&start=0"
>>> ```
>>>
>>> Specifically, the problem in the above sample is the `referrer` field,
>>> the value of which contains double quotes. I think that it makes sense to
>>> escape `\` and `"` characters before they are written to the error log.
>>>
>>> diff -r a42afc225e98 -r 30a9ce42aadf src/core/ngx_string.c
>>> --- a/src/core/ngx_string.c     Tue Dec 27 17:23:08 2016 +0300
>>> +++ b/src/core/ngx_string.c     Tue Jan 03 16:09:38 2017 +1100
>>> @@ -15,7 +15,7 @@
>>>      const u_char *basis, ngx_uint_t padding);
>>>  static ngx_int_t ngx_decode_base64_internal(ngx_str_t *dst, ngx_str_t
>>> *src,
>>>      const u_char *basis);
>>> -
>>> +static uintptr_t ngx_escape_string(u_char *dst, u_char *src, size_t
>>> size);
>>>
>>>  void
>>>  ngx_strlow(u_char *dst, u_char *src, size_t n)
>>> @@ -235,7 +235,7 @@
>>>                  v = va_arg(args, ngx_str_t *);
>>>
>>>                  len = ngx_min(((size_t) (last - buf)), v->len);
>>> -                buf = ngx_cpymem(buf, v->data, len);
>>> +                buf = (u_char *) ngx_escape_string(buf, v->data, len);
>>>                  fmt++;
>>>
>>>                  continue;
>>> @@ -1844,6 +1844,43 @@
>>>  }
>>>
>>>
>>> +static uintptr_t
>>> +ngx_escape_string(u_char *dst, u_char *src, size_t size)
>>> +{
>>> +    u_char      ch;
>>> +    ngx_uint_t  len;
>>> +
>>> +    if (dst == NULL) {
>>> +        len = 0;
>>> +
>>> +        while (size) {
>>> +            ch = *src++;
>>> +
>>> +            if (ch == '\\' || ch == '"') {
>>> +                len++;
>>> +            }
>>> +
>>> +            size--;
>>> +        }
>>> +
>>> +        return (uintptr_t) len;
>>> +    }
>>> +
>>> +    while (size) {
>>> +        ch = *src++;
>>> +
>>> +        if (ch == '\\' || ch == '"') {
>>> +            *dst++ = '\\';
>>> +        }
>>> +
>>> +        *dst++ = ch;
>>> +        size--;
>>> +    }
>>> +
>>> +    return (uintptr_t) dst;
>>> +}
>>> +
>>> +
>>>  void
>>>  ngx_str_rbtree_insert_value(ngx_rbtree_node_t *temp,
>>>      ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel)
>>>
>>
>>
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>
>
>
>
> --
> Oscar Fernandez Sierra
> oscaretu at gmail.com
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170104/ba2c3e06/attachment-0001.html>

From oscaretu at gmail.com  Tue Jan  3 21:31:11 2017
From: oscaretu at gmail.com (oscaretu .)
Date: Tue, 3 Jan 2017 22:31:11 +0100
Subject: [PATCH] Escape strings produced by the `%V` format specification
In-Reply-To: <CADO4dzPNn3ooipiTZJkYDd3d6M9vfFT3ZGyEhfboYieW5WQ8ng@mail.gmail.com>
References: <30a9ce42aadfcf135f3a.1483426237@deagle>
 <CADO4dzNgRytA6c_tTAq4WGvZJB7Cies3reyYvAS2M429bcCbDA@mail.gmail.com>
 <CAFztJwQ5tU6CK=zKSXKkim7swoUGbBA3h2znLyoHUnnUHL5s=g@mail.gmail.com>
 <CADO4dzPNn3ooipiTZJkYDd3d6M9vfFT3ZGyEhfboYieW5WQ8ng@mail.gmail.com>
Message-ID: <CAFztJwSWZs7gNcLJSnqHbN-QZ_QGG8gEuTX=GCYj5jFY_2eejg@mail.gmail.com>

Hello, Joshua. Thanks a lot for your explanation,

Agentzh says in the documentation about testing Nginx and OpenResty

 https://openresty.gitbooks.io/programming-openresty/content/

that there are two different Perl modules Test::Nginx. It seems that the
one you were using isn't the same one I supposed you were using. I think it
isn't a good idea having given the same name to two different Perl modules
:-(

Oscar


On Tue, Jan 3, 2017 at 10:01 PM, Joshua Spence <josh at joshuaspence.com>
wrote:

> %%TEST_GLOBALS%% and %%TEST_GLOBALS_HTTP%% are macros inserted by
> Test::Nginx. I don't know if it is the same Test::Nginx from CPAN, it is
> vendored in the nginx-tests repository.
>
> *Joshua Spence*
>
> *mobile:* +61 422 990 263 <+61%20422%20990%20263>
> *email:* josh at joshuaspence.com
> *web:* http://www.joshuaspence.com
>
> On 3 January 2017 at 23:57, oscaretu . <oscaretu at gmail.com> wrote:
>
>> Hello, Joshua.
>>
>> What values take %%TEST_GLOBALS%% and %%TEST_GLOBALS_HTTP%%?
>>
>> I've searched them in Google and I didn't find anything related. Nor I
>> was unable to find anything about them in the documentation for Test::Nginx
>> (http://search.cpan.org/~agent/Test-Nginx-0.25/lib/Test/Nginx.pm)
>> neither in https://openresty.gitbooks.io/programming-openresty/content/
>>
>> Kind regards,
>> Oscar
>>
>>
>>
>> On Tue, Jan 3, 2017 at 12:14 PM, Joshua Spence <josh at joshuaspence.com>
>> wrote:
>>
>>> I tested this change with the following script:
>>>
>>> ```
>>> #!/usr/bin/perl
>>>
>>> ############################################################
>>> ###################
>>>
>>> use warnings;
>>> use strict;
>>>
>>> use Test::More;
>>>
>>> BEGIN { use FindBin; chdir($FindBin::Bin); }
>>>
>>> use lib 'lib';
>>> use Test::Nginx;
>>>
>>> ############################################################
>>> ###################
>>>
>>> select STDERR; $| = 1;
>>> select STDOUT; $| = 1;
>>>
>>> my $t = Test::Nginx->new()
>>>     ->has(qw/http limit_req/)
>>>     ->plan(2)
>>>     ->write_file_expand('nginx.conf', <<'EOF');
>>>
>>> %%TEST_GLOBALS%%
>>>
>>> daemon off;
>>>
>>> events {}
>>>
>>> http {
>>>     %%TEST_GLOBALS_HTTP%%
>>>
>>>     limit_req_zone $binary_remote_addr zone=one:1m rate=1r/m;
>>>     limit_req zone=one;
>>>
>>>     server {
>>>         listen       127.0.0.1:8080;
>>>         server_name  localhost;
>>>
>>>         location / {
>>>             error_log stderr;
>>>         }
>>>     }
>>> }
>>>
>>> EOF
>>>
>>> open OLDERR, '>&', \*STDERR;
>>> open STDERR, '>', $t->testdir() . '/stderr' or die "Can't reopen STDERR:
>>> $!";
>>> open my $stderr, '<', $t->testdir() . '/stderr' or die "Can't open
>>> stderr file: $!";
>>>
>>> $t->run();
>>>
>>> open STDERR, '>&', \*OLDERR;
>>>
>>> ############################################################
>>> ###################
>>>
>>> # charge limit_req
>>> http_get('/');
>>>
>>> http_get('/"foo bar"');
>>> is(lines($t, 'request: "GET /\"foo bar\" HTTP/1.0"'), 1);
>>>
>>> http_get_with_referrer('/', 'https://www.google.com.au/search?hl=en&q="BauGutachter"
>>> +41 2016 .ch&num=100&start=0');
>>> is(lines($t, 'referrer: "https://www.google.com.au/search?hl=en&q=\"BauGutachter\"
>>> +41 2016 .ch&num=100&start=0"'), 1);
>>>
>>> ############################################################
>>> ###################
>>>
>>> sub http_get_with_referrer {
>>>     my ($url, $referer) = @_;
>>>     return http(<<EOF);
>>> GET $url HTTP/1.0
>>> Host: localhost
>>> Referer: $referer
>>>
>>> EOF
>>> }
>>>
>>> sub lines {
>>>     my ($t, $pattern) = @_;
>>>     return map { $_ =~ /\Q$pattern\E/ } (<$stderr>);
>>> }
>>>
>>> ############################################################
>>> ###################
>>> ```
>>>
>>> *Joshua Spence*
>>>
>>> *mobile:* +61 422 990 263 <+61%20422%20990%20263>
>>> *email:* josh at joshuaspence.com
>>> *web:* http://www.joshuaspence.com
>>>
>>> On 3 January 2017 at 17:50, Joshua Spence <josh at joshuaspence.com> wrote:
>>>
>>>> # HG changeset patch
>>>> # User Joshua Spence <josh at joshuaspence.com>
>>>> # Date 1483420178 -39600
>>>> #      Tue Jan 03 16:09:38 2017 +1100
>>>> # Branch error_log_escape
>>>> # Node ID 30a9ce42aadfcf135f3ad95a296c95e7044d5f9d
>>>> # Parent  a42afc225e98afe1f7c3b428c81c8af7eba362c0
>>>> Escape strings produced by the `%V` format specification
>>>>
>>>> Currently nginx error logs loosely adhere to the following pattern:
>>>>
>>>> ```
>>>> %{TIMESTAMP} [%{SEVERITY}] %{PROCESS_ID}#%{THREAD_ID}:(
>>>> *%{CONNECTION_ID})?%{MESSAGE}(, %{KEY}: "%{VALUE}")*
>>>> ```
>>>>
>>>> In our environment we are using Filebeat to ship nginx error logs into
>>>> Logstash. Our Logstash filter for parsing nginx error logs is as follows:
>>>>
>>>> ```
>>>> filter {
>>>>   grok {
>>>>     match => {
>>>>       "message" => "(?<timestamp>%{YEAR}/%{MONTHNUM2}/%{MONTHDAY}
>>>> %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:process_id:int}#%{NONNEGINT:thread_id:int}:(?:
>>>> \*%{NONNEGINT:connection_id:int})? (?<message>[^,]*)(?:,
>>>> %{GREEDYDATA:nginx_keyvalue_data})?"
>>>>     }
>>>>     overwrite => ["message"]
>>>>   }
>>>>
>>>>   date {
>>>>     match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
>>>>     remove_field => ["timestamp"]
>>>>   }
>>>>
>>>>   kv {
>>>>     field_split => ", "
>>>>     remove_field => "nginx_keyvalue_data"
>>>>     source => "nginx_keyvalue_data"
>>>>     value_split => ": "
>>>>   }
>>>> }
>>>> ```
>>>>
>>>> The problem, however, is that it is difficult to unambigiously parse
>>>> the key-value data from the nginx error log format because of a lack of
>>>> data sanitization. Specifically, consider the following entry from our
>>>> nginx error logs:
>>>>
>>>> ```
>>>> 2016/11/15 03:45:51 [error] 23770#23770: *21535000 open()
>>>> "/REDACTED/+"BauGutachter"++41+2016+.ch&tbo=1" failed (2: No such file
>>>> or directory), client: REDACTED, server: REDACTED, request: "GET
>>>> /job/+%22BauGutachter%22++41+2016+.ch&tbo=1 HTTP/1.1", host:
>>>> "REDACTED", referrer: "https://www.google.com.au/search?hl=en&q="BauGutachter"
>>>> +41 2016 .ch&num=100&start=0"
>>>> ```
>>>>
>>>> Specifically, the problem in the above sample is the `referrer` field,
>>>> the value of which contains double quotes. I think that it makes sense to
>>>> escape `\` and `"` characters before they are written to the error log.
>>>>
>>>> diff -r a42afc225e98 -r 30a9ce42aadf src/core/ngx_string.c
>>>> --- a/src/core/ngx_string.c     Tue Dec 27 17:23:08 2016 +0300
>>>> +++ b/src/core/ngx_string.c     Tue Jan 03 16:09:38 2017 +1100
>>>> @@ -15,7 +15,7 @@
>>>>      const u_char *basis, ngx_uint_t padding);
>>>>  static ngx_int_t ngx_decode_base64_internal(ngx_str_t *dst, ngx_str_t
>>>> *src,
>>>>      const u_char *basis);
>>>> -
>>>> +static uintptr_t ngx_escape_string(u_char *dst, u_char *src, size_t
>>>> size);
>>>>
>>>>  void
>>>>  ngx_strlow(u_char *dst, u_char *src, size_t n)
>>>> @@ -235,7 +235,7 @@
>>>>                  v = va_arg(args, ngx_str_t *);
>>>>
>>>>                  len = ngx_min(((size_t) (last - buf)), v->len);
>>>> -                buf = ngx_cpymem(buf, v->data, len);
>>>> +                buf = (u_char *) ngx_escape_string(buf, v->data, len);
>>>>                  fmt++;
>>>>
>>>>                  continue;
>>>> @@ -1844,6 +1844,43 @@
>>>>  }
>>>>
>>>>
>>>> +static uintptr_t
>>>> +ngx_escape_string(u_char *dst, u_char *src, size_t size)
>>>> +{
>>>> +    u_char      ch;
>>>> +    ngx_uint_t  len;
>>>> +
>>>> +    if (dst == NULL) {
>>>> +        len = 0;
>>>> +
>>>> +        while (size) {
>>>> +            ch = *src++;
>>>> +
>>>> +            if (ch == '\\' || ch == '"') {
>>>> +                len++;
>>>> +            }
>>>> +
>>>> +            size--;
>>>> +        }
>>>> +
>>>> +        return (uintptr_t) len;
>>>> +    }
>>>> +
>>>> +    while (size) {
>>>> +        ch = *src++;
>>>> +
>>>> +        if (ch == '\\' || ch == '"') {
>>>> +            *dst++ = '\\';
>>>> +        }
>>>> +
>>>> +        *dst++ = ch;
>>>> +        size--;
>>>> +    }
>>>> +
>>>> +    return (uintptr_t) dst;
>>>> +}
>>>> +
>>>> +
>>>>  void
>>>>  ngx_str_rbtree_insert_value(ngx_rbtree_node_t *temp,
>>>>      ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel)
>>>>
>>>
>>>
>>> _______________________________________________
>>> nginx-devel mailing list
>>> nginx-devel at nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>>
>>
>>
>>
>> --
>> Oscar Fernandez Sierra
>> oscaretu at gmail.com
>>
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>
>
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>



-- 
Oscar Fernandez Sierra
oscaretu at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170103/cb3d5c6c/attachment.html>

From mdounin at mdounin.ru  Thu Jan  5 00:18:36 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 5 Jan 2017 03:18:36 +0300
Subject: [PATCH] Escape strings produced by the `%V` format specification
In-Reply-To: <8e734d01f026c4346b46.1483424699@deagle>
References: <8e734d01f026c4346b46.1483424699@deagle>
Message-ID: <20170105001833.GB93993@mdounin.ru>

Hello!

On Tue, Jan 03, 2017 at 05:24:59PM +1100, josh at joshuaspence.com wrote:

> # HG changeset patch
> # User Joshua Spence <josh at joshuaspence.com>
> # Date 1483420178 -39600
> #      Tue Jan 03 16:09:38 2017 +1100
> # Node ID 8e734d01f026c4346b46006ee9b558031439be61
> # Parent  a42afc225e98afe1f7c3b428c81c8af7eba362c0
> Escape strings produced by the `%V` format specification
> 
> Currently nginx error logs loosely adhere to the following pattern:

The %V format specification is used not only by error logging, so 
this patch is certainly wrong.  Note well that not escaped strings 
in some cases are logged intentionally, e.g., in debugging logs.  
Another question to consider is performance, it may be quite 
important during logging, especially debug one.  And note well 
that not only %V format specification can be used to log strings.

We have ticket #191 in Trac about making error logs 
machine-parseable (see https://trac.nginx.org/nginx/ticket/191), 
but there are lots of open questions on how to do it properly.

-- 
Maxim Dounin
http://nginx.org/

From igor at sysoev.ru  Thu Jan  5 13:19:47 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Thu, 05 Jan 2017 13:19:47 +0000
Subject: [njs] Functions were not exported.
Message-ID: <hg.c11a7e133205.1483622387.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/c11a7e133205
branches:  
changeset: 298:c11a7e133205
user:      Igor Sysoev <igor at sysoev.ru>
date:      Thu Jan 05 15:55:49 2017 +0300
description:
Functions were not exported.

The bug was introduced in 4337ed48d6d6.

diffstat:

 njs/njs_generator.c |  8 +++++++-
 njs/njs_parser.c    |  8 ++++++--
 njs/njs_variable.c  |  3 +--
 njs/njscript.c      |  9 ++++++++-
 4 files changed, 22 insertions(+), 6 deletions(-)

diffs (98 lines):

diff -r fe8027493a08 -r c11a7e133205 njs/njs_generator.c
--- a/njs/njs_generator.c	Tue Jan 03 19:38:17 2017 +0300
+++ b/njs/njs_generator.c	Thu Jan 05 15:55:49 2017 +0300
@@ -1964,9 +1964,15 @@ static nxt_int_t
 njs_generate_function_declaration(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node)
 {
+    njs_variable_t         *var;
     njs_function_lambda_t  *lambda;
 
-    lambda = node->u.value.data.u.function->u.lambda;
+    var = njs_variable_get(vm, node, NJS_NAME_DECLARATION);
+    if (nxt_slow_path(var == NULL)) {
+        return NXT_ERROR;
+    }
+
+    lambda = var->value.data.u.function->u.lambda;
 
     return njs_generate_function_scope(vm, lambda, node);
 }
diff -r fe8027493a08 -r c11a7e133205 njs/njs_parser.c
--- a/njs/njs_parser.c	Tue Jan 03 19:38:17 2017 +0300
+++ b/njs/njs_parser.c	Thu Jan 05 15:55:49 2017 +0300
@@ -383,6 +383,7 @@ njs_parser_match(njs_vm_t *vm, njs_parse
 static njs_token_t
 njs_parser_function_declaration(njs_vm_t *vm, njs_parser_t *parser)
 {
+    njs_ret_t          ret;
     njs_token_t        token;
     njs_variable_t     *var;
     njs_function_t     *function;
@@ -409,6 +410,11 @@ njs_parser_function_declaration(njs_vm_t
         return NJS_TOKEN_ERROR;
     }
 
+    ret = njs_variable_reference(vm, parser, node);
+    if (nxt_slow_path(ret != NXT_OK)) {
+        return NJS_TOKEN_ERROR;
+    }
+
     token = njs_parser_token(parser);
     if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
         return token;
@@ -425,8 +431,6 @@ njs_parser_function_declaration(njs_vm_t
     var->value.type = NJS_FUNCTION;
     var->value.data.truth = 1;
 
-    node->u.value = var->value;
-
     parser = njs_parser_function_create(vm, parser);
     if (nxt_slow_path(parser == NULL)) {
         return NJS_TOKEN_ERROR;
diff -r fe8027493a08 -r c11a7e133205 njs/njs_variable.c
--- a/njs/njs_variable.c	Tue Jan 03 19:38:17 2017 +0300
+++ b/njs/njs_variable.c	Thu Jan 05 15:55:49 2017 +0300
@@ -439,8 +439,7 @@ njs_vm_function(njs_vm_t *vm, nxt_str_t 
 
     var = lhq.value;
 
-    value = (njs_value_t *) ((u_char *) vm->global_scope
-                           + njs_offset(var->index) - NJS_INDEX_GLOBAL_OFFSET);
+    value = njs_global_variable_value(vm, var);
 
     if (njs_is_function(value)) {
         return value->data.u.function;
diff -r fe8027493a08 -r c11a7e133205 njs/njscript.c
--- a/njs/njscript.c	Tue Jan 03 19:38:17 2017 +0300
+++ b/njs/njscript.c	Thu Jan 05 15:55:49 2017 +0300
@@ -190,6 +190,7 @@ njs_vm_compile(njs_vm_t *vm, u_char **st
     nxt_int_t          ret;
     njs_lexer_t        *lexer;
     njs_parser_t       *parser;
+    njs_variable_t     *var;
     njs_parser_node_t  *node;
 
     parser = nxt_mem_cache_zalloc(vm->mem_cache_pool, sizeof(njs_parser_t));
@@ -221,7 +222,12 @@ njs_vm_compile(njs_vm_t *vm, u_char **st
 
     if (function != NULL) {
         if (node->token == NJS_TOKEN_CALL) {
-            *function = node->right->u.value.data.u.function;
+            var = njs_variable_get(vm, node->right, NJS_NAME_DECLARATION);
+            if (nxt_slow_path(var == NULL)) {
+                return NJS_ERROR;
+            }
+
+            *function = var->value.data.u.function;
 
         } else {
             *function = NULL;
@@ -239,6 +245,7 @@ njs_vm_compile(njs_vm_t *vm, u_char **st
 
     vm->global_scope = parser->local_scope;
     vm->scope_size = parser->scope_size;
+    vm->variables_hash = parser->scope->variables;
 
     vm->parser = NULL;
 

From igor at sysoev.ru  Thu Jan  5 13:19:49 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Thu, 05 Jan 2017 13:19:49 +0000
Subject: [njs] Removed unused things.
Message-ID: <hg.e4f695e81689.1483622389.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/e4f695e81689
branches:  
changeset: 299:e4f695e81689
user:      Igor Sysoev <igor at sysoev.ru>
date:      Thu Jan 05 15:55:52 2017 +0300
description:
Removed unused things.

diffstat:

 njs/njs_parser.c |  20 --------------------
 njs/njs_parser.h |   2 --
 njs/njscript.c   |   1 -
 3 files changed, 0 insertions(+), 23 deletions(-)

diffs (60 lines):

diff -r c11a7e133205 -r e4f695e81689 njs/njs_parser.c
--- a/njs/njs_parser.c	Thu Jan 05 15:55:49 2017 +0300
+++ b/njs/njs_parser.c	Thu Jan 05 15:55:52 2017 +0300
@@ -2434,26 +2434,6 @@ invalid:
 }
 
 
-njs_index_t
-njs_parser_index(njs_parser_t *parser, uint32_t scope)
-{
-    nxt_uint_t   n;
-    njs_index_t  index;
-
-    /* Skip absolute scope. */
-    n = scope - NJS_INDEX_CACHE;
-
-    index = parser->index[n];
-    parser->index[n] += sizeof(njs_value_t);
-
-    index |= scope;
-
-    nxt_thread_log_debug("GET %p", index);
-
-    return index;
-}
-
-
 nxt_bool_t
 njs_parser_has_side_effect(njs_parser_node_t *node)
 {
diff -r c11a7e133205 -r e4f695e81689 njs/njs_parser.h
--- a/njs/njs_parser.h	Thu Jan 05 15:55:49 2017 +0300
+++ b/njs/njs_parser.h	Thu Jan 05 15:55:52 2017 +0300
@@ -309,7 +309,6 @@ struct njs_parser_s {
     njs_parser_scope_t              *scope;
 
     nxt_array_t                     *index_cache;
-    njs_index_t                     index[NJS_SCOPES - NJS_INDEX_CACHE];
 
     uint8_t                         branch;       /* 1 bit */
 
@@ -354,7 +353,6 @@ njs_token_t njs_parser_property_name(njs
 njs_token_t njs_parser_property_token(njs_parser_t *parser);
 njs_token_t njs_parser_token(njs_parser_t *parser);
 nxt_int_t njs_parser_string_create(njs_vm_t *vm, njs_value_t *value);
-njs_index_t njs_parser_index(njs_parser_t *parser, uint32_t scope);
 njs_ret_t njs_variable_reference(njs_vm_t *vm, njs_parser_t *parser,
     njs_parser_node_t *node);
 njs_variable_t *njs_variable_get(njs_vm_t *vm, njs_parser_node_t *node,
diff -r c11a7e133205 -r e4f695e81689 njs/njscript.c
--- a/njs/njscript.c	Thu Jan 05 15:55:49 2017 +0300
+++ b/njs/njscript.c	Thu Jan 05 15:55:52 2017 +0300
@@ -213,7 +213,6 @@ njs_vm_compile(njs_vm_t *vm, u_char **st
 
     parser->code_size = sizeof(njs_vmcode_stop_t);
     parser->scope_offset = NJS_INDEX_GLOBAL_OFFSET;
-    parser->index[NJS_SCOPE_GLOBAL - NJS_INDEX_CACHE] = NJS_INDEX_GLOBAL_OFFSET;
 
     node = njs_parser(vm, parser);
     if (nxt_slow_path(node == NULL)) {

From vakul.garg at nxp.com  Fri Jan  6 19:10:22 2017
From: vakul.garg at nxp.com (Vakul Garg)
Date: Fri, 6 Jan 2017 19:10:22 +0000
Subject: NGINx async SSL handshake
In-Reply-To: <DE6835BE-C4CC-4061-A356-6168CCCCBC2E@gmail.com>
References: <AM3PR04MB1284A6925938F5FE45B70C6D8BB30@AM3PR04MB1284.eurprd04.prod.outlook.com>
 <DE6835BE-C4CC-4061-A356-6168CCCCBC2E@gmail.com>
Message-ID: <AM3PR04MB128446C994AB13840482B5AC8B630@AM3PR04MB1284.eurprd04.prod.outlook.com>

Hi Brian

I am using nginx openssl async: https://01.org/sites/default/files/page/nginx-1.10.0-async.l.0.3.0-001_0.tgz with my async RSA accelerator.
Where can I find the updated version of this work?

Can you share your plans to upstream this work?

Regards

Vakul


-----Original Message-----
From: nginx-devel [mailto:nginx-devel-bounces at nginx.org] On Behalf Of Alexey Ivanov
Sent: Saturday, November 19, 2016 2:00 PM
To: nginx-devel at nginx.org
Cc: brian.will at intel.com
Subject: Re: NGINx async SSL handshake

+Brian Will from Intel, to correct me if I'm wrong.

My two cents here(Intel Quick Assist specific, based on conversations during Nginx.Conf):
1) Even without hardware offload async handshake helps in cases where you have high TLS connection rates, because right now handshake is basically a 2ms+ blocking operation(specific timing depend on AVX/AVX2 support[0]) inside the event loop. Therefore after some TLS connection rate nginx performance falls off the cliff.
2) Hardware offload numbers look very impressive[1](TL;DR: 5x improvement for RSA 2048, for ECDSA, imho, it is neither impressive, nor needed). Also they prove asymmetric part of the accelerator to be future proof, so that it is possible to add new handshake types(e.g. Ed25519). Disclaimer: we did not test that hardware yet.

As for patches, you can check 01.org for:
a) nginx openssl async: https://01.org/sites/default/files/page/nginx-1.10.0-async.l.0.3.0-001_0.tgz
b) zlib[2]: https://01.org/sites/default/files/page/zlib_shim_0.4.9-001.tgz
(Full list of docs: https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches )

Question for Brian/Maxim: are you planning on integrating it into mainline nginx? 1000+ line diffs are usually rather hard to integrate.


[0] FWIW, speaking about OpenSSL performance: using OpenSSL 1.0.2 + Intel Xeon v2 processors with AVX2 gives 2x performance boost(over OpenSSL 1.0.1 and v1).
[1] https://twitter.com/SaveTheRbtz/status/773962669166428161
[2] There are also cloudflare and intel patches for zlib for faster deflation (i.e. compression only)
> On Nov 18, 2016, at 8:12 PM, Vakul Garg <vakul.garg at nxp.com> wrote:
> 
> Hi
> 
> I am a newbie to nginx.
> I am integrating a public key hardware accelerator in OpenSSL using engine interface.
> The engine is async capable.
> 
> Recently openssl-1.1.0 has added support for ASYNC_JOB.
> IIUC, nginx would also require changes in order to do SSL handshake in async way.
> 
> Any pointers where can I get the nginx code changes done for async openssl
> 
> Regards
> 
> Vakul
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel


From vakul.garg at nxp.com  Mon Jan  9 07:51:35 2017
From: vakul.garg at nxp.com (Vakul Garg)
Date: Mon, 9 Jan 2017 07:51:35 +0000
Subject: NGINx async SSL handshake
In-Reply-To: <D1A3A80A1B97944385C8C290B1BB02D05C53D0DB@fmsmsx118.amr.corp.intel.com>
References: <AM3PR04MB1284A6925938F5FE45B70C6D8BB30@AM3PR04MB1284.eurprd04.prod.outlook.com>
 <DE6835BE-C4CC-4061-A356-6168CCCCBC2E@gmail.com>
 <AM3PR04MB128446C994AB13840482B5AC8B630@AM3PR04MB1284.eurprd04.prod.outlook.com>
 <D1A3A80A1B97944385C8C290B1BB02D05C53D0DB@fmsmsx118.amr.corp.intel.com>
Message-ID: <HE1PR04MB129169B4E569A1CCD818F1A28B640@HE1PR04MB1291.eurprd04.prod.outlook.com>

Hi Brian

We are using your nginx patch with our PKC accelerator (exposed through cryptodev-linux ioctls).
The OpenSSL cryptodev engine has been enhanced to make async PKC calls.

The system is running functionally well, but the performance is poor.

Last year, we used older OpenSSL versions (without ASYNC job infrastructure) with Geoff Thorpe's https://github.com/libfibre to achieve async crypto acceleration.
The performance was better than we are getting with OpenSSL-1.1.0 + Nginx patch in question.

On studying the Intel nginx changes, I feel that the way the asynchronous APIs have been integrated are somewhat heavy and there is possibility of improvement.
Specifically, the eventfd based job wakeup mechanism seems heavy.

I am looking towards making changes over your nginx patch to introduce a newer job wakeup method.
Further, I want to get rid of creating set of accelerator response poll threads in engine and provide an option to do this inline in nginx worker thread itself.

For this, I hoped your patches are submitted upstream and in process of review 
or 
I can find your work in some github repo where we can submit our changes.

Let me know your comments.

Regards

Vakul

-----Original Message-----
From: Will, Brian [mailto:brian.will at intel.com] 
Sent: Saturday, January 07, 2017 2:40 AM
To: Vakul Garg <vakul.garg at nxp.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>; Schuetze, Joel D <joel.d.schuetze at intel.com>; Yu, Ping Y <ping.y.yu at intel.com>; Finn, Coleman <coleman.finn at intel.com>
Subject: RE: NGINx async SSL handshake

Hello Vakul,
The link you have included is the latest patch for NGINX to support OpenSSL-1.1.0 that we are maintaining. We are working to try and get these changes included into the mainline of NGINX, but in the interim this patch should work for the OpenSSL-1.1.0 branch of code.
Were there specific issues that you have run into?

Thanks,
Brian

-----Original Message-----
From: Vakul Garg [mailto:vakul.garg at nxp.com] 
Sent: Friday, January 06, 2017 12:10 PM
To: Will, Brian <brian.will at intel.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>
Subject: RE: NGINx async SSL handshake

Hi Brian

I am using nginx openssl async: https://01.org/sites/default/files/page/nginx-1.10.0-async.l.0.3.0-001_0.tgz with my async RSA accelerator.
Where can I find the updated version of this work?

Can you share your plans to upstream this work?

Regards

Vakul


-----Original Message-----
From: nginx-devel [mailto:nginx-devel-bounces at nginx.org] On Behalf Of Alexey Ivanov
Sent: Saturday, November 19, 2016 2:00 PM
To: nginx-devel at nginx.org
Cc: brian.will at intel.com
Subject: Re: NGINx async SSL handshake

+Brian Will from Intel, to correct me if I'm wrong.

My two cents here(Intel Quick Assist specific, based on conversations during Nginx.Conf):
1) Even without hardware offload async handshake helps in cases where you have high TLS connection rates, because right now handshake is basically a 2ms+ blocking operation(specific timing depend on AVX/AVX2 support[0]) inside the event loop. Therefore after some TLS connection rate nginx performance falls off the cliff.
2) Hardware offload numbers look very impressive[1](TL;DR: 5x improvement for RSA 2048, for ECDSA, imho, it is neither impressive, nor needed). Also they prove asymmetric part of the accelerator to be future proof, so that it is possible to add new handshake types(e.g. Ed25519). Disclaimer: we did not test that hardware yet.

As for patches, you can check 01.org for:
a) nginx openssl async: https://01.org/sites/default/files/page/nginx-1.10.0-async.l.0.3.0-001_0.tgz
b) zlib[2]: https://01.org/sites/default/files/page/zlib_shim_0.4.9-001.tgz
(Full list of docs: https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches )

Question for Brian/Maxim: are you planning on integrating it into mainline nginx? 1000+ line diffs are usually rather hard to integrate.


[0] FWIW, speaking about OpenSSL performance: using OpenSSL 1.0.2 + Intel Xeon v2 processors with AVX2 gives 2x performance boost(over OpenSSL 1.0.1 and v1).
[1] https://twitter.com/SaveTheRbtz/status/773962669166428161
[2] There are also cloudflare and intel patches for zlib for faster deflation (i.e. compression only)
> On Nov 18, 2016, at 8:12 PM, Vakul Garg <vakul.garg at nxp.com> wrote:
> 
> Hi
> 
> I am a newbie to nginx.
> I am integrating a public key hardware accelerator in OpenSSL using engine interface.
> The engine is async capable.
> 
> Recently openssl-1.1.0 has added support for ASYNC_JOB.
> IIUC, nginx would also require changes in order to do SSL handshake in async way.
> 
> Any pointers where can I get the nginx code changes done for async openssl
> 
> Regards
> 
> Vakul
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel


From ru at nginx.com  Tue Jan 10 14:13:41 2017
From: ru at nginx.com (Ruslan Ermilov)
Date: Tue, 10 Jan 2017 14:13:41 +0000
Subject: [nginx] Version bump.
Message-ID: <hg.0a268ae349ab.1484057621.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/0a268ae349ab
branches:  
changeset: 6866:0a268ae349ab
user:      Ruslan Ermilov <ru at nginx.com>
date:      Tue Jan 10 17:13:01 2017 +0300
description:
Version bump.

diffstat:

 src/core/nginx.h |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff -r a42afc225e98 -r 0a268ae349ab src/core/nginx.h
--- a/src/core/nginx.h	Tue Dec 27 17:23:08 2016 +0300
+++ b/src/core/nginx.h	Tue Jan 10 17:13:01 2017 +0300
@@ -9,8 +9,8 @@
 #define _NGINX_H_INCLUDED_
 
 
-#define nginx_version      1011008
-#define NGINX_VERSION      "1.11.8"
+#define nginx_version      1011009
+#define NGINX_VERSION      "1.11.9"
 #define NGINX_VER          "nginx/" NGINX_VERSION
 
 #ifdef NGX_BUILD

From ru at nginx.com  Tue Jan 10 14:13:44 2017
From: ru at nginx.com (Ruslan Ermilov)
Date: Tue, 10 Jan 2017 14:13:44 +0000
Subject: [nginx] Year 2017.
Message-ID: <hg.8c4d07ef08f5.1484057624.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/8c4d07ef08f5
branches:  
changeset: 6867:8c4d07ef08f5
user:      Ruslan Ermilov <ru at nginx.com>
date:      Tue Jan 10 17:13:06 2017 +0300
description:
Year 2017.

diffstat:

 docs/text/LICENSE |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (12 lines):

diff -r 0a268ae349ab -r 8c4d07ef08f5 docs/text/LICENSE
--- a/docs/text/LICENSE	Tue Jan 10 17:13:01 2017 +0300
+++ b/docs/text/LICENSE	Tue Jan 10 17:13:06 2017 +0300
@@ -1,6 +1,6 @@
 /* 
- * Copyright (C) 2002-2016 Igor Sysoev
- * Copyright (C) 2011-2016 Nginx, Inc.
+ * Copyright (C) 2002-2017 Igor Sysoev
+ * Copyright (C) 2011-2017 Nginx, Inc.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without

From erahn at arista.com  Tue Jan 10 23:41:14 2017
From: erahn at arista.com (Ethan Rahn)
Date: Tue, 10 Jan 2017 15:41:14 -0800
Subject: How to contribute fix for checking x509 extended key attrs to nginx?
Message-ID: <CAMk6QYMY4OB7-zFPbMp_vbnUiMHiEBLkDxb6uAGRnRhd9YUSig@mail.gmail.com>

Hello,

I noticed that nginx does not check x509v3 certificates ( in
event/ngx_event_openssl.c::ngx_ssl_get_client_verify as an example ) to see
that the optional extended key usage settings are correct. I have a patch
for this that I would like to contribute, but I'm unable to find
contribution guidelines on the nginx web-site.

The effect of this issue is that someone could offer a client certificate
that has extended key usage set to say, serverAuth. This would be a
violation of RFC 5280 - Section 4.2.1.12. I fix this by checking the
bitfield manually to see that the settings are correct.

Cheers,

Ethan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170110/833c9d05/attachment.html>

From savetherbtz at gmail.com  Wed Jan 11 02:58:21 2017
From: savetherbtz at gmail.com (Alexey Ivanov)
Date: Tue, 10 Jan 2017 18:58:21 -0800
Subject: How to contribute fix for checking x509 extended key attrs to
 nginx?
In-Reply-To: <CAMk6QYMY4OB7-zFPbMp_vbnUiMHiEBLkDxb6uAGRnRhd9YUSig@mail.gmail.com>
References: <CAMk6QYMY4OB7-zFPbMp_vbnUiMHiEBLkDxb6uAGRnRhd9YUSig@mail.gmail.com>
Message-ID: <91A247D3-E508-4B7B-93C9-4CC6CCD9282A@gmail.com>

On Jan 10, 2017, at 3:41 PM, Ethan Rahn via nginx-devel <nginx-devel at nginx.org> wrote:
> 
> Hello,
> 
> I noticed that nginx does not check x509v3 certificates ( in event/ngx_event_openssl.c::ngx_ssl_get_client_verify as an example ) to see that the optional extended key usage settings are correct. I have a patch for this that I would like to contribute, but I'm unable to find contribution guidelines on the nginx web-site.
http://nginx.org/en/docs/contributing_changes.html

> The effect of this issue is that someone could offer a client certificate that has extended key usage set to say, serverAuth. This would be a violation of RFC 5280 - Section 4.2.1.12. I fix this by checking the bitfield manually to see that the settings are correct.
> 
> Cheers,
> 
> Ethan
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170110/63d4bde8/attachment.bin>

From mdounin at mdounin.ru  Wed Jan 11 14:02:12 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 11 Jan 2017 17:02:12 +0300
Subject: How to contribute fix for checking x509 extended key attrs to
 nginx?
In-Reply-To: <CAMk6QYMY4OB7-zFPbMp_vbnUiMHiEBLkDxb6uAGRnRhd9YUSig@mail.gmail.com>
References: <CAMk6QYMY4OB7-zFPbMp_vbnUiMHiEBLkDxb6uAGRnRhd9YUSig@mail.gmail.com>
Message-ID: <20170111140212.GF47718@mdounin.ru>

Hello!

On Tue, Jan 10, 2017 at 03:41:14PM -0800, Ethan Rahn via nginx-devel wrote:

> Hello,
> 
> I noticed that nginx does not check x509v3 certificates ( in
> event/ngx_event_openssl.c::ngx_ssl_get_client_verify as an example ) to see
> that the optional extended key usage settings are correct. I have a patch
> for this that I would like to contribute, but I'm unable to find
> contribution guidelines on the nginx web-site.
> 
> The effect of this issue is that someone could offer a client certificate
> that has extended key usage set to say, serverAuth. This would be a
> violation of RFC 5280 - Section 4.2.1.12. I fix this by checking the
> bitfield manually to see that the settings are correct.

Note that nginx relies on OpenSSL to verify certificates, and 
checking things manually might not be a good idea.  If you think 
that somthing is missing, a better solution might be to improve 
OpenSSL checking instead.

-- 
Maxim Dounin
http://nginx.org/

From Jakub.Moscicki at cern.ch  Wed Jan 11 15:03:34 2017
From: Jakub.Moscicki at cern.ch (Jakub Moscicki)
Date: Wed, 11 Jan 2017 15:03:34 +0000
Subject: internal X-Accel-Redirect for PUT
Message-ID: <18BDB0AF-9537-49A5-99D7-3614AF1E069D@cern.ch>

Hello,

Was there a change of behaviour as per how PUT requests are redirected via X-Accel-Redirect between 1.6.2 and 1.10.2 versions?

In nginx 1.6.2 the PUT request from the client is internally redirected as PUT via X-Accel-Redirect. In 1.10.2 it is redirected as GET. So the method name is changes when the redirect happens.

The redirect is between two upstream servers (and not the client) via an internal location.

Is there a way to achieve 1.6.2 behaviour?

Here is the diagram to clarify what happens:


[cid:B0590114-1D8A-4D13-998D-7AB0FBE2FA63 at cern.ch]




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170111/f2b63682/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 39902 bytes
Desc: PastedGraphic-1.png
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170111/f2b63682/attachment-0001.png>

From mdounin at mdounin.ru  Wed Jan 11 16:15:59 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 11 Jan 2017 19:15:59 +0300
Subject: internal X-Accel-Redirect for PUT
In-Reply-To: <18BDB0AF-9537-49A5-99D7-3614AF1E069D@cern.ch>
References: <18BDB0AF-9537-49A5-99D7-3614AF1E069D@cern.ch>
Message-ID: <20170111161559.GI47718@mdounin.ru>

Hello!

On Wed, Jan 11, 2017 at 03:03:34PM +0000, Jakub Moscicki wrote:

> Was there a change of behaviour as per how PUT requests are 
> redirected via X-Accel-Redirect between 1.6.2 and 1.10.2 
> versions?
> 
> In nginx 1.6.2 the PUT request from the client is internally 
> redirected as PUT via X-Accel-Redirect. In 1.10.2 it is 
> redirected as GET. So the method name is changes when the 
> redirect happens.
> 
> The redirect is between two upstream servers (and not the 
> client) via an internal location.
> 
> Is there a way to achieve 1.6.2 behaviour?

The previous behaviour was a bug, as nginx interpreted 
X-Accel-Redirect'ed requests as GET internally, but still used 
original request method in requests to upstream servers.  This was 
fixed in 1.9.10:

    *) Bugfix: proxying used the HTTP method of the original request after
       an "X-Accel-Redirect" redirection.

If you want to preserve original request method, use 
X-Accel-Redirect to a named location.  Much like in case of 
error_page (http://nginx.org/r/error_page), this will preserve URI 
and request method during redirection.

-- 
Maxim Dounin
http://nginx.org/

From vl at nginx.com  Wed Jan 11 18:18:14 2017
From: vl at nginx.com (Vladimir Homutov)
Date: Wed, 11 Jan 2017 18:18:14 +0000
Subject: [nginx] Stream: avoid infinite loop in case of socket read error.
Message-ID: <hg.ee3645078759.1484158694.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/ee3645078759
branches:  
changeset: 6868:ee3645078759
user:      Vladimir Homutov <vl at nginx.com>
date:      Wed Jan 11 12:01:56 2017 +0300
description:
Stream: avoid infinite loop in case of socket read error.

diffstat:

 src/stream/ngx_stream_proxy_module.c |  5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diffs (15 lines):

diff -r 8c4d07ef08f5 -r ee3645078759 src/stream/ngx_stream_proxy_module.c
--- a/src/stream/ngx_stream_proxy_module.c	Tue Jan 10 17:13:06 2017 +0300
+++ b/src/stream/ngx_stream_proxy_module.c	Wed Jan 11 12:01:56 2017 +0300
@@ -1534,8 +1534,9 @@ ngx_stream_proxy_process(ngx_stream_sess
 
         size = b->end - b->last;
 
-        if (size && src->read->ready && !src->read->delayed) {
-
+        if (size && src->read->ready && !src->read->delayed
+            && !src->read->error)
+        {
             if (limit_rate) {
                 limit = (off_t) limit_rate * (ngx_time() - u->start_sec + 1)
                         - *received;

From selivan5 at yandex.ru  Wed Jan 11 18:31:14 2017
From: selivan5 at yandex.ru (selivan)
Date: Wed, 11 Jan 2017 21:31:14 +0300
Subject: Proposed documentation change for ticket #1176
Message-ID: <2065561484159474@web5m.yandex.ru>

https://trac.nginx.org/nginx/ticket/1176

Specified at compile time default error log(--error-log-path) is always used at global level if there are no other file-based logs configured. I think this should be mentioned in documentation.

Patch in attachment.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: docs.patch
Type: text/x-diff
Size: 4569 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170111/51f50ed5/attachment.bin>

From pluknet at nginx.com  Thu Jan 12 16:26:28 2017
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Thu, 12 Jan 2017 16:26:28 +0000
Subject: [nginx] Mail: make it possible to disable SASL EXTERNAL.
Message-ID: <hg.b2915d99ee8d.1484238388.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/b2915d99ee8d
branches:  
changeset: 6869:b2915d99ee8d
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Thu Jan 12 19:22:03 2017 +0300
description:
Mail: make it possible to disable SASL EXTERNAL.

diffstat:

 src/mail/ngx_mail_imap_handler.c |  8 ++++++--
 src/mail/ngx_mail_pop3_handler.c |  4 ++++
 src/mail/ngx_mail_smtp_handler.c |  8 ++++++--
 3 files changed, 16 insertions(+), 4 deletions(-)

diffs (78 lines):

diff -r ee3645078759 -r b2915d99ee8d src/mail/ngx_mail_imap_handler.c
--- a/src/mail/ngx_mail_imap_handler.c	Wed Jan 11 12:01:56 2017 +0300
+++ b/src/mail/ngx_mail_imap_handler.c	Thu Jan 12 19:22:03 2017 +0300
@@ -356,6 +356,8 @@ ngx_mail_imap_authenticate(ngx_mail_sess
     }
 #endif
 
+    iscf = ngx_mail_get_module_srv_conf(s, ngx_mail_imap_module);
+
     rc = ngx_mail_auth_parse(s, c);
 
     switch (rc) {
@@ -383,8 +385,6 @@ ngx_mail_imap_authenticate(ngx_mail_sess
 
     case NGX_MAIL_AUTH_CRAM_MD5:
 
-        iscf = ngx_mail_get_module_srv_conf(s, ngx_mail_imap_module);
-
         if (!(iscf->auth_methods & NGX_MAIL_AUTH_CRAM_MD5_ENABLED)) {
             return NGX_MAIL_PARSE_INVALID_COMMAND;
         }
@@ -406,6 +406,10 @@ ngx_mail_imap_authenticate(ngx_mail_sess
 
     case NGX_MAIL_AUTH_EXTERNAL:
 
+        if (!(iscf->auth_methods & NGX_MAIL_AUTH_EXTERNAL_ENABLED)) {
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
         ngx_str_set(&s->out, imap_username);
         s->mail_state = ngx_imap_auth_external;
 
diff -r ee3645078759 -r b2915d99ee8d src/mail/ngx_mail_pop3_handler.c
--- a/src/mail/ngx_mail_pop3_handler.c	Wed Jan 11 12:01:56 2017 +0300
+++ b/src/mail/ngx_mail_pop3_handler.c	Thu Jan 12 19:22:03 2017 +0300
@@ -501,6 +501,10 @@ ngx_mail_pop3_auth(ngx_mail_session_t *s
 
     case NGX_MAIL_AUTH_EXTERNAL:
 
+        if (!(pscf->auth_methods & NGX_MAIL_AUTH_EXTERNAL_ENABLED)) {
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
         ngx_str_set(&s->out, pop3_username);
         s->mail_state = ngx_pop3_auth_external;
 
diff -r ee3645078759 -r b2915d99ee8d src/mail/ngx_mail_smtp_handler.c
--- a/src/mail/ngx_mail_smtp_handler.c	Wed Jan 11 12:01:56 2017 +0300
+++ b/src/mail/ngx_mail_smtp_handler.c	Thu Jan 12 19:22:03 2017 +0300
@@ -609,6 +609,8 @@ ngx_mail_smtp_auth(ngx_mail_session_t *s
         return NGX_OK;
     }
 
+    sscf = ngx_mail_get_module_srv_conf(s, ngx_mail_smtp_module);
+
     rc = ngx_mail_auth_parse(s, c);
 
     switch (rc) {
@@ -636,8 +638,6 @@ ngx_mail_smtp_auth(ngx_mail_session_t *s
 
     case NGX_MAIL_AUTH_CRAM_MD5:
 
-        sscf = ngx_mail_get_module_srv_conf(s, ngx_mail_smtp_module);
-
         if (!(sscf->auth_methods & NGX_MAIL_AUTH_CRAM_MD5_ENABLED)) {
             return NGX_MAIL_PARSE_INVALID_COMMAND;
         }
@@ -659,6 +659,10 @@ ngx_mail_smtp_auth(ngx_mail_session_t *s
 
     case NGX_MAIL_AUTH_EXTERNAL:
 
+        if (!(sscf->auth_methods & NGX_MAIL_AUTH_EXTERNAL_ENABLED)) {
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
         ngx_str_set(&s->out, smtp_username);
         s->mail_state = ngx_smtp_auth_external;
 

From vakul.garg at nxp.com  Fri Jan 13 08:51:40 2017
From: vakul.garg at nxp.com (Vakul Garg)
Date: Fri, 13 Jan 2017 08:51:40 +0000
Subject: NGINx async SSL handshake
In-Reply-To: <D1A3A80A1B97944385C8C290B1BB02D05C53EF78@fmsmsx118.amr.corp.intel.com>
References: <AM3PR04MB1284A6925938F5FE45B70C6D8BB30@AM3PR04MB1284.eurprd04.prod.outlook.com>
 <DE6835BE-C4CC-4061-A356-6168CCCCBC2E@gmail.com>
 <AM3PR04MB128446C994AB13840482B5AC8B630@AM3PR04MB1284.eurprd04.prod.outlook.com>
 <D1A3A80A1B97944385C8C290B1BB02D05C53D0DB@fmsmsx118.amr.corp.intel.com>
 <HE1PR04MB129169B4E569A1CCD818F1A28B640@HE1PR04MB1291.eurprd04.prod.outlook.com>
 <D1A3A80A1B97944385C8C290B1BB02D05C53EF78@fmsmsx118.amr.corp.intel.com>
Message-ID: <AM3PR04MB128485EC217B88ED2DF1B6008B780@AM3PR04MB1284.eurprd04.prod.outlook.com>

Hi Brian

Yes, one of the things I am looking for is a replacement for engine private receive thread with "poll inline" from nginx worker itself.
It might not necessarily require timer based mechanism to get the nginx event loop to poll accelerator responses.

The requirement is to give some time slice to the engine to do receive side processing from the accelerator.
This time slice can be given after completing ready list of FDs in event loop.

If any SSL connection is pending on WANT_ASYNC, then event poll loop should wakeup with zero timeout, because we know accelerator response would come soon and it makes sense to give timeslice for engine accelerator response processing.
For engine accelerator response processing an ENGINE control api can be built and directly called from nginx event loop.

Please let me know your comments.

Regards

Vakul

-----Original Message-----
From: Will, Brian [mailto:brian.will at intel.com] 
Sent: Friday, January 13, 2017 4:54 AM
To: Vakul Garg <vakul.garg at nxp.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>; Schuetze, Joel D <joel.d.schuetze at intel.com>; Yu, Ping Y <ping.y.yu at intel.com>; Finn, Coleman <coleman.finn at intel.com>; Pankaj Gupta <pankaj.gupta at nxp.com>; Arun Pathak <arun.pathak at nxp.com>; Chambers, Mark A <mark.a.chambers at intel.com>
Subject: RE: NGINx async SSL handshake

Hi Vakul,
Sorry for not getting back sooner. We are currently in the process of working with NGINX to get our patch integrated into the development tree. This process may result in a very different patch to what we have currently published, resulting in closer integration and fit into the infrastructure. As of right now we do not have the patch posted either as a repo or a review request. 

Your work on the wakeup mechanism would be very interesting, specifically what are you looking at as a replacement for the activation thread which the engine would own? On the option to "poll inline" from the nginx worker itself this is something we had explored and does result in increased performance, but then you would require some timer based mechanism to get the NGINX event loop to poll the SSL* API in question. This can be added and is supported through the current OpenSSL async features, but seemed like a different processing model to the way NGINX handles all other activity.

Thanks,
Brian

-----Original Message-----
From: Vakul Garg [mailto:vakul.garg at nxp.com]
Sent: Monday, January 09, 2017 12:52 AM
To: Will, Brian <brian.will at intel.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>; Schuetze, Joel D <joel.d.schuetze at intel.com>; Yu, Ping Y <ping.y.yu at intel.com>; Finn, Coleman <coleman.finn at intel.com>; Pankaj Gupta <pankaj.gupta at nxp.com>; Arun Pathak <arun.pathak at nxp.com>
Subject: RE: NGINx async SSL handshake

Hi Brian

We are using your nginx patch with our PKC accelerator (exposed through cryptodev-linux ioctls).
The OpenSSL cryptodev engine has been enhanced to make async PKC calls.

The system is running functionally well, but the performance is poor.

Last year, we used older OpenSSL versions (without ASYNC job infrastructure) with Geoff Thorpe's https://github.com/libfibre to achieve async crypto acceleration.
The performance was better than we are getting with OpenSSL-1.1.0 + Nginx patch in question.

On studying the Intel nginx changes, I feel that the way the asynchronous APIs have been integrated are somewhat heavy and there is possibility of improvement.
Specifically, the eventfd based job wakeup mechanism seems heavy.

I am looking towards making changes over your nginx patch to introduce a newer job wakeup method.
Further, I want to get rid of creating set of accelerator response poll threads in engine and provide an option to do this inline in nginx worker thread itself.

For this, I hoped your patches are submitted upstream and in process of review or I can find your work in some github repo where we can submit our changes.

Let me know your comments.

Regards

Vakul

-----Original Message-----
From: Will, Brian [mailto:brian.will at intel.com]
Sent: Saturday, January 07, 2017 2:40 AM
To: Vakul Garg <vakul.garg at nxp.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>; Schuetze, Joel D <joel.d.schuetze at intel.com>; Yu, Ping Y <ping.y.yu at intel.com>; Finn, Coleman <coleman.finn at intel.com>
Subject: RE: NGINx async SSL handshake

Hello Vakul,
The link you have included is the latest patch for NGINX to support OpenSSL-1.1.0 that we are maintaining. We are working to try and get these changes included into the mainline of NGINX, but in the interim this patch should work for the OpenSSL-1.1.0 branch of code.
Were there specific issues that you have run into?

Thanks,
Brian

-----Original Message-----
From: Vakul Garg [mailto:vakul.garg at nxp.com]
Sent: Friday, January 06, 2017 12:10 PM
To: Will, Brian <brian.will at intel.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>
Subject: RE: NGINx async SSL handshake

Hi Brian

I am using nginx openssl async: https://01.org/sites/default/files/page/nginx-1.10.0-async.l.0.3.0-001_0.tgz with my async RSA accelerator.
Where can I find the updated version of this work?

Can you share your plans to upstream this work?

Regards

Vakul


-----Original Message-----
From: nginx-devel [mailto:nginx-devel-bounces at nginx.org] On Behalf Of Alexey Ivanov
Sent: Saturday, November 19, 2016 2:00 PM
To: nginx-devel at nginx.org
Cc: brian.will at intel.com
Subject: Re: NGINx async SSL handshake

+Brian Will from Intel, to correct me if I'm wrong.

My two cents here(Intel Quick Assist specific, based on conversations during Nginx.Conf):
1) Even without hardware offload async handshake helps in cases where you have high TLS connection rates, because right now handshake is basically a 2ms+ blocking operation(specific timing depend on AVX/AVX2 support[0]) inside the event loop. Therefore after some TLS connection rate nginx performance falls off the cliff.
2) Hardware offload numbers look very impressive[1](TL;DR: 5x improvement for RSA 2048, for ECDSA, imho, it is neither impressive, nor needed). Also they prove asymmetric part of the accelerator to be future proof, so that it is possible to add new handshake types(e.g. Ed25519). Disclaimer: we did not test that hardware yet.

As for patches, you can check 01.org for:
a) nginx openssl async: https://01.org/sites/default/files/page/nginx-1.10.0-async.l.0.3.0-001_0.tgz
b) zlib[2]: https://01.org/sites/default/files/page/zlib_shim_0.4.9-001.tgz
(Full list of docs: https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches )

Question for Brian/Maxim: are you planning on integrating it into mainline nginx? 1000+ line diffs are usually rather hard to integrate.


[0] FWIW, speaking about OpenSSL performance: using OpenSSL 1.0.2 + Intel Xeon v2 processors with AVX2 gives 2x performance boost(over OpenSSL 1.0.1 and v1).
[1] https://twitter.com/SaveTheRbtz/status/773962669166428161
[2] There are also cloudflare and intel patches for zlib for faster deflation (i.e. compression only)
> On Nov 18, 2016, at 8:12 PM, Vakul Garg <vakul.garg at nxp.com> wrote:
> 
> Hi
> 
> I am a newbie to nginx.
> I am integrating a public key hardware accelerator in OpenSSL using engine interface.
> The engine is async capable.
> 
> Recently openssl-1.1.0 has added support for ASYNC_JOB.
> IIUC, nginx would also require changes in order to do SSL handshake in async way.
> 
> Any pointers where can I get the nginx code changes done for async 
> openssl
> 
> Regards
> 
> Vakul
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel


From xingwu.yang at gmail.com  Fri Jan 13 08:53:53 2017
From: xingwu.yang at gmail.com (yangxw)
Date: Fri, 13 Jan 2017 16:53:53 +0800
Subject: "exists" filed of ngx_shm_t
Message-ID: <CA+7U5Jsc=iMdjOnoQaHBT3kBDQYrWW2fYZ77dkrm0mu-FqP3xg@mail.gmail.com>

Hi,

I search the source codes, it seems that no one would ever to set the value
of 'exists' filed of ngx_shm_t.

is it deprecated now?

thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170113/c1d8d962/attachment.html>

From vbart at nginx.com  Fri Jan 13 11:55:07 2017
From: vbart at nginx.com (Valentin V. Bartenev)
Date: Fri, 13 Jan 2017 14:55:07 +0300
Subject: "exists" filed of ngx_shm_t
In-Reply-To: <CA+7U5Jsc=iMdjOnoQaHBT3kBDQYrWW2fYZ77dkrm0mu-FqP3xg@mail.gmail.com>
References: <CA+7U5Jsc=iMdjOnoQaHBT3kBDQYrWW2fYZ77dkrm0mu-FqP3xg@mail.gmail.com>
Message-ID: <2891108.SP0iVrNL4G@vbart-workstation>

On Friday 13 January 2017 16:53:53 yangxw wrote:
> Hi,
> 
> I search the source codes, it seems that no one would ever to set the value
> of 'exists' filed of ngx_shm_t.
> 
> is it deprecated now?
> 
> thanks.

http://hg.nginx.org/nginx/file/tip/src/os/win32/ngx_shmem.c#l79

  wbr, Valentin V. Bartenev


From igor at sysoev.ru  Mon Jan 16 12:00:54 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Mon, 16 Jan 2017 12:00:54 +0000
Subject: [njs] Fixed parsing semicolons.
Message-ID: <hg.07dc20c1206d.1484568054.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/07dc20c1206d
branches:  
changeset: 300:07dc20c1206d
user:      Igor Sysoev <igor at sysoev.ru>
date:      Mon Jan 16 14:51:29 2017 +0300
description:
Fixed parsing semicolons.

Found with AddressSanitizer.

diffstat:

 njs/njs_parser.c |  26 ++++++++++++++++++--------
 1 files changed, 18 insertions(+), 8 deletions(-)

diffs (76 lines):

diff -r e4f695e81689 -r 07dc20c1206d njs/njs_parser.c
--- a/njs/njs_parser.c	Thu Jan 05 15:55:52 2017 +0300
+++ b/njs/njs_parser.c	Mon Jan 16 14:51:29 2017 +0300
@@ -229,11 +229,9 @@ njs_parser_statement_chain(njs_vm_t *vm,
 
     if (nxt_fast_path(token > NJS_TOKEN_ILLEGAL)) {
 
-        if (parser->node != last) {
-            /*
-             * The statement is not empty block, not just semicolon,
-             * and not variables declaration without initialization.
-             */
+        if (parser->node != NULL) {
+            /* The statement is not empty block or just semicolon. */
+
             node = njs_parser_node_alloc(vm);
             if (nxt_slow_path(node == NULL)) {
                 return NJS_TOKEN_ERROR;
@@ -243,6 +241,15 @@ njs_parser_statement_chain(njs_vm_t *vm,
             node->left = last;
             node->right = parser->node;
             parser->node = node;
+
+            while (token == NJS_TOKEN_SEMICOLON
+                   || token == NJS_TOKEN_LINE_END)
+            {
+                token = njs_parser_token(parser);
+                if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
+                    break;
+                }
+            }
         }
 
     } else if (vm->exception == NULL) {
@@ -257,6 +264,8 @@ static njs_token_t
 njs_parser_statement(njs_vm_t *vm, njs_parser_t *parser,
     njs_token_t token)
 {
+    parser->node = NULL;
+
     switch (token) {
 
     case NJS_TOKEN_FUNCTION:
@@ -296,7 +305,6 @@ njs_parser_statement(njs_vm_t *vm, njs_p
         return njs_parser_throw_statement(vm, parser);
 
     case NJS_TOKEN_SEMICOLON:
-        parser->node = NULL;
         return njs_parser_token(parser);
 
     case NJS_TOKEN_OPEN_BRACE:
@@ -348,13 +356,13 @@ njs_parser_block_statement(njs_vm_t *vm,
         return token;
     }
 
-    parser->node = NULL;
-
     ret = njs_parser_scope_begin(vm, parser, NJS_SCOPE_BLOCK);
     if (nxt_slow_path(ret != NXT_OK)) {
         return NJS_TOKEN_ERROR;
     }
 
+    parser->node = NULL;
+
     while (token != NJS_TOKEN_CLOSE_BRACE) {
         token = njs_parser_statement_chain(vm, parser, token);
         if (nxt_slow_path(token <= NJS_TOKEN_ILLEGAL)) {
@@ -836,6 +844,8 @@ njs_parser_var_statement(njs_vm_t *vm, n
     switch (token) {
 
     case NJS_TOKEN_SEMICOLON:
+        return njs_parser_token(parser);
+
     case NJS_TOKEN_CLOSE_BRACE:
     case NJS_TOKEN_END:
         return token;

From igor at sysoev.ru  Mon Jan 16 14:51:53 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Mon, 16 Jan 2017 14:51:53 +0000
Subject: [njs] Comprehensive test of "return" statement location.
Message-ID: <hg.6cc16ad934a4.1484578313.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/6cc16ad934a4
branches:  
changeset: 301:6cc16ad934a4
user:      Igor Sysoev <igor at sysoev.ru>
date:      Mon Jan 16 17:32:10 2017 +0300
description:
Comprehensive test of "return" statement location.

Found with afl-fuzz.

diffstat:

 njs/njs_parser.c         |  22 ++++++++++++++--------
 njs/test/njs_unit_test.c |   3 +++
 2 files changed, 17 insertions(+), 8 deletions(-)

diffs (45 lines):

diff -r 07dc20c1206d -r 6cc16ad934a4 njs/njs_parser.c
--- a/njs/njs_parser.c	Mon Jan 16 14:51:29 2017 +0300
+++ b/njs/njs_parser.c	Mon Jan 16 17:32:10 2017 +0300
@@ -697,14 +697,20 @@ njs_parser_function_lambda(njs_vm_t *vm,
 static njs_token_t
 njs_parser_return_statement(njs_vm_t *vm, njs_parser_t *parser)
 {
-    njs_token_t        token;
-    njs_parser_node_t  *node;
-
-    if (parser->scope->type == NJS_SCOPE_GLOBAL) {
-        nxt_alert(&vm->trace, NXT_LEVEL_ERROR,
-                  "SyntaxError: Illegal return statement");
-
-        return NXT_ERROR;
+    njs_token_t         token;
+    njs_parser_node_t   *node;
+    njs_parser_scope_t  *scope;
+
+    for (scope = parser->scope;
+         scope->type != NJS_SCOPE_FUNCTION;
+         scope = scope->parent)
+    {
+        if (scope->type == NJS_SCOPE_GLOBAL) {
+            nxt_alert(&vm->trace, NXT_LEVEL_ERROR,
+                      "SyntaxError: Illegal return statement");
+
+            return NXT_ERROR;
+        }
     }
 
     node = njs_parser_node_alloc(vm);
diff -r 07dc20c1206d -r 6cc16ad934a4 njs/test/njs_unit_test.c
--- a/njs/test/njs_unit_test.c	Mon Jan 16 14:51:29 2017 +0300
+++ b/njs/test/njs_unit_test.c	Mon Jan 16 17:32:10 2017 +0300
@@ -3976,6 +3976,9 @@ static njs_unit_test_t  njs_test[] =
     { nxt_string("return"),
       nxt_string("SyntaxError: Illegal return statement in 1") },
 
+    { nxt_string("{return}"),
+      nxt_string("SyntaxError: Illegal return statement in 1") },
+
     { nxt_string("function f() { return f() } f()"),
       nxt_string("RangeError: Maximum call stack size exceeded") },
 

From igor at sysoev.ru  Mon Jan 16 15:15:21 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Mon, 16 Jan 2017 15:15:21 +0000
Subject: [njs] A scope has been added to literal nodes.
Message-ID: <hg.5db6d7af16b4.1484579721.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/5db6d7af16b4
branches:  
changeset: 302:5db6d7af16b4
user:      Igor Sysoev <igor at sysoev.ru>
date:      Mon Jan 16 18:14:01 2017 +0300
description:
A scope has been added to literal nodes.

Found with afl-fuzz.

diffstat:

 njs/njs_parser.c |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (11 lines):

diff -r 6cc16ad934a4 -r 5db6d7af16b4 njs/njs_parser.c
--- a/njs/njs_parser.c	Mon Jan 16 17:32:10 2017 +0300
+++ b/njs/njs_parser.c	Mon Jan 16 18:14:01 2017 +0300
@@ -1778,6 +1778,7 @@ njs_parser_terminal(njs_vm_t *vm, njs_pa
     }
 
     node->token = token;
+    node->scope = parser->scope;
 
     switch (token) {
 

From vakul.garg at nxp.com  Mon Jan 16 15:42:53 2017
From: vakul.garg at nxp.com (Vakul Garg)
Date: Mon, 16 Jan 2017 15:42:53 +0000
Subject: NGINx async SSL handshake
In-Reply-To: <D1A3A80A1B97944385C8C290B1BB02D05C53F5C7@fmsmsx118.amr.corp.intel.com>
References: <AM3PR04MB1284A6925938F5FE45B70C6D8BB30@AM3PR04MB1284.eurprd04.prod.outlook.com>
 <DE6835BE-C4CC-4061-A356-6168CCCCBC2E@gmail.com>
 <AM3PR04MB128446C994AB13840482B5AC8B630@AM3PR04MB1284.eurprd04.prod.outlook.com>
 <D1A3A80A1B97944385C8C290B1BB02D05C53D0DB@fmsmsx118.amr.corp.intel.com>
 <HE1PR04MB129169B4E569A1CCD818F1A28B640@HE1PR04MB1291.eurprd04.prod.outlook.com>
 <D1A3A80A1B97944385C8C290B1BB02D05C53EF78@fmsmsx118.amr.corp.intel.com>
 <AM3PR04MB128485EC217B88ED2DF1B6008B780@AM3PR04MB1284.eurprd04.prod.outlook.com>
 <D1A3A80A1B97944385C8C290B1BB02D05C53F5C7@fmsmsx118.amr.corp.intel.com>
Message-ID: <AM3PR04MB12849A77A51BA99BE525239A8B7D0@AM3PR04MB1284.eurprd04.prod.outlook.com>

Hi Brian

> For the engine response processing another alternative is to simply recall the appropriate SSL_* API which will switch to the correct JOB context and complete the crypto processing.

IIUC, for this to work, the response from the accelerator corresponding to the paused job associate with SSL connection should have been dequeued prior. 
The engine's poll thread does just that. It dequeues accelerator responses and finds out wait_ctx to wakeup through its file descriptor.
This in turn wakes triggers NGINx's epoll_wait() to return. 

It then calls SSL_* API for the connection wokenup.
Now the SSL_* API would internally call ASYNC_start_job and it resumes previously paused job to complete the operation.

Thus, IMO, simply recalling SSL_* API is not sufficient unless you meant to say that you would dequeue accelerator response directly from its command rings after job gets resumed.
In that case, if we do this, then we would have a problem if accelerator reorders the responses. 

Regards

Vakul

-----Original Message-----
From: Will, Brian [mailto:brian.will at intel.com] 
Sent: Saturday, January 14, 2017 1:51 AM
To: Vakul Garg <vakul.garg at nxp.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>; Schuetze, Joel D <joel.d.schuetze at intel.com>; Yu, Ping Y <ping.y.yu at intel.com>; Finn, Coleman <coleman.finn at intel.com>; Pankaj Gupta <pankaj.gupta at nxp.com>; Arun Pathak <arun.pathak at nxp.com>; Chambers, Mark A <mark.a.chambers at intel.com>
Subject: RE: NGINx async SSL handshake

Hi Vakul,
This sounds interesting and would give the capability to do the "polling inline". I'd like to see the changes you are proposing to the event loop.

For the engine response processing another alternative is to simply recall the appropriate SSL_* API which will switch to the correct JOB context and complete the crypto processing. This way you are able to continue using a standard OpenSSL API rather than having to create a new standardized ENGINE ctrl message. It shouldn't be much in the difference wrt performance as the context switch happens very soon after the SSL_* API invocations.

Thanks,
Brian

-----Original Message-----
From: Vakul Garg [mailto:vakul.garg at nxp.com]
Sent: Friday, January 13, 2017 1:52 AM
To: Will, Brian <brian.will at intel.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>; Schuetze, Joel D <joel.d.schuetze at intel.com>; Yu, Ping Y <ping.y.yu at intel.com>; Finn, Coleman <coleman.finn at intel.com>; Pankaj Gupta <pankaj.gupta at nxp.com>; Arun Pathak <arun.pathak at nxp.com>; Chambers, Mark A <mark.a.chambers at intel.com>
Subject: RE: NGINx async SSL handshake

Hi Brian

Yes, one of the things I am looking for is a replacement for engine private receive thread with "poll inline" from nginx worker itself.
It might not necessarily require timer based mechanism to get the nginx event loop to poll accelerator responses.

The requirement is to give some time slice to the engine to do receive side processing from the accelerator.
This time slice can be given after completing ready list of FDs in event loop.

If any SSL connection is pending on WANT_ASYNC, then event poll loop should wakeup with zero timeout, because we know accelerator response would come soon and it makes sense to give timeslice for engine accelerator response processing.
For engine accelerator response processing an ENGINE control api can be built and directly called from nginx event loop.

Please let me know your comments.

Regards

Vakul

-----Original Message-----
From: Will, Brian [mailto:brian.will at intel.com]
Sent: Friday, January 13, 2017 4:54 AM
To: Vakul Garg <vakul.garg at nxp.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>; Schuetze, Joel D <joel.d.schuetze at intel.com>; Yu, Ping Y <ping.y.yu at intel.com>; Finn, Coleman <coleman.finn at intel.com>; Pankaj Gupta <pankaj.gupta at nxp.com>; Arun Pathak <arun.pathak at nxp.com>; Chambers, Mark A <mark.a.chambers at intel.com>
Subject: RE: NGINx async SSL handshake

Hi Vakul,
Sorry for not getting back sooner. We are currently in the process of working with NGINX to get our patch integrated into the development tree. This process may result in a very different patch to what we have currently published, resulting in closer integration and fit into the infrastructure. As of right now we do not have the patch posted either as a repo or a review request. 

Your work on the wakeup mechanism would be very interesting, specifically what are you looking at as a replacement for the activation thread which the engine would own? On the option to "poll inline" from the nginx worker itself this is something we had explored and does result in increased performance, but then you would require some timer based mechanism to get the NGINX event loop to poll the SSL* API in question. This can be added and is supported through the current OpenSSL async features, but seemed like a different processing model to the way NGINX handles all other activity.

Thanks,
Brian

-----Original Message-----
From: Vakul Garg [mailto:vakul.garg at nxp.com]
Sent: Monday, January 09, 2017 12:52 AM
To: Will, Brian <brian.will at intel.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>; Schuetze, Joel D <joel.d.schuetze at intel.com>; Yu, Ping Y <ping.y.yu at intel.com>; Finn, Coleman <coleman.finn at intel.com>; Pankaj Gupta <pankaj.gupta at nxp.com>; Arun Pathak <arun.pathak at nxp.com>
Subject: RE: NGINx async SSL handshake

Hi Brian

We are using your nginx patch with our PKC accelerator (exposed through cryptodev-linux ioctls).
The OpenSSL cryptodev engine has been enhanced to make async PKC calls.

The system is running functionally well, but the performance is poor.

Last year, we used older OpenSSL versions (without ASYNC job infrastructure) with Geoff Thorpe's https://github.com/libfibre to achieve async crypto acceleration.
The performance was better than we are getting with OpenSSL-1.1.0 + Nginx patch in question.

On studying the Intel nginx changes, I feel that the way the asynchronous APIs have been integrated are somewhat heavy and there is possibility of improvement.
Specifically, the eventfd based job wakeup mechanism seems heavy.

I am looking towards making changes over your nginx patch to introduce a newer job wakeup method.
Further, I want to get rid of creating set of accelerator response poll threads in engine and provide an option to do this inline in nginx worker thread itself.

For this, I hoped your patches are submitted upstream and in process of review or I can find your work in some github repo where we can submit our changes.

Let me know your comments.

Regards

Vakul

-----Original Message-----
From: Will, Brian [mailto:brian.will at intel.com]
Sent: Saturday, January 07, 2017 2:40 AM
To: Vakul Garg <vakul.garg at nxp.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>; Schuetze, Joel D <joel.d.schuetze at intel.com>; Yu, Ping Y <ping.y.yu at intel.com>; Finn, Coleman <coleman.finn at intel.com>
Subject: RE: NGINx async SSL handshake

Hello Vakul,
The link you have included is the latest patch for NGINX to support OpenSSL-1.1.0 that we are maintaining. We are working to try and get these changes included into the mainline of NGINX, but in the interim this patch should work for the OpenSSL-1.1.0 branch of code.
Were there specific issues that you have run into?

Thanks,
Brian

-----Original Message-----
From: Vakul Garg [mailto:vakul.garg at nxp.com]
Sent: Friday, January 06, 2017 12:10 PM
To: Will, Brian <brian.will at intel.com>
Cc: nginx-devel at nginx.org; Michael Kardonik <michael.kardonik at nxp.com>
Subject: RE: NGINx async SSL handshake

Hi Brian

I am using nginx openssl async: https://01.org/sites/default/files/page/nginx-1.10.0-async.l.0.3.0-001_0.tgz with my async RSA accelerator.
Where can I find the updated version of this work?

Can you share your plans to upstream this work?

Regards

Vakul


-----Original Message-----
From: nginx-devel [mailto:nginx-devel-bounces at nginx.org] On Behalf Of Alexey Ivanov
Sent: Saturday, November 19, 2016 2:00 PM
To: nginx-devel at nginx.org
Cc: brian.will at intel.com
Subject: Re: NGINx async SSL handshake

+Brian Will from Intel, to correct me if I'm wrong.

My two cents here(Intel Quick Assist specific, based on conversations during Nginx.Conf):
1) Even without hardware offload async handshake helps in cases where you have high TLS connection rates, because right now handshake is basically a 2ms+ blocking operation(specific timing depend on AVX/AVX2 support[0]) inside the event loop. Therefore after some TLS connection rate nginx performance falls off the cliff.
2) Hardware offload numbers look very impressive[1](TL;DR: 5x improvement for RSA 2048, for ECDSA, imho, it is neither impressive, nor needed). Also they prove asymmetric part of the accelerator to be future proof, so that it is possible to add new handshake types(e.g. Ed25519). Disclaimer: we did not test that hardware yet.

As for patches, you can check 01.org for:
a) nginx openssl async: https://01.org/sites/default/files/page/nginx-1.10.0-async.l.0.3.0-001_0.tgz
b) zlib[2]: https://01.org/sites/default/files/page/zlib_shim_0.4.9-001.tgz
(Full list of docs: https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches )

Question for Brian/Maxim: are you planning on integrating it into mainline nginx? 1000+ line diffs are usually rather hard to integrate.


[0] FWIW, speaking about OpenSSL performance: using OpenSSL 1.0.2 + Intel Xeon v2 processors with AVX2 gives 2x performance boost(over OpenSSL 1.0.1 and v1).
[1] https://twitter.com/SaveTheRbtz/status/773962669166428161
[2] There are also cloudflare and intel patches for zlib for faster deflation (i.e. compression only)
> On Nov 18, 2016, at 8:12 PM, Vakul Garg <vakul.garg at nxp.com> wrote:
> 
> Hi
> 
> I am a newbie to nginx.
> I am integrating a public key hardware accelerator in OpenSSL using engine interface.
> The engine is async capable.
> 
> Recently openssl-1.1.0 has added support for ASYNC_JOB.
> IIUC, nginx would also require changes in order to do SSL handshake in async way.
> 
> Any pointers where can I get the nginx code changes done for async 
> openssl
> 
> Regards
> 
> Vakul
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel


From joel.cunningham at me.com  Tue Jan 17 15:37:58 2017
From: joel.cunningham at me.com (Joel Cunningham)
Date: Tue, 17 Jan 2017 09:37:58 -0600
Subject: [PATCH] Fix drain logic for small number of connections
In-Reply-To: <20161231020414.GJ18639@mdounin.ru>
References: <691A2C9B-3561-4223-895B-EDBA1884743A@me.com>
 <9FA73531-083D-40B8-9A45-A571CA55EAA2@me.com>
 <20161228170515.GE18639@mdounin.ru>
 <ADAF3CA2-23BF-4542-A643-FD7ACCF11F2A@me.com>
 <20161229191334.GI18639@mdounin.ru>
 <492FC288-6BCE-46D4-9EB1-C20C85C3F8A4@me.com>
 <20161231020414.GJ18639@mdounin.ru>
Message-ID: <BA4FE7DF-8F27-4C89-B0DC-384FE46FA973@me.com>

Maxim,

> On Dec 30, 2016, at 8:04 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> On Fri, Dec 30, 2016 at 12:13:01PM -0600, Joel Cunningham wrote:
> 
>>> On Dec 29, 2016, at 1:13 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> [...]
> 
>>> I agree though that if connection shortage is occasional the 
>>> current behaviour might not be optimal with worker_connections set 
>>> to a small value, or there are only a few connections can be 
>>> reused.  Instead of gradually reducing keepalive (and post accept) 
>>> timeout to a smaller value it closes all keepalive connections 
>>> from time to time.
>>> 
>>> Probably optimal solution would be to close no more than some 
>>> fraction of reusable connections at once.  Patch series below.
>> 
>> I tested out the patch series and I did run into a problem.  
>> Looks like the reusable_connections_n increment and decrement 
>> are swapped.  The first call to ngx_reusable_connection causes 
>> unsigned integer wrap around.  I swapped the increment/decrement 
>> and the patch gives the expected behavior of only freeing 1/8 
>> (or at least 1) of the reusable connections.
> 
> Thanks for noticing this, fixed.
> 

Was your patch series going to be comitted?  I didn?t see anything in the commit history yet

Thanks,

Joel


From sreekanth_sk at yahoo.com  Tue Jan 17 16:37:17 2017
From: sreekanth_sk at yahoo.com (Sreekanth M)
Date: Tue, 17 Jan 2017 16:37:17 +0000 (UTC)
Subject: HTTP/2 upstream support
References: <1463025666.5010152.1484671037148.ref@mail.yahoo.com>
Message-ID: <1463025666.5010152.1484671037148@mail.yahoo.com>


Is HTTP/2 proxy support planned ?
-Sreekanth
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170117/14bd7d5b/attachment.html>

From savetherbtz at gmail.com  Wed Jan 18 11:59:38 2017
From: savetherbtz at gmail.com (Alexey Ivanov)
Date: Wed, 18 Jan 2017 03:59:38 -0800
Subject: HTTP/2 upstream support
In-Reply-To: <1463025666.5010152.1484671037148@mail.yahoo.com>
References: <1463025666.5010152.1484671037148.ref@mail.yahoo.com>
 <1463025666.5010152.1484671037148@mail.yahoo.com>
Message-ID: <7E684788-CF39-41E5-ACA1-312AC4F09DDA@gmail.com>

Just as a datapoint: why do you need that functionality? Can you describe your particular usecase?

> On Jan 17, 2017, at 8:37 AM, Sreekanth M via nginx-devel <nginx-devel at nginx.org> wrote:
> 
> 
> Is HTTP/2 proxy support planned ?
> 
> -Sreekanth
> 
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170118/a3f6836c/attachment.bin>

From mdounin at mdounin.ru  Thu Jan 19 13:03:11 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 19 Jan 2017 16:03:11 +0300
Subject: [PATCH] Fix drain logic for small number of connections
In-Reply-To: <BA4FE7DF-8F27-4C89-B0DC-384FE46FA973@me.com>
References: <691A2C9B-3561-4223-895B-EDBA1884743A@me.com>
 <9FA73531-083D-40B8-9A45-A571CA55EAA2@me.com>
 <20161228170515.GE18639@mdounin.ru>
 <ADAF3CA2-23BF-4542-A643-FD7ACCF11F2A@me.com>
 <20161229191334.GI18639@mdounin.ru>
 <492FC288-6BCE-46D4-9EB1-C20C85C3F8A4@me.com>
 <20161231020414.GJ18639@mdounin.ru>
 <BA4FE7DF-8F27-4C89-B0DC-384FE46FA973@me.com>
Message-ID: <20170119130311.GH45866@mdounin.ru>

Hello!

On Tue, Jan 17, 2017 at 09:37:58AM -0600, Joel Cunningham wrote:

> Maxim,
> 
> > On Dec 30, 2016, at 8:04 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> > 
> > Hello!
> > 
> > On Fri, Dec 30, 2016 at 12:13:01PM -0600, Joel Cunningham wrote:
> > 
> >>> On Dec 29, 2016, at 1:13 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> > 
> > [...]
> > 
> >>> I agree though that if connection shortage is occasional the 
> >>> current behaviour might not be optimal with worker_connections set 
> >>> to a small value, or there are only a few connections can be 
> >>> reused.  Instead of gradually reducing keepalive (and post accept) 
> >>> timeout to a smaller value it closes all keepalive connections 
> >>> from time to time.
> >>> 
> >>> Probably optimal solution would be to close no more than some 
> >>> fraction of reusable connections at once.  Patch series below.
> >> 
> >> I tested out the patch series and I did run into a problem.  
> >> Looks like the reusable_connections_n increment and decrement 
> >> are swapped.  The first call to ngx_reusable_connection causes 
> >> unsigned integer wrap around.  I swapped the increment/decrement 
> >> and the patch gives the expected behavior of only freeing 1/8 
> >> (or at least 1) of the reusable connections.
> > 
> > Thanks for noticing this, fixed.
> > 
> 
> Was your patch series going to be comitted?  I didn?t see anything in the commit history yet

It is expected to be committed in a couple of days.

-- 
Maxim Dounin
http://nginx.org/

From vl at nginx.com  Thu Jan 19 13:32:15 2017
From: vl at nginx.com (Vladimir Homutov)
Date: Thu, 19 Jan 2017 13:32:15 +0000
Subject: [nginx] Stream: fixed handling of non-ssl sessions.
Message-ID: <hg.0a08a8babf53.1484832735.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/0a08a8babf53
branches:  
changeset: 6870:0a08a8babf53
user:      Vladimir Homutov <vl at nginx.com>
date:      Thu Jan 19 16:17:05 2017 +0300
description:
Stream: fixed handling of non-ssl sessions.

A missing check could cause ngx_stream_ssl_handler() to be applied
to a non-ssl session, which resulted in a null pointer dereference
if ssl_verify_client is enabled.

The bug had appeared in 1.11.8 (41cb1b64561d).

diffstat:

 src/stream/ngx_stream_ssl_module.c |  6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diffs (20 lines):

diff -r b2915d99ee8d -r 0a08a8babf53 src/stream/ngx_stream_ssl_module.c
--- a/src/stream/ngx_stream_ssl_module.c	Thu Jan 12 19:22:03 2017 +0300
+++ b/src/stream/ngx_stream_ssl_module.c	Thu Jan 19 16:17:05 2017 +0300
@@ -287,11 +287,15 @@ ngx_stream_ssl_handler(ngx_stream_sessio
     ngx_connection_t       *c;
     ngx_stream_ssl_conf_t  *sslcf;
 
+    if (!s->ssl) {
+        return NGX_OK;
+    }
+
     c = s->connection;
 
     sslcf = ngx_stream_get_module_srv_conf(s, ngx_stream_ssl_module);
 
-    if (s->ssl && c->ssl == NULL) {
+    if (c->ssl == NULL) {
         c->log->action = "SSL handshaking";
 
         if (sslcf->ssl.ctx == NULL) {

From vl at nginx.com  Thu Jan 19 13:32:18 2017
From: vl at nginx.com (Vladimir Homutov)
Date: Thu, 19 Jan 2017 13:32:18 +0000
Subject: [nginx] Stream: client SSL certificates were not checked in some
 cases.
Message-ID: <hg.1818acd8442f.1484832738.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/1818acd8442f
branches:  
changeset: 6871:1818acd8442f
user:      Vladimir Homutov <vl at nginx.com>
date:      Thu Jan 19 16:20:07 2017 +0300
description:
Stream: client SSL certificates were not checked in some cases.

If ngx_stream_ssl_init_connection() succeeded immediately, the check was not
done.

The bug had appeared in 1.11.8 (41cb1b64561d).

diffstat:

 src/stream/ngx_stream_ssl_module.c |  7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diffs (24 lines):

diff -r 0a08a8babf53 -r 1818acd8442f src/stream/ngx_stream_ssl_module.c
--- a/src/stream/ngx_stream_ssl_module.c	Thu Jan 19 16:17:05 2017 +0300
+++ b/src/stream/ngx_stream_ssl_module.c	Thu Jan 19 16:20:07 2017 +0300
@@ -284,6 +284,7 @@ ngx_stream_ssl_handler(ngx_stream_sessio
 {
     long                    rc;
     X509                   *cert;
+    ngx_int_t               rv;
     ngx_connection_t       *c;
     ngx_stream_ssl_conf_t  *sslcf;
 
@@ -305,7 +306,11 @@ ngx_stream_ssl_handler(ngx_stream_sessio
             return NGX_ERROR;
         }
 
-        return ngx_stream_ssl_init_connection(&sslcf->ssl, c);
+        rv = ngx_stream_ssl_init_connection(&sslcf->ssl, c);
+
+        if (rv != NGX_OK) {
+            return rv;
+        }
     }
 
     if (sslcf->verify) {

From mdounin at mdounin.ru  Fri Jan 20 11:03:54 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 20 Jan 2017 11:03:54 +0000
Subject: [nginx] Added cycle parameter to ngx_drain_connections().
Message-ID: <hg.6ed0922d316b.1484910234.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/6ed0922d316b
branches:  
changeset: 6872:6ed0922d316b
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jan 20 14:03:19 2017 +0300
description:
Added cycle parameter to ngx_drain_connections().

No functional changes, mostly style.

diffstat:

 src/core/ngx_connection.c |  10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diffs (43 lines):

diff --git a/src/core/ngx_connection.c b/src/core/ngx_connection.c
--- a/src/core/ngx_connection.c
+++ b/src/core/ngx_connection.c
@@ -13,7 +13,7 @@
 ngx_os_io_t  ngx_io;
 
 
-static void ngx_drain_connections(void);
+static void ngx_drain_connections(ngx_cycle_t *cycle);
 
 
 ngx_listening_t *
@@ -1046,7 +1046,7 @@ ngx_get_connection(ngx_socket_t s, ngx_l
     c = ngx_cycle->free_connections;
 
     if (c == NULL) {
-        ngx_drain_connections();
+        ngx_drain_connections((ngx_cycle_t *) ngx_cycle);
         c = ngx_cycle->free_connections;
     }
 
@@ -1226,18 +1226,18 @@ ngx_reusable_connection(ngx_connection_t
 
 
 static void
-ngx_drain_connections(void)
+ngx_drain_connections(ngx_cycle_t *cycle)
 {
     ngx_int_t          i;
     ngx_queue_t       *q;
     ngx_connection_t  *c;
 
     for (i = 0; i < 32; i++) {
-        if (ngx_queue_empty(&ngx_cycle->reusable_connections_queue)) {
+        if (ngx_queue_empty(&cycle->reusable_connections_queue)) {
             break;
         }
 
-        q = ngx_queue_last(&ngx_cycle->reusable_connections_queue);
+        q = ngx_queue_last(&cycle->reusable_connections_queue);
         c = ngx_queue_data(q, ngx_connection_t, queue);
 
         ngx_log_debug0(NGX_LOG_DEBUG_CORE, c->log, 0,

From mdounin at mdounin.ru  Fri Jan 20 11:04:02 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 20 Jan 2017 11:04:02 +0000
Subject: [nginx] Improved connection draining with small number of connections.
Message-ID: <hg.426828549afc.1484910242.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/426828549afc
branches:  
changeset: 6873:426828549afc
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jan 20 14:03:20 2017 +0300
description:
Improved connection draining with small number of connections.

Closing up to 32 connections might be too aggressive if worker_connections
is set to a comparable number (and/or there are only a small number of
reusable connections).  If an occasional connection shorage happens in
such a configuration, it leads to closing all reusable connections instead
of gradually reducing keepalive timeout to a smaller value.  To improve
granularity in such configurations we now close no more than 1/8 of all
reusable connections at once.

Suggested by Joel Cunningham.

diffstat:

 src/core/ngx_connection.c |  8 ++++++--
 src/core/ngx_cycle.h      |  1 +
 2 files changed, 7 insertions(+), 2 deletions(-)

diffs (46 lines):

diff --git a/src/core/ngx_connection.c b/src/core/ngx_connection.c
--- a/src/core/ngx_connection.c
+++ b/src/core/ngx_connection.c
@@ -1204,6 +1204,7 @@ ngx_reusable_connection(ngx_connection_t
 
     if (c->reusable) {
         ngx_queue_remove(&c->queue);
+        ngx_cycle->reusable_connections_n--;
 
 #if (NGX_STAT_STUB)
         (void) ngx_atomic_fetch_add(ngx_stat_waiting, -1);
@@ -1217,6 +1218,7 @@ ngx_reusable_connection(ngx_connection_t
 
         ngx_queue_insert_head(
             (ngx_queue_t *) &ngx_cycle->reusable_connections_queue, &c->queue);
+        ngx_cycle->reusable_connections_n++;
 
 #if (NGX_STAT_STUB)
         (void) ngx_atomic_fetch_add(ngx_stat_waiting, 1);
@@ -1228,11 +1230,13 @@ ngx_reusable_connection(ngx_connection_t
 static void
 ngx_drain_connections(ngx_cycle_t *cycle)
 {
-    ngx_int_t          i;
+    ngx_uint_t         i, n;
     ngx_queue_t       *q;
     ngx_connection_t  *c;
 
-    for (i = 0; i < 32; i++) {
+    n = ngx_max(ngx_min(32, cycle->reusable_connections_n / 8), 1);
+
+    for (i = 0; i < n; i++) {
         if (ngx_queue_empty(&cycle->reusable_connections_queue)) {
             break;
         }
diff --git a/src/core/ngx_cycle.h b/src/core/ngx_cycle.h
--- a/src/core/ngx_cycle.h
+++ b/src/core/ngx_cycle.h
@@ -53,6 +53,7 @@ struct ngx_cycle_s {
     ngx_uint_t                modules_used;    /* unsigned  modules_used:1; */
 
     ngx_queue_t               reusable_connections_queue;
+    ngx_uint_t                reusable_connections_n;
 
     ngx_array_t               listening;
     ngx_array_t               paths;

From igor at sysoev.ru  Fri Jan 20 13:11:23 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Fri, 20 Jan 2017 13:11:23 +0000
Subject: [njs] A small rbtree insert fixup optimization.
Message-ID: <hg.8401ae77cf40.1484917883.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/8401ae77cf40
branches:  
changeset: 303:8401ae77cf40
user:      Igor Sysoev <igor at sysoev.ru>
date:      Fri Jan 20 16:10:48 2017 +0300
description:
A small rbtree insert fixup optimization.

Thanks to ??? (Hong Zhi Dao).

diffstat:

 nxt/nxt_rbtree.c |  11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)

diffs (28 lines):

diff -r 5db6d7af16b4 -r 8401ae77cf40 nxt/nxt_rbtree.c
--- a/nxt/nxt_rbtree.c	Mon Jan 16 18:14:01 2017 +0300
+++ b/nxt/nxt_rbtree.c	Fri Jan 20 16:10:48 2017 +0300
@@ -135,8 +135,12 @@ nxt_rbtree_insert_fixup(nxt_rbtree_node_
                 grandparent = parent->parent;
                 grandparent->color = NXT_RBTREE_RED;
                 nxt_rbtree_right_rotate(grandparent);
-
-                continue;
+                /*
+                 * nxt_rbtree_right_rotate() does not change node->parent
+                 * color which is now black, so testing color is not required
+                 * to return from function.
+                 */
+                return;
             }
 
         } else {
@@ -156,7 +160,8 @@ nxt_rbtree_insert_fixup(nxt_rbtree_node_
                 grandparent->color = NXT_RBTREE_RED;
                 nxt_rbtree_left_rotate(grandparent);
 
-                continue;
+                /* See the comment in the symmetric branch above. */
+                return;
             }
         }
 

From mdounin at mdounin.ru  Fri Jan 20 18:14:00 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 20 Jan 2017 18:14:00 +0000
Subject: [nginx] Fixed trailer construction with limit on FreeBSD and macOS.
Message-ID: <hg.7cc2d3a96ea3.1484936040.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/7cc2d3a96ea3
branches:  
changeset: 6874:7cc2d3a96ea3
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jan 20 21:12:48 2017 +0300
description:
Fixed trailer construction with limit on FreeBSD and macOS.

The ngx_chain_coalesce_file() function may produce more bytes to send then
requested in the limit passed, as it aligns the last file position
to send to memory page boundary.  As a result, (limit - send) may become
negative.  This resulted in big positive number when converted to size_t
while calling ngx_output_chain_to_iovec().

Another part of the problem is in ngx_chain_coalesce_file(): it changes cl
to the next chain link even if the current buffer is only partially sent
due to limit.

Therefore, if a file buffer was not expected to be fully sent due to limit,
and was followed by a memory buffer, nginx called sendfile() with a part
of the file buffer, and the memory buffer in trailer.  If there were enough
room in the socket buffer, this resulted in a part of the file buffer being
skipped, and corresponding part of the memory buffer sent instead.

The bug was introduced in 8e903522c17a (1.7.8).  Configurations affected
are ones using limits, that is, limit_rate and/or sendfile_max_chunk, and
memory buffers after file ones (may happen when using subrequests or
with proxying with disk buffering).

Fix is to explicitly check if (send < limit) before constructing trailer
with ngx_output_chain_to_iovec().  Additionally, ngx_chain_coalesce_file()
was modified to preserve unfinished file buffers in cl.

diffstat:

 src/core/ngx_buf.c                       |   3 +++
 src/os/unix/ngx_darwin_sendfile_chain.c  |   2 +-
 src/os/unix/ngx_freebsd_sendfile_chain.c |  22 +++++++++++++++-------
 3 files changed, 19 insertions(+), 8 deletions(-)

diffs (60 lines):

diff --git a/src/core/ngx_buf.c b/src/core/ngx_buf.c
--- a/src/core/ngx_buf.c
+++ b/src/core/ngx_buf.c
@@ -246,6 +246,9 @@ ngx_chain_coalesce_file(ngx_chain_t **in
             if (aligned <= cl->buf->file_last) {
                 size = aligned - cl->buf->file_pos;
             }
+
+            total += size;
+            break;
         }
 
         total += size;
diff --git a/src/os/unix/ngx_darwin_sendfile_chain.c b/src/os/unix/ngx_darwin_sendfile_chain.c
--- a/src/os/unix/ngx_darwin_sendfile_chain.c
+++ b/src/os/unix/ngx_darwin_sendfile_chain.c
@@ -98,7 +98,7 @@ ngx_darwin_sendfile_chain(ngx_connection
 
             send += file_size;
 
-            if (header.count == 0) {
+            if (header.count == 0 && send < limit) {
 
                 /*
                  * create the trailer iovec and coalesce the neighbouring bufs
diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -114,16 +114,24 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
             send += file_size;
 
-            /* create the trailer iovec and coalesce the neighbouring bufs */
-
-            cl = ngx_output_chain_to_iovec(&trailer, cl, limit - send, c->log);
+            if (send < limit) {
 
-            if (cl == NGX_CHAIN_ERROR) {
-                return NGX_CHAIN_ERROR;
+                /*
+                 * create the trailer iovec and coalesce the neighbouring bufs
+                 */
+
+                cl = ngx_output_chain_to_iovec(&trailer, cl, limit - send,
+                                               c->log);
+                if (cl == NGX_CHAIN_ERROR) {
+                    return NGX_CHAIN_ERROR;
+                }
+
+                send += trailer.size;
+
+            } else {
+                trailer.count = 0;
             }
 
-            send += trailer.size;
-
             if (ngx_freebsd_use_tcp_nopush
                 && c->tcp_nopush == NGX_TCP_NOPUSH_UNSET)
             {

From mdounin at mdounin.ru  Fri Jan 20 18:15:22 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 20 Jan 2017 18:15:22 +0000
Subject: [nginx] Removed pthread mutex / conditional variables debug messages.
Message-ID: <hg.022ea0d17177.1484936122.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/022ea0d17177
branches:  
changeset: 6875:022ea0d17177
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jan 20 21:14:18 2017 +0300
description:
Removed pthread mutex / conditional variables debug messages.

These messages doesn't seem to be needed in practice and only make
debugging logs harder to read.

diffstat:

 src/os/unix/ngx_thread_cond.c  |  11 -----------
 src/os/unix/ngx_thread_mutex.c |   9 ---------
 2 files changed, 0 insertions(+), 20 deletions(-)

diffs (89 lines):

diff --git a/src/os/unix/ngx_thread_cond.c b/src/os/unix/ngx_thread_cond.c
--- a/src/os/unix/ngx_thread_cond.c
+++ b/src/os/unix/ngx_thread_cond.c
@@ -16,8 +16,6 @@ ngx_thread_cond_create(ngx_thread_cond_t
 
     err = pthread_cond_init(cond, NULL);
     if (err == 0) {
-        ngx_log_debug1(NGX_LOG_DEBUG_CORE, log, 0,
-                       "pthread_cond_init(%p)", cond);
         return NGX_OK;
     }
 
@@ -33,8 +31,6 @@ ngx_thread_cond_destroy(ngx_thread_cond_
 
     err = pthread_cond_destroy(cond);
     if (err == 0) {
-        ngx_log_debug1(NGX_LOG_DEBUG_CORE, log, 0,
-                       "pthread_cond_destroy(%p)", cond);
         return NGX_OK;
     }
 
@@ -50,8 +46,6 @@ ngx_thread_cond_signal(ngx_thread_cond_t
 
     err = pthread_cond_signal(cond);
     if (err == 0) {
-        ngx_log_debug1(NGX_LOG_DEBUG_CORE, log, 0,
-                       "pthread_cond_signal(%p)", cond);
         return NGX_OK;
     }
 
@@ -66,9 +60,6 @@ ngx_thread_cond_wait(ngx_thread_cond_t *
 {
     ngx_err_t  err;
 
-    ngx_log_debug1(NGX_LOG_DEBUG_CORE, log, 0,
-                   "pthread_cond_wait(%p) enter", cond);
-
     err = pthread_cond_wait(cond, mtx);
 
 #if 0
@@ -76,8 +67,6 @@ ngx_thread_cond_wait(ngx_thread_cond_t *
 #endif
 
     if (err == 0) {
-        ngx_log_debug1(NGX_LOG_DEBUG_CORE, log, 0,
-                       "pthread_cond_wait(%p) exit", cond);
         return NGX_OK;
     }
 
diff --git a/src/os/unix/ngx_thread_mutex.c b/src/os/unix/ngx_thread_mutex.c
--- a/src/os/unix/ngx_thread_mutex.c
+++ b/src/os/unix/ngx_thread_mutex.c
@@ -108,8 +108,6 @@ ngx_thread_mutex_create(ngx_thread_mutex
                       "pthread_mutexattr_destroy() failed");
     }
 
-    ngx_log_debug1(NGX_LOG_DEBUG_CORE, log, 0,
-                   "pthread_mutex_init(%p)", mtx);
     return NGX_OK;
 }
 
@@ -126,8 +124,6 @@ ngx_thread_mutex_destroy(ngx_thread_mute
         return NGX_ERROR;
     }
 
-    ngx_log_debug1(NGX_LOG_DEBUG_CORE, log, 0,
-                   "pthread_mutex_destroy(%p)", mtx);
     return NGX_OK;
 }
 
@@ -137,9 +133,6 @@ ngx_thread_mutex_lock(ngx_thread_mutex_t
 {
     ngx_err_t  err;
 
-    ngx_log_debug1(NGX_LOG_DEBUG_CORE, log, 0,
-                   "pthread_mutex_lock(%p) enter", mtx);
-
     err = pthread_mutex_lock(mtx);
     if (err == 0) {
         return NGX_OK;
@@ -163,8 +156,6 @@ ngx_thread_mutex_unlock(ngx_thread_mutex
 #endif
 
     if (err == 0) {
-        ngx_log_debug1(NGX_LOG_DEBUG_CORE, log, 0,
-                       "pthread_mutex_unlock(%p) exit", mtx);
         return NGX_OK;
     }
 

From mdounin at mdounin.ru  Fri Jan 20 18:15:25 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 20 Jan 2017 18:15:25 +0000
Subject: [nginx] Upstream: fixed cache corruption and socket leaks with
 aio_write.
Message-ID: <hg.a5d1b1383dea.1484936125.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/a5d1b1383dea
branches:  
changeset: 6876:a5d1b1383dea
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jan 20 21:14:19 2017 +0300
description:
Upstream: fixed cache corruption and socket leaks with aio_write.

The ngx_event_pipe() function wasn't called on write events with
wev->delayed set.  As a result, threaded writing results weren't
properly collected in ngx_event_pipe_write_to_downstream() when a
write event was triggered for a completed write.

Further, this wasn't detected, as p->aio was reset by a thread completion
handler, and results were later collected in ngx_event_pipe_read_upstream()
instead of scheduling a new write of additional data.  If this happened
on the last reading from an upstream, last part of the response was never
written to the cache file.

Similar problems might also happen in case of timeouts when writing to
client, as this also results in ngx_event_pipe() not being called on write
events.  In this scenario socket leaks were observed.

Fix is to check if p->writing is set in ngx_event_pipe_read_upstream(), and
therefore collect results of previous write operations in case of read events
as well, similar to how we do so in ngx_event_pipe_write_downstream().
This is enough to fix the wev->delayed case.  Additionally, we now call
ngx_event_pipe() from ngx_http_upstream_process_request() if there are
uncollected write operations (p->writing and !p->aio).  This also fixes
the wev->timedout case.

diffstat:

 src/event/ngx_event_pipe.c   |  13 +++++++++++++
 src/http/ngx_http_upstream.c |  15 +++++++++++++++
 2 files changed, 28 insertions(+), 0 deletions(-)

diffs (56 lines):

diff --git a/src/event/ngx_event_pipe.c b/src/event/ngx_event_pipe.c
--- a/src/event/ngx_event_pipe.c
+++ b/src/event/ngx_event_pipe.c
@@ -113,11 +113,24 @@ ngx_event_pipe_read_upstream(ngx_event_p
     }
 
 #if (NGX_THREADS)
+
     if (p->aio) {
         ngx_log_debug0(NGX_LOG_DEBUG_EVENT, p->log, 0,
                        "pipe read upstream: aio");
         return NGX_AGAIN;
     }
+
+    if (p->writing) {
+        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, p->log, 0,
+                       "pipe read upstream: writing");
+
+        rc = ngx_event_pipe_write_chain_to_temp_file(p);
+
+        if (rc != NGX_OK) {
+            return rc;
+        }
+    }
+
 #endif
 
     ngx_log_debug1(NGX_LOG_DEBUG_EVENT, p->log, 0,
diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c
+++ b/src/http/ngx_http_upstream.c
@@ -3848,9 +3848,24 @@ ngx_http_upstream_process_request(ngx_ht
     p = u->pipe;
 
 #if (NGX_THREADS)
+
+    if (p->writing && !p->aio) {
+
+        /*
+         * make sure to call ngx_event_pipe()
+         * if there is an incomplete aio write
+         */
+
+        if (ngx_event_pipe(p, 1) == NGX_ABORT) {
+            ngx_http_upstream_finalize_request(r, u, NGX_ERROR);
+            return;
+        }
+    }
+
     if (p->writing) {
         return;
     }
+
 #endif
 
     if (u->peer.connection) {

From piotrsikora at google.com  Sat Jan 21 01:02:18 2017
From: piotrsikora at google.com (Piotr Sikora)
Date: Fri, 20 Jan 2017 17:02:18 -0800
Subject: [PATCH 1 of 2] HTTP: add support for "429 Too Many Requests"
 response (RFC6585)
In-Reply-To: <CAF-CG++o82ZtGp-hC1TZLp5QPvfpE4dRT7upjqA89c9hQOUONQ@mail.gmail.com>
References: <1eec5355ef1e4a8b0aec.1476863503@piotrsikora.sfo.corp.google.com>
 <CAF-CG++o82ZtGp-hC1TZLp5QPvfpE4dRT7upjqA89c9hQOUONQ@mail.gmail.com>
Message-ID: <CAF-CG+JKbiLS2r3d=sNhyGKppCTXLJ6r=g873eF4s4NdfLepZA@mail.gmail.com>

Hey,

> # HG changeset patch
> # User Piotr Sikora <piotrsikora at google.com>
> # Date 1476859139 25200
> #      Tue Oct 18 23:38:59 2016 -0700
> # Node ID 1eec5355ef1e4a8b0aecebdec84c744734c0d36e
> # Parent  8081e1f3ab8b9ccb4e2d7f9240cbfb8e404a3c95
> HTTP: add support for "429 Too Many Requests" response (RFC6585).
>
> This change adds reason phrase in status line and pretty response body
> when "429" status code is used in "return", "limit_conn_status" and/or
> "limit_req_status" directives.
>
> It also adds "http_429" parameter to "proxy_next_upstream" for retrying
> rate-limited requests, and to "proxy_cache_use_stale" for serving stale
> cached responses after being rate-limited.

Ping.

Best regards,
Piotr Sikora

From alon.blayergat at gmail.com  Sun Jan 22 14:06:50 2017
From: alon.blayergat at gmail.com (Alon Blayer-Gat)
Date: Sun, 22 Jan 2017 14:06:50 +0000
Subject: [PATCH] Http gunzip: additional configuration
Message-ID: <CABbx4XBKRNC0bQzfDq9FkoMbo1ZJTamZ04e4OstoWX5hJGnKdg@mail.gmail.com>

Sure. Thanks for the feedback. I made is simpler.
evil 'if in location' removed :). I also removed the 'gunzip types;' option.
And now we now only have 'gunzip off|on|always'

But then again, with 'always', one must specify with 'gunzip_types' the
mime types to always gunzip.
The rational is that if we want to always gunzip, then it's probably not
because the client does not support it but rather because we would like to
modify the response. In which case, it is needed only for specific content
types.
Default mime type is text/html ( similar to gzip_types).

I thought of your suggestion of making a more generic mechanism to be used
by modules directly. It would require modifications in existing modules to
make the feature usable.
This patch suggests a simple change for the benefit of existing
text-related body-filter modules.

Hope it makes sense.
Patch attached.

Thanks,
/Alon


On Wed, Dec 7, 2016 at 5:58 PM Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Sun, Nov 27, 2016 at 02:27:56PM +0200, Alon Blayer-Gat wrote:
>
> > Hi,
> >
> > 1) 'gunzip always' option will gunzip even if the client supports it.
> > 2) 'gunzip types', like 'always' but only for file types specified
> > with 'gunzip_types <mime-types>'
> > 3) Allow gunzip and gunzip_types directives within "if in location"
> > block (rewrite phase condition).
> >
> > The suggested changes are needed, mainly, to allow dynamic
> > modification of compressed response (e.g. with the 'sub_filter'
> > module)
> > 'types' and 'if in location' may allow a more selective operation.
>
> No, thanks.
>
> "If in location" is evil, don't even try to suggest patches to
> allow directives in the "if in location" context.
>
> As for other changes - I can't say I like them as well.  We may
> consider something as simple as "gunzip always", but additional
> types filter certainly looks like an overkill.  Rather it should
> be some more generic mechanism to require gunzipping, may be
> useable by modules directly.
>
> [...]
>
> --
> Maxim Dounin
> http://nginx.org/
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170122/863db42e/attachment.html>

From alon.blayergat at gmail.com  Mon Jan 23 06:23:05 2017
From: alon.blayergat at gmail.com (Alon Blayer-Gat)
Date: Mon, 23 Jan 2017 06:23:05 +0000
Subject: [PATCH] Http gunzip: additional configuration
Message-ID: <CABbx4XDqpRFBnjshK5zU4sQbuu=qOCWgoGARsjOKoNJu3VJY_Q@mail.gmail.com>

Patch attached

On Sun, Jan 22, 2017 at 4:07 PM Alon Blayer-Gat <alon.blayergat at gmail.com>
wrote:

> Sure. Thanks for the feedback. I made is simpler.
>
> evil 'if in location' removed :). I also removed the 'gunzip types;'
> option.
> And now we now only have 'gunzip off|on|always'
>
> But then again, with 'always', one must specify with 'gunzip_types' the
> mime types to always gunzip.
> The rational is that if we want to always gunzip, then it's probably not
> because the client does not support it but rather because we would like to
> modify the response. In which case, it is needed only for specific content
> types.
> Default mime type is text/html ( similar to gzip_types).
>
> I thought of your suggestion of making a more generic mechanism to be used
> by modules directly. It would require modifications in existing modules to
> make the feature usable.
> This patch suggests a simple change for the benefit of existing
> text-related body-filter modules.
>
> Hope it makes sense.
> Patch attached.
>
> Thanks,
>
> /Alon
>
>
> On Wed, Dec 7, 2016 at 5:58 PM Maxim Dounin <mdounin at mdounin.ru> wrote:
>
> > Hello!
> >
> > On Sun, Nov 27, 2016 at 02:27:56PM +0200, Alon Blayer-Gat wrote:
> >
> > > Hi,
> > >
> > > 1) 'gunzip always' option will gunzip even if the client supports it.
> > > 2) 'gunzip types', like 'always' but only for file types specified
> > > with 'gunzip_types <mime-types>'
> > > 3) Allow gunzip and gunzip_types directives within "if in location"
> > > block (rewrite phase condition).
> > >
> > > The suggested changes are needed, mainly, to allow dynamic
> > > modification of compressed response (e.g. with the 'sub_filter'
> > > module)
> > > 'types' and 'if in location' may allow a more selective operation.
> >
> > No, thanks.
> >
> > "If in location" is evil, don't even try to suggest patches to
> > allow directives in the "if in location" context.
> >
> > As for other changes - I can't say I like them as well.  We may
> > consider something as simple as "gunzip always", but additional
> > types filter certainly looks like an overkill.  Rather it should
> > be some more generic mechanism to require gunzipping, may be
> > useable by modules directly.
> >
> > [...]
> >
> > --
> > Maxim Dounin
> > http://nginx.org/
> > _______________________________________________
> > nginx-devel mailing list
> > nginx-devel at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> >
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170123/49408417/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ngx_http_gunzip_filter_module.patch
Type: text/x-patch
Size: 4150 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170123/49408417/attachment.bin>

From igor at sysoev.ru  Tue Jan 24 11:39:12 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Tue, 24 Jan 2017 11:39:12 +0000
Subject: [njs] Version 0.1.8.
Message-ID: <hg.a29f29d48112.1485257952.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/a29f29d48112
branches:  
changeset: 304:a29f29d48112
user:      Igor Sysoev <igor at sysoev.ru>
date:      Tue Jan 24 14:38:48 2017 +0300
description:
Version 0.1.8.

diffstat:

 Makefile |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (10 lines):

diff -r 8401ae77cf40 -r a29f29d48112 Makefile
--- a/Makefile	Fri Jan 20 16:10:48 2017 +0300
+++ b/Makefile	Tue Jan 24 14:38:48 2017 +0300
@@ -1,5 +1,5 @@
 
-NJS_VER =	0.1.7
+NJS_VER =	0.1.8
 
 NXT_LIB =	nxt
 

From igor at sysoev.ru  Tue Jan 24 11:39:13 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Tue, 24 Jan 2017 11:39:13 +0000
Subject: [njs] Added tag 0.1.8 for changeset a29f29d48112
Message-ID: <hg.f136239eebff.1485257953.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/f136239eebff
branches:  
changeset: 305:f136239eebff
user:      Igor Sysoev <igor at sysoev.ru>
date:      Tue Jan 24 14:38:59 2017 +0300
description:
Added tag 0.1.8 for changeset a29f29d48112

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff -r a29f29d48112 -r f136239eebff .hgtags
--- a/.hgtags	Tue Jan 24 14:38:48 2017 +0300
+++ b/.hgtags	Tue Jan 24 14:38:59 2017 +0300
@@ -6,3 +6,4 @@ 508689c1fb94c23f6b24be087c1dc63b2f9e6654
 9c813c2bb2acfd5b6e9d1e9b6699af928baea15a 0.1.5
 44b524f7e313369cd062a387511ea6fdc427875f 0.1.6
 15dc54100400f99c3ec044d8fb0175dd3d69adcb 0.1.7
+a29f29d481125db6101ecdc23dc20187c143cdc9 0.1.8

From mdounin at mdounin.ru  Tue Jan 24 13:59:19 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 24 Jan 2017 13:59:19 +0000
Subject: [nginx] Updated zlib and PCRE used for win32 builds.
Message-ID: <hg.8acc44141e44.1485266359.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/8acc44141e44
branches:  
changeset: 6877:8acc44141e44
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jan 24 16:41:29 2017 +0300
description:
Updated zlib and PCRE used for win32 builds.

diffstat:

 misc/GNUmakefile |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff --git a/misc/GNUmakefile b/misc/GNUmakefile
--- a/misc/GNUmakefile
+++ b/misc/GNUmakefile
@@ -7,8 +7,8 @@ TEMP =		tmp
 CC =		cl
 OBJS =		objs.msvc8
 OPENSSL =	openssl-1.0.2j
-ZLIB =		zlib-1.2.8
-PCRE =		pcre-8.39
+ZLIB =		zlib-1.2.11
+PCRE =		pcre-8.40
 
 
 release: export

From mdounin at mdounin.ru  Tue Jan 24 14:06:20 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 24 Jan 2017 14:06:20 +0000
Subject: [nginx] nginx-1.11.9-RELEASE
Message-ID: <hg.20a45c768e5e.1485266780.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/20a45c768e5e
branches:  
changeset: 6878:20a45c768e5e
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jan 24 17:02:18 2017 +0300
description:
nginx-1.11.9-RELEASE

diffstat:

 docs/xml/nginx/changes.xml |  95 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 95 insertions(+), 0 deletions(-)

diffs (105 lines):

diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml
--- a/docs/xml/nginx/changes.xml
+++ b/docs/xml/nginx/changes.xml
@@ -5,6 +5,101 @@
 <change_log title="nginx">
 
 
+<changes ver="1.11.9" date="24.01.2017">
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ?????? stream nginx ??? ????????? ?????????;
+?????? ????????? ? 1.11.5.
+</para>
+<para lang="en">
+nginx might hog CPU when using the stream module;
+the bug had appeared in 1.11.5.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+????? ?????????????? EXTERNAL ? ???????? ??????-???????
+????? ???? ????????????, ???? ???? ?? ?? ??? ???????? ? ????????????.
+</para>
+<para lang="en">
+EXTERNAL authentication mechanism in mail proxy
+was accepted even if it was not enabled in the configuration.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ????????? ssl_verify_client ?????? stream
+? ??????? ???????? ??? ????????? segmentation fault.
+</para>
+<para lang="en">
+a segmentation fault might occur in a worker process
+if the "ssl_verify_client" directive of the stream module was used.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+????????? ssl_verify_client ?????? stream ????? ?? ????????.
+</para>
+<para lang="en">
+the "ssl_verify_client" directive of the stream module might not work.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ?????????? ??????? ????????? ????????? ??????????
+keepalive-?????????? ????? ??????????? ??????? ??????????.<br/>
+??????? Joel Cunningham.
+</para>
+<para lang="en">
+closing keepalive connections due to no free worker connections
+might be too aggressive.<br/>
+Thanks to Joel Cunningham.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ????????? sendfile ?? FreeBSD ? macOS
+??? ???????????? ???????????? ?????;
+?????? ????????? ? 1.7.8.
+</para>
+<para lang="en">
+an incorrect response might be returned
+when using the "sendfile" directive on FreeBSD and macOS;
+the bug had appeared in 1.7.8.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ????????? aio_write
+????? ??? ??????????? ? ??? ?? ?????????.
+</para>
+<para lang="en">
+a truncated response might be stored in cache
+when using the "aio_write" directive.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ????????? aio_write
+????? ??????????? ?????? ???????.
+</para>
+<para lang="en">
+a socket leak might occur
+when using the "aio_write" directive.
+</para>
+</change>
+
+</changes>
+
+
 <changes ver="1.11.8" date="27.12.2016">
 
 <change type="feature">

From mdounin at mdounin.ru  Tue Jan 24 14:06:22 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 24 Jan 2017 14:06:22 +0000
Subject: [nginx] release-1.11.9 tag
Message-ID: <hg.d84f48e571e4.1485266782.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/d84f48e571e4
branches:  
changeset: 6879:d84f48e571e4
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jan 24 17:02:19 2017 +0300
description:
release-1.11.9 tag

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -407,3 +407,4 @@ 5253015a339aaca0a3111473d3e931b6d4752393
 5e371426b3bcba4312ce08606194b89b758927d1 release-1.11.6
 5c8f60faf33ca8926473d2da27b4c3c417bd4630 release-1.11.7
 4591da489a30f790def29bc5987f43409b503cae release-1.11.8
+20a45c768e5ed26b740679d0e22045c98727c3cc release-1.11.9

From szh.subs at gmail.com  Tue Jan 24 17:35:43 2017
From: szh.subs at gmail.com (szh.subs at gmail.com)
Date: Tue, 24 Jan 2017 20:35:43 +0300
Subject: [nginx] Updated zlib and PCRE used for win32 builds.
In-Reply-To: <hg.8acc44141e44.1485266359.6026610855610030274@dev.nginx.com>
Message-ID: <26d9e4dd-a49d-4b6f-bc69-3dc96e3eb057@email.android.com>

An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20170124/ff7de2f6/attachment.html>

From thibaultcha at fastmail.com  Wed Jan 25 01:56:31 2017
From: thibaultcha at fastmail.com (Thibault Charbonnier)
Date: Tue, 24 Jan 2017 17:56:31 -0800
Subject: [PATCH] Upstream: remove unused struct field
Message-ID: <c016968a-cf21-7235-618f-76d4124deddc@fastmail.com>

Hello!

Quite minor, but as far as I can tell, this field is not being used 
anywhere.

# HG changeset patch
# User Thibault Charbonnier <thibaultcha at fastmail.com>
# Date 1485309159 28800
#      Tue Jan 24 17:52:39 2017 -0800
# Branch remove-unused-upstream-struct-field
# Node ID bb5d6b4762998d1ca8f9ad487fa0e9bf6e0094e9
# Parent  d84f48e571e449ee6c072a8d52cdea8e06b88ef7
Upstream: remove unused struct field

diff -r d84f48e571e4 -r bb5d6b476299 src/http/ngx_http_upstream.h
--- a/src/http/ngx_http_upstream.h    Tue Jan 24 17:02:19 2017 +0300
+++ b/src/http/ngx_http_upstream.h    Tue Jan 24 17:52:39 2017 -0800
@@ -151,7 +151,6 @@
      ngx_msec_t                       connect_timeout;
      ngx_msec_t                       send_timeout;
      ngx_msec_t                       read_timeout;
-    ngx_msec_t                       timeout;
      ngx_msec_t                       next_upstream_timeout;

      size_t                           send_lowat;


From vl at nginx.com  Thu Jan 26 08:49:29 2017
From: vl at nginx.com (Vladimir Homutov)
Date: Thu, 26 Jan 2017 08:49:29 +0000
Subject: [nginx] Version bump.
Message-ID: <hg.b14d4e5a123f.1485420569.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/b14d4e5a123f
branches:  
changeset: 6880:b14d4e5a123f
user:      Vladimir Homutov <vl at nginx.com>
date:      Thu Jan 26 11:44:55 2017 +0300
description:
Version bump.

diffstat:

 src/core/nginx.h |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff -r d84f48e571e4 -r b14d4e5a123f src/core/nginx.h
--- a/src/core/nginx.h	Tue Jan 24 17:02:19 2017 +0300
+++ b/src/core/nginx.h	Thu Jan 26 11:44:55 2017 +0300
@@ -9,8 +9,8 @@
 #define _NGINX_H_INCLUDED_
 
 
-#define nginx_version      1011009
-#define NGINX_VERSION      "1.11.9"
+#define nginx_version      1011010
+#define NGINX_VERSION      "1.11.10"
 #define NGINX_VER          "nginx/" NGINX_VERSION
 
 #ifdef NGX_BUILD

From vl at nginx.com  Thu Jan 26 08:49:32 2017
From: vl at nginx.com (Vladimir Homutov)
Date: Thu, 26 Jan 2017 08:49:32 +0000
Subject: [nginx] Upstream: removed compatibility shims from
 ngx_http_upstream_t.
Message-ID: <hg.7113b9b2dd6c.1485420572.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/7113b9b2dd6c
branches:  
changeset: 6881:7113b9b2dd6c
user:      Vladimir Homutov <vl at nginx.com>
date:      Wed Jan 25 15:39:22 2017 +0300
description:
Upstream: removed compatibility shims from ngx_http_upstream_t.

The type is no longer modified in NGINX Plus.

diffstat:

 src/http/ngx_http_upstream.h |  3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)

diffs (13 lines):

diff -r b14d4e5a123f -r 7113b9b2dd6c src/http/ngx_http_upstream.h
--- a/src/http/ngx_http_upstream.h	Thu Jan 26 11:44:55 2017 +0300
+++ b/src/http/ngx_http_upstream.h	Wed Jan 25 15:39:22 2017 +0300
@@ -390,9 +390,6 @@ struct ngx_http_upstream_s {
     unsigned                         request_sent:1;
     unsigned                         request_body_sent:1;
     unsigned                         header_sent:1;
-
-    NGX_COMPAT_BEGIN(1)
-    NGX_COMPAT_END
 };
 
 

From mdounin at mdounin.ru  Thu Jan 26 13:50:19 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 26 Jan 2017 13:50:19 +0000
Subject: [nginx] Upstream: removed unused ngx_http_upstream_conf_t.timeout
 field.
Message-ID: <hg.a1ec33df7ce0.1485438619.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/a1ec33df7ce0
branches:  
changeset: 6882:a1ec33df7ce0
user:      Thibault Charbonnier <thibaultcha at fastmail.com>
date:      Tue Jan 24 17:52:39 2017 -0800
description:
Upstream: removed unused ngx_http_upstream_conf_t.timeout field.

diffstat:

 src/http/ngx_http_upstream.h |  1 -
 1 files changed, 0 insertions(+), 1 deletions(-)

diffs (11 lines):

diff --git a/src/http/ngx_http_upstream.h b/src/http/ngx_http_upstream.h
--- a/src/http/ngx_http_upstream.h
+++ b/src/http/ngx_http_upstream.h
@@ -151,7 +151,6 @@ typedef struct {
     ngx_msec_t                       connect_timeout;
     ngx_msec_t                       send_timeout;
     ngx_msec_t                       read_timeout;
-    ngx_msec_t                       timeout;
     ngx_msec_t                       next_upstream_timeout;
 
     size_t                           send_lowat;

From mdounin at mdounin.ru  Thu Jan 26 13:50:22 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 26 Jan 2017 13:50:22 +0000
Subject: [nginx] Upstream: removed unused bl_time and bl_state fields.
Message-ID: <hg.8b822ff79916.1485438622.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/8b822ff79916
branches:  
changeset: 6883:8b822ff79916
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Thu Jan 26 16:16:48 2017 +0300
description:
Upstream: removed unused bl_time and bl_state fields.

diffstat:

 src/http/ngx_http_upstream.h |  3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)

diffs (13 lines):

diff --git a/src/http/ngx_http_upstream.h b/src/http/ngx_http_upstream.h
--- a/src/http/ngx_http_upstream.h
+++ b/src/http/ngx_http_upstream.h
@@ -55,9 +55,6 @@
 
 
 typedef struct {
-    ngx_msec_t                       bl_time;
-    ngx_uint_t                       bl_state;
-
     ngx_uint_t                       status;
     ngx_msec_t                       response_time;
     ngx_msec_t                       connect_time;

From mdounin at mdounin.ru  Thu Jan 26 13:51:25 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 26 Jan 2017 16:51:25 +0300
Subject: [PATCH] Upstream: remove unused struct field
In-Reply-To: <c016968a-cf21-7235-618f-76d4124deddc@fastmail.com>
References: <c016968a-cf21-7235-618f-76d4124deddc@fastmail.com>
Message-ID: <20170126135125.GF80190@mdounin.ru>

Hello!

On Tue, Jan 24, 2017 at 05:56:31PM -0800, Thibault Charbonnier wrote:

> Hello!
> 
> Quite minor, but as far as I can tell, this field is not being used 
> anywhere.
> 
> # HG changeset patch
> # User Thibault Charbonnier <thibaultcha at fastmail.com>
> # Date 1485309159 28800
> #      Tue Jan 24 17:52:39 2017 -0800
> # Branch remove-unused-upstream-struct-field
> # Node ID bb5d6b4762998d1ca8f9ad487fa0e9bf6e0094e9
> # Parent  d84f48e571e449ee6c072a8d52cdea8e06b88ef7
> Upstream: remove unused struct field
> 
> diff -r d84f48e571e4 -r bb5d6b476299 src/http/ngx_http_upstream.h
> --- a/src/http/ngx_http_upstream.h    Tue Jan 24 17:02:19 2017 +0300
> +++ b/src/http/ngx_http_upstream.h    Tue Jan 24 17:52:39 2017 -0800
> @@ -151,7 +151,6 @@
>       ngx_msec_t                       connect_timeout;
>       ngx_msec_t                       send_timeout;
>       ngx_msec_t                       read_timeout;
> -    ngx_msec_t                       timeout;
>       ngx_msec_t                       next_upstream_timeout;
> 
>       size_t                           send_lowat;

Thanks, committed (with slightly modified committ log).
And also removed a couple of additional unused fields.

-- 
Maxim Dounin
http://nginx.org/

From mdounin at mdounin.ru  Thu Jan 26 14:01:48 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 26 Jan 2017 17:01:48 +0300
Subject: [PATCH] Http gunzip: additional configuration
In-Reply-To: <CABbx4XDqpRFBnjshK5zU4sQbuu=qOCWgoGARsjOKoNJu3VJY_Q@mail.gmail.com>
References: <CABbx4XDqpRFBnjshK5zU4sQbuu=qOCWgoGARsjOKoNJu3VJY_Q@mail.gmail.com>
Message-ID: <20170126140148.GG80190@mdounin.ru>

Hello!

On Mon, Jan 23, 2017 at 06:23:05AM +0000, Alon Blayer-Gat wrote:

> Patch attached
> 
> On Sun, Jan 22, 2017 at 4:07 PM Alon Blayer-Gat <alon.blayergat at gmail.com>
> wrote:
> 
> > Sure. Thanks for the feedback. I made is simpler.
> >
> > evil 'if in location' removed :). I also removed the 'gunzip types;'
> > option.
> > And now we now only have 'gunzip off|on|always'
> >
> > But then again, with 'always', one must specify with 'gunzip_types' the
> > mime types to always gunzip.
> > The rational is that if we want to always gunzip, then it's probably not
> > because the client does not support it but rather because we would like to
> > modify the response. In which case, it is needed only for specific content
> > types.
> > Default mime type is text/html ( similar to gzip_types).
> >
> > I thought of your suggestion of making a more generic mechanism to be used
> > by modules directly. It would require modifications in existing modules to
> > make the feature usable.
> > This patch suggests a simple change for the benefit of existing
> > text-related body-filter modules.
> >
> > Hope it makes sense.
> > Patch attached.
> >
> > Thanks,
> >
> > /Alon
> >
> >
> > On Wed, Dec 7, 2016 at 5:58 PM Maxim Dounin <mdounin at mdounin.ru> wrote:
> >
> > > Hello!
> > >
> > > On Sun, Nov 27, 2016 at 02:27:56PM +0200, Alon Blayer-Gat wrote:
> > >
> > > > Hi,
> > > >
> > > > 1) 'gunzip always' option will gunzip even if the client supports it.
> > > > 2) 'gunzip types', like 'always' but only for file types specified
> > > > with 'gunzip_types <mime-types>'
> > > > 3) Allow gunzip and gunzip_types directives within "if in location"
> > > > block (rewrite phase condition).
> > > >
> > > > The suggested changes are needed, mainly, to allow dynamic
> > > > modification of compressed response (e.g. with the 'sub_filter'
> > > > module)
> > > > 'types' and 'if in location' may allow a more selective operation.
> > >
> > > No, thanks.
> > >
> > > "If in location" is evil, don't even try to suggest patches to
> > > allow directives in the "if in location" context.
> > >
> > > As for other changes - I can't say I like them as well.  We may
> > > consider something as simple as "gunzip always", but additional
> > > types filter certainly looks like an overkill.  Rather it should
> > > be some more generic mechanism to require gunzipping, may be
> > > useable by modules directly.

[...]

> @@ -140,13 +161,20 @@
>  
>      r->gzip_vary = 1;
>  
> -    if (!r->gzip_tested) {
> -        if (ngx_http_gzip_ok(r) == NGX_OK) {
> -            return ngx_http_next_header_filter(r);
> +    if (conf->enable == NGX_HTTP_GUNZIP_ON) {
> +        if (!r->gzip_tested) {
> +            if (ngx_http_gzip_ok(r) == NGX_OK) {
> +                return ngx_http_next_header_filter(r);
> +            }
> +
> +        } else if (r->gzip_ok) {
> +                   return ngx_http_next_header_filter(r);
>          }
>  
> -    } else if (r->gzip_ok) {
> -        return ngx_http_next_header_filter(r);
> +    } else if (conf->enable == NGX_HTTP_GUNZIP_ALWAYS
> +               && ngx_http_test_content_type(r, &conf->types) == NULL)
> +    {
> +               return ngx_http_next_header_filter(r);
>      }

With such a code the behaviour is going to be very inconsistent, 
for example, if a client does not support gzip:

- "gunzip on" will gunzip everything if gzip is not supported by 
  the client;

- "gunzip always" will _not_ gunzip anything not listed in 
  gunzip_types.

Not to mention that "always" is expected to do it always.  
Introducing additional filtering with such a name is likely to 
confuse users.

As previously suggested, I would rather consider "gunzip always" 
to do exactly what it says, without any filtering.

-- 
Maxim Dounin
http://nginx.org/

From me+lists.nginx at tomthorogood.co.uk  Fri Jan 27 04:16:38 2017
From: me+lists.nginx at tomthorogood.co.uk (Tom Thorogood)
Date: Fri, 27 Jan 2017 14:46:38 +1030
Subject: [PATCH] HTTP: NGINX_VER_BUILD in Server header and error page
 tail.
In-Reply-To: <1462703226.1314805.601366873.679532C0@webmail.messagingengine.com>
References: <1462703226.1314805.601366873.679532C0@webmail.messagingengine.com>
Message-ID: <1485490598.2897802.861127232.499F85E0@webmail.messagingengine.com>

Hi,

Ping. Any chance this might be able to be merged?

Kind Regards,
Tom Thorogood.

On Sun, 8 May 2016, at 08:57 PM, Tom Thorogood wrote:
> Hi,
> 
> This patch adds a build option to the server_tokens directive. When build is specified the Server header will have a value of NGINX_VER_BUILD. Also the tail of built in error pages will be NGINX_VER_BUILD. off and on are retained as valid values and behave as they always have.
> 
> This makes it possible to specify semi-custom server tokens of the form "nginx/<version> (<value>)". <value> is specified with the --build flag for ./configure.
> 
> A potential use case is including the name of a custom build for example with --build="super-nginx/0.1". This would give: nginx/1.9.15 (super-nginx/0.1).
> 
> Another potential use case of this feature is to include the git revision of a parent build with --build="`git rev-parse --short HEAD`". This would give: nginx/1.9.15 (c3ebabd). In that case the feature can be used to enhance troubleshooting and/or provide more information to end users.
> 
> Kind Regards,
> Tom Thorogood.
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
> Email had 1 attachment:
> + nginx-server-token-build.patch
>   15k (text/x-patch)

From igor at sysoev.ru  Fri Jan 27 13:34:23 2017
From: igor at sysoev.ru (Igor Sysoev)
Date: Fri, 27 Jan 2017 13:34:23 +0000
Subject: [njs] Removed obsolete function call interface.
Message-ID: <hg.fb2ec3176a14.1485524063.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/fb2ec3176a14
branches:  
changeset: 306:fb2ec3176a14
user:      Igor Sysoev <igor at sysoev.ru>
date:      Fri Jan 27 16:01:31 2017 +0300
description:
Removed obsolete function call interface.

diffstat:

 nginx/ngx_http_js_module.c   |   2 +-
 nginx/ngx_stream_js_module.c |   2 +-
 njs/njs_generator.c          |   3 ---
 njs/njs_parser.c             |   9 +--------
 njs/njs_parser.h             |   1 -
 njs/njscript.c               |  18 +-----------------
 njs/njscript.h               |   2 +-
 njs/test/njs_unit_test.c     |  29 ++++-------------------------
 8 files changed, 9 insertions(+), 57 deletions(-)

diffs (197 lines):

diff -r f136239eebff -r fb2ec3176a14 nginx/ngx_http_js_module.c
--- a/nginx/ngx_http_js_module.c	Tue Jan 24 14:38:59 2017 +0300
+++ b/nginx/ngx_http_js_module.c	Fri Jan 27 16:01:31 2017 +0300
@@ -1322,7 +1322,7 @@ ngx_http_js_include(ngx_conf_t *cf, ngx_
         return NGX_CONF_ERROR;
     }
 
-    rc = njs_vm_compile(jlcf->vm, &start, end, NULL, &export);
+    rc = njs_vm_compile(jlcf->vm, &start, end, &export);
 
     if (rc != NJS_OK) {
         njs_vm_exception(jlcf->vm, &text);
diff -r f136239eebff -r fb2ec3176a14 nginx/ngx_stream_js_module.c
--- a/nginx/ngx_stream_js_module.c	Tue Jan 24 14:38:59 2017 +0300
+++ b/nginx/ngx_stream_js_module.c	Fri Jan 27 16:01:31 2017 +0300
@@ -1030,7 +1030,7 @@ ngx_stream_js_include(ngx_conf_t *cf, ng
         return NGX_CONF_ERROR;
     }
 
-    rc = njs_vm_compile(jscf->vm, &start, end, NULL, &export);
+    rc = njs_vm_compile(jscf->vm, &start, end, &export);
 
     if (rc != NJS_OK) {
         njs_vm_exception(jscf->vm, &text);
diff -r f136239eebff -r fb2ec3176a14 njs/njs_generator.c
--- a/njs/njs_generator.c	Tue Jan 24 14:38:59 2017 +0300
+++ b/njs/njs_generator.c	Fri Jan 27 16:01:31 2017 +0300
@@ -170,9 +170,6 @@ njs_generator(njs_vm_t *vm, njs_parser_t
     case NJS_TOKEN_END:
         return njs_generate_stop_statement(vm, parser, node);
 
-    case NJS_TOKEN_CALL:
-        return njs_generate_children(vm, parser, node);
-
     case NJS_TOKEN_COMMA:
         return njs_generate_comma_expression(vm, parser, node);
 
diff -r f136239eebff -r fb2ec3176a14 njs/njs_parser.c
--- a/njs/njs_parser.c	Tue Jan 24 14:38:59 2017 +0300
+++ b/njs/njs_parser.c	Fri Jan 27 16:01:31 2017 +0300
@@ -123,14 +123,7 @@ njs_parser(njs_vm_t *vm, njs_parser_t *p
 
     node = parser->node;
 
-    if (node != NULL && node->right != NULL) {
-        if (node->right->token == NJS_TOKEN_FUNCTION) {
-            node->token = NJS_TOKEN_CALL;
-            node->scope = parser->scope;
-            return node;
-        }
-
-    } else {
+    if (node == NULL) {
         /* Empty string, just semicolons or variables declarations. */
 
         node = njs_parser_node_alloc(vm);
diff -r f136239eebff -r fb2ec3176a14 njs/njs_parser.h
--- a/njs/njs_parser.h	Tue Jan 24 14:38:59 2017 +0300
+++ b/njs/njs_parser.h	Fri Jan 27 16:01:31 2017 +0300
@@ -14,7 +14,6 @@ typedef enum {
     NJS_TOKEN_ILLEGAL = 0,
 
     NJS_TOKEN_END,
-    NJS_TOKEN_CALL,
     NJS_TOKEN_SPACE,
     NJS_TOKEN_LINE_END,
 
diff -r f136239eebff -r fb2ec3176a14 njs/njscript.c
--- a/njs/njscript.c	Tue Jan 24 14:38:59 2017 +0300
+++ b/njs/njscript.c	Fri Jan 27 16:01:31 2017 +0300
@@ -184,13 +184,11 @@ njs_vm_destroy(njs_vm_t *vm)
 
 
 nxt_int_t
-njs_vm_compile(njs_vm_t *vm, u_char **start, u_char *end,
-    njs_function_t **function, nxt_str_t **export)
+njs_vm_compile(njs_vm_t *vm, u_char **start, u_char *end, nxt_str_t **export)
 {
     nxt_int_t          ret;
     njs_lexer_t        *lexer;
     njs_parser_t       *parser;
-    njs_variable_t     *var;
     njs_parser_node_t  *node;
 
     parser = nxt_mem_cache_zalloc(vm->mem_cache_pool, sizeof(njs_parser_t));
@@ -219,20 +217,6 @@ njs_vm_compile(njs_vm_t *vm, u_char **st
         return NJS_ERROR;
     }
 
-    if (function != NULL) {
-        if (node->token == NJS_TOKEN_CALL) {
-            var = njs_variable_get(vm, node->right, NJS_NAME_DECLARATION);
-            if (nxt_slow_path(var == NULL)) {
-                return NJS_ERROR;
-            }
-
-            *function = var->value.data.u.function;
-
-        } else {
-            *function = NULL;
-        }
-    }
-
     *start = parser->lexer->start;
 
     ret = njs_generate_scope(vm, parser, node);
diff -r f136239eebff -r fb2ec3176a14 njs/njscript.h
--- a/njs/njscript.h	Tue Jan 24 14:38:59 2017 +0300
+++ b/njs/njscript.h	Fri Jan 27 16:01:31 2017 +0300
@@ -84,7 +84,7 @@ NXT_EXPORT njs_vm_t *njs_vm_create(nxt_m
 NXT_EXPORT void njs_vm_destroy(njs_vm_t *vm);
 
 NXT_EXPORT nxt_int_t njs_vm_compile(njs_vm_t *vm, u_char **start, u_char *end,
-    njs_function_t **function, nxt_str_t **export);
+    nxt_str_t **export);
 NXT_EXPORT njs_vm_t *njs_vm_clone(njs_vm_t *vm, nxt_mem_cache_pool_t *mcp,
     void **external);
 NXT_EXPORT nxt_int_t njs_vm_call(njs_vm_t *vm, njs_function_t *function,
diff -r f136239eebff -r fb2ec3176a14 njs/test/njs_unit_test.c
--- a/njs/test/njs_unit_test.c	Tue Jan 24 14:38:59 2017 +0300
+++ b/njs/test/njs_unit_test.c	Fri Jan 27 16:01:31 2017 +0300
@@ -6758,11 +6758,6 @@ static njs_unit_test_t  njs_test[] =
     { nxt_string("parseFloat('12345abc')"),
       nxt_string("12345") },
 
-    /* External interface. */
-
-    { nxt_string("function f(req) { return req.uri }"),
-      nxt_string("???") },
-
     /* Trick: number to boolean. */
 
     { nxt_string("var a = 0; !!a"),
@@ -7056,14 +7051,12 @@ njs_unit_test(nxt_bool_t disassemble)
     u_char                *start;
     njs_vm_t              *vm, *nvm;
     nxt_int_t             ret;
-    nxt_str_t             s, r_name, *export;
+    nxt_str_t             s, *export;
     nxt_uint_t            i;
     nxt_bool_t            success;
     nxt_lvlhsh_t          externals;
-    njs_function_t        *function;
     njs_vm_shared_t       *shared;
     njs_unit_test_req     r;
-    njs_opaque_value_t    value;
     nxt_mem_cache_pool_t  *mcp;
 
     /*
@@ -7105,7 +7098,7 @@ njs_unit_test(nxt_bool_t disassemble)
         start = njs_test[i].script.start;
 
         ret = njs_vm_compile(vm, &start, start + njs_test[i].script.length,
-                             &function, &export);
+                             &export);
 
         if (ret == NXT_OK) {
             if (disassemble) {
@@ -7121,20 +7114,7 @@ njs_unit_test(nxt_bool_t disassemble)
             r.uri.length = 6;
             r.uri.start = (u_char *) "???";
 
-            if (function != NULL) {
-                r_name.length = 2;
-                r_name.start = (u_char *) "$r";
-
-                ret = njs_vm_external(nvm, NULL, &r_name, &value);
-                if (ret != NXT_OK) {
-                    return NXT_ERROR;
-                }
-
-                ret = njs_vm_call(nvm, function, &value, 1);
-
-            } else {
-                ret = njs_vm_run(nvm);
-            }
+            ret = njs_vm_run(nvm);
 
             if (ret == NXT_OK) {
                 if (njs_vm_retval(nvm, &s) != NXT_OK) {
@@ -7219,7 +7199,7 @@ njs_unit_test_benchmark(nxt_str_t *scrip
 
     start = script->start;
 
-    ret = njs_vm_compile(vm, &start, start + script->length, NULL, &export);
+    ret = njs_vm_compile(vm, &start, start + script->length, &export);
     if (ret != NXT_OK) {
         return NXT_ERROR;
     }
@@ -7232,7 +7212,6 @@ njs_unit_test_benchmark(nxt_str_t *scrip
         }
 
         if (njs_vm_run(nvm) == NXT_OK) {
-
             if (njs_vm_retval(nvm, &s) != NXT_OK) {
                 return NXT_ERROR;
             }

From arut at nginx.com  Fri Jan 27 14:38:26 2017
From: arut at nginx.com (Roman Arutyunyan)
Date: Fri, 27 Jan 2017 14:38:26 +0000
Subject: [njs] Initialize global variables in njs virtual machines.
Message-ID: <hg.e9c89068140d.1485527906.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/e9c89068140d
branches:  
changeset: 307:e9c89068140d
user:      Roman Arutyunyan <arut at nginx.com>
date:      Fri Jan 27 16:45:34 2017 +0300
description:
Initialize global variables in njs virtual machines.

Previously, global variables in njs code remained uninitialized even if an
initializer existed in njs code.  To fix this, njs_vm_run() function is called
right after a virtual machine is cloned.  This call initializes global
variables.

diffstat:

 nginx/ngx_http_js_module.c   |  4 ++++
 nginx/ngx_stream_js_module.c |  4 ++++
 2 files changed, 8 insertions(+), 0 deletions(-)

diffs (28 lines):

diff -r fb2ec3176a14 -r e9c89068140d nginx/ngx_http_js_module.c
--- a/nginx/ngx_http_js_module.c	Fri Jan 27 16:01:31 2017 +0300
+++ b/nginx/ngx_http_js_module.c	Fri Jan 27 16:45:34 2017 +0300
@@ -573,6 +573,10 @@ ngx_http_js_init_vm(ngx_http_request_t *
         return NGX_ERROR;
     }
 
+    if (njs_vm_run(ctx->vm) != NJS_OK) {
+        return NGX_ERROR;
+    }
+
     ctx->args = &jlcf->args[0];
 
     return NGX_OK;
diff -r fb2ec3176a14 -r e9c89068140d nginx/ngx_stream_js_module.c
--- a/nginx/ngx_stream_js_module.c	Fri Jan 27 16:01:31 2017 +0300
+++ b/nginx/ngx_stream_js_module.c	Fri Jan 27 16:45:34 2017 +0300
@@ -668,6 +668,10 @@ ngx_stream_js_init_vm(ngx_stream_session
         return NGX_ERROR;
     }
 
+    if (njs_vm_run(ctx->vm) != NJS_OK) {
+        return NGX_ERROR;
+    }
+
     ctx->arg = &jscf->arg;
 
     return NGX_OK;

From mdounin at mdounin.ru  Fri Jan 27 18:47:04 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 27 Jan 2017 18:47:04 +0000
Subject: [nginx] Updated OpenSSL used for win32 builds.
Message-ID: <hg.640f03529395.1485542824.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/640f03529395
branches:  
changeset: 6884:640f03529395
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jan 27 19:06:35 2017 +0300
description:
Updated OpenSSL used for win32 builds.

diffstat:

 misc/GNUmakefile |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff --git a/misc/GNUmakefile b/misc/GNUmakefile
--- a/misc/GNUmakefile
+++ b/misc/GNUmakefile
@@ -6,7 +6,7 @@ TEMP =		tmp
 
 CC =		cl
 OBJS =		objs.msvc8
-OPENSSL =	openssl-1.0.2j
+OPENSSL =	openssl-1.0.2k
 ZLIB =		zlib-1.2.11
 PCRE =		pcre-8.40
 

From matwey.kornilov at gmail.com  Sat Jan 28 11:08:29 2017
From: matwey.kornilov at gmail.com (Matwey V. Kornilov)
Date: Sat, 28 Jan 2017 14:08:29 +0300
Subject: [PATCH] SSI: Implement #fsize SSI command
Message-ID: <f3bb0258beb24b975b94.1485601709@oak.local>

# HG changeset patch
# User Matwey V. Kornilov <matwey.kornilov at gmail.com>
# Date 1485600652 -10800
#      Sat Jan 28 13:50:52 2017 +0300
# Branch fsize
# Node ID f3bb0258beb24b975b94966a94d45e9a9f5c3c39
# Parent  640f035293959b2d4b0ba5939d954bc517f57f77
SSI: Implement #fsize SSI command

diff -r 640f03529395 -r f3bb0258beb2 src/http/modules/ngx_http_ssi_filter_module.c
--- a/src/http/modules/ngx_http_ssi_filter_module.c	Fri Jan 27 19:06:35 2017 +0300
+++ b/src/http/modules/ngx_http_ssi_filter_module.c	Sat Jan 28 13:50:52 2017 +0300
@@ -89,6 +89,10 @@
     ngx_int_t rc);
 static ngx_int_t ngx_http_ssi_set_variable(ngx_http_request_t *r, void *data,
     ngx_int_t rc);
+static ngx_int_t ngx_http_ssi_fsize(ngx_http_request_t *r,
+    ngx_http_ssi_ctx_t *ctx, ngx_str_t **params);
+static ngx_int_t ngx_http_ssi_fsize_output(ngx_http_request_t *r, void *data,
+    ngx_int_t rc);
 static ngx_int_t ngx_http_ssi_echo(ngx_http_request_t *r,
     ngx_http_ssi_ctx_t *ctx, ngx_str_t **params);
 static ngx_int_t ngx_http_ssi_config(ngx_http_request_t *r,
@@ -211,6 +215,7 @@
 
 
 static u_char ngx_http_ssi_string[] = "<!--";
+static u_char ngx_http_ssi_si_prefix[] = " kMGTPEZY";
 
 static ngx_str_t ngx_http_ssi_none = ngx_string("(none)");
 static ngx_str_t ngx_http_ssi_timefmt = ngx_string("%A, %d-%b-%Y %H:%M:%S %Z");
@@ -223,12 +228,16 @@
 #define  NGX_HTTP_SSI_INCLUDE_SET      3
 #define  NGX_HTTP_SSI_INCLUDE_STUB     4
 
+#define  NGX_HTTP_SSI_FSIZE_VIRTUAL    0
+#define  NGX_HTTP_SSI_FSIZE_FILE       1
+
 #define  NGX_HTTP_SSI_ECHO_VAR         0
 #define  NGX_HTTP_SSI_ECHO_DEFAULT     1
 #define  NGX_HTTP_SSI_ECHO_ENCODING    2
 
 #define  NGX_HTTP_SSI_CONFIG_ERRMSG    0
 #define  NGX_HTTP_SSI_CONFIG_TIMEFMT   1
+#define  NGX_HTTP_SSI_CONFIG_SIZEFMT   2
 
 #define  NGX_HTTP_SSI_SET_VAR          0
 #define  NGX_HTTP_SSI_SET_VALUE        1
@@ -248,6 +257,13 @@
 };
 
 
+static ngx_http_ssi_param_t  ngx_http_ssi_fsize_params[] = {
+    { ngx_string("virtual"), NGX_HTTP_SSI_FSIZE_VIRTUAL, 0, 0 },
+    { ngx_string("file"), NGX_HTTP_SSI_FSIZE_FILE, 0, 0 },
+    { ngx_null_string, 0, 0, 0 }
+};
+
+
 static ngx_http_ssi_param_t  ngx_http_ssi_echo_params[] = {
     { ngx_string("var"), NGX_HTTP_SSI_ECHO_VAR, 1, 0 },
     { ngx_string("default"), NGX_HTTP_SSI_ECHO_DEFAULT, 0, 0 },
@@ -259,6 +275,7 @@
 static ngx_http_ssi_param_t  ngx_http_ssi_config_params[] = {
     { ngx_string("errmsg"), NGX_HTTP_SSI_CONFIG_ERRMSG, 0, 0 },
     { ngx_string("timefmt"), NGX_HTTP_SSI_CONFIG_TIMEFMT, 0, 0 },
+    { ngx_string("sizefmt"), NGX_HTTP_SSI_CONFIG_SIZEFMT, 0, 0 },
     { ngx_null_string, 0, 0, 0 }
 };
 
@@ -290,6 +307,8 @@
 static ngx_http_ssi_command_t  ngx_http_ssi_commands[] = {
     { ngx_string("include"), ngx_http_ssi_include,
                        ngx_http_ssi_include_params, 0, 0, 1 },
+    { ngx_string("fsize"), ngx_http_ssi_fsize,
+                       ngx_http_ssi_fsize_params, 0, 0, 1 },
     { ngx_string("echo"), ngx_http_ssi_echo,
                        ngx_http_ssi_echo_params, 0, 0, 0 },
     { ngx_string("config"), ngx_http_ssi_config,
@@ -2239,6 +2258,116 @@
     return rc;
 }
 
+static ngx_int_t ngx_http_ssi_fsize(ngx_http_request_t *r,
+    ngx_http_ssi_ctx_t *ctx, ngx_str_t **params)
+{
+    ngx_int_t                    rc;
+    ngx_str_t                   *uri, *file, args;
+    ngx_uint_t                   flags;
+    ngx_http_request_t          *sr;
+    ngx_http_post_subrequest_t  *psr;
+
+    uri = params[NGX_HTTP_SSI_FSIZE_VIRTUAL];
+    file = params[NGX_HTTP_SSI_FSIZE_FILE];
+
+    if (uri && file) {
+        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+                      "fsize may be either virtual=\"%V\" or file=\"%V\"",
+                      uri, file);
+        return NGX_HTTP_SSI_ERROR;
+    }
+
+    if (uri == NULL && file == NULL) {
+        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+                      "no parameter in \"fsize\" SSI command");
+        return NGX_HTTP_SSI_ERROR;
+    }
+
+    if (uri == NULL) {
+        uri = file;
+    }
+
+    rc = ngx_http_ssi_evaluate_string(r, ctx, uri, NGX_HTTP_SSI_ADD_PREFIX);
+
+    if (rc != NGX_OK) {
+        return rc;
+    }
+
+    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                   "ssi fsize: \"%V\"", uri);
+
+    ngx_str_null(&args);
+    flags = NGX_HTTP_LOG_UNSAFE;
+
+    if (ngx_http_parse_unsafe_uri(r, uri, &args, &flags) != NGX_OK) {
+        return NGX_HTTP_SSI_ERROR;
+    }
+
+    psr = ngx_palloc(r->pool, sizeof(ngx_http_post_subrequest_t));
+    if (psr == NULL) {
+        return NGX_ERROR;
+    }
+
+    psr->handler = ngx_http_ssi_fsize_output;
+    psr->data = ctx;
+
+    if (ngx_http_subrequest(r, uri, &args, &sr, psr, flags) != NGX_OK) {
+        return NGX_HTTP_SSI_ERROR;
+    }
+    sr->header_only = 1;
+
+    return NGX_OK;
+}
+
+
+static ngx_int_t ngx_http_ssi_fsize_output(ngx_http_request_t *r, void *data,
+    ngx_int_t rc)
+{
+    ngx_chain_t        *out;
+    ngx_buf_t          *b;
+    u_char             *p;
+    size_t              len, i;
+    ngx_http_ssi_ctx_t *ctx;
+    unsigned            sizefmt_bytes;
+
+    ctx = data;
+    sizefmt_bytes = (ctx ? ctx->sizefmt_bytes : 0);
+
+    p = ngx_pnalloc(r->pool, NGX_OFF_T_LEN + 1);
+    if (p == NULL) {
+        return NGX_ERROR;
+    }
+
+    len = ngx_sprintf(p, "%O", r->headers_out.content_length_n) -  p;
+
+    if (!sizefmt_bytes) {
+        i = len / 3;
+        len = len % 3;
+
+        p[len] = ngx_http_ssi_si_prefix[i];
+        len++;
+    }
+
+    b = ngx_calloc_buf(r->pool);
+    if (b == NULL) {
+        return NGX_ERROR;
+    }
+
+    b->memory = 1;
+    b->pos = p;
+    b->last = p + len;
+
+    out= ngx_alloc_chain_link(r->pool);
+    if (out == NULL) {
+        return NGX_ERROR;
+    }
+
+    out->buf = b;
+    out->next = NULL;
+
+    return ngx_http_output_filter(r, out);
+}
+
 
 static ngx_int_t
 ngx_http_ssi_echo(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx,
@@ -2400,6 +2529,22 @@
         ctx->errmsg = *value;
     }
 
+    value = params[NGX_HTTP_SSI_CONFIG_SIZEFMT];
+
+    if (value) {
+       if (value->len == 5
+           && ngx_strncasecmp(value->data, (u_char*)"bytes", 5) == 0) {
+           ctx->sizefmt_bytes = 1;
+       } else if (value->len == 6
+                  && ngx_strncasecmp(value->data, (u_char*)"abbrev", 6) == 0) {
+           ctx->sizefmt_bytes = 0;
+       } else {
+           ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+                      "#config sizefmt may be either \"bytes\" or \"abbrev\"");
+           return NGX_HTTP_SSI_ERROR;
+       }
+    }
+
     return NGX_OK;
 }
 
diff -r 640f03529395 -r f3bb0258beb2 src/http/modules/ngx_http_ssi_filter_module.h
--- a/src/http/modules/ngx_http_ssi_filter_module.h	Fri Jan 27 19:06:35 2017 +0300
+++ b/src/http/modules/ngx_http_ssi_filter_module.h	Sat Jan 28 13:50:52 2017 +0300
@@ -76,6 +76,7 @@
     unsigned                  block:1;
     unsigned                  output:1;
     unsigned                  output_chosen:1;
+    unsigned                  sizefmt_bytes:1;
 
     ngx_http_request_t       *wait;
     void                     *value_buf;

From mdounin at mdounin.ru  Mon Jan 30 14:26:45 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 30 Jan 2017 17:26:45 +0300
Subject: [PATCH] SSI: Implement #fsize SSI command
In-Reply-To: <f3bb0258beb24b975b94.1485601709@oak.local>
References: <f3bb0258beb24b975b94.1485601709@oak.local>
Message-ID: <20170130142645.GD46625@mdounin.ru>

Hello!

On Sat, Jan 28, 2017 at 02:08:29PM +0300, Matwey V. Kornilov wrote:

> # HG changeset patch
> # User Matwey V. Kornilov <matwey.kornilov at gmail.com>
> # Date 1485600652 -10800
> #      Sat Jan 28 13:50:52 2017 +0300
> # Branch fsize
> # Node ID f3bb0258beb24b975b94966a94d45e9a9f5c3c39
> # Parent  640f035293959b2d4b0ba5939d954bc517f57f77
> SSI: Implement #fsize SSI command

Thanks, looks interesting, though certainly it needs more work.  
Some comments below.

[...]

> +static ngx_int_t ngx_http_ssi_fsize(ngx_http_request_t *r,
> +    ngx_http_ssi_ctx_t *ctx, ngx_str_t **params)

Style:
The function type should be on a line by itself preceding the 
function.  That is, function name should start on its own line:

static ngx_int_t
ngx_http_ssi_fsize(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx,
    ngx_str_t **params)

[...]

> +    psr->handler = ngx_http_ssi_fsize_output;
> +    psr->data = ctx;
> +
> +    if (ngx_http_subrequest(r, uri, &args, &sr, psr, flags) != NGX_OK) {
> +        return NGX_HTTP_SSI_ERROR;
> +    }
> +    sr->header_only = 1;
> +
> +    return NGX_OK;
> +}
> +
> +
> +static ngx_int_t ngx_http_ssi_fsize_output(ngx_http_request_t *r, void *data,
> +    ngx_int_t rc)
> +{
> +    ngx_chain_t        *out;
> +    ngx_buf_t          *b;
> +    u_char             *p;
> +    size_t              len, i;
> +    ngx_http_ssi_ctx_t *ctx;
> +    unsigned            sizefmt_bytes;
> +
> +    ctx = data;
> +    sizefmt_bytes = (ctx ? ctx->sizefmt_bytes : 0);

Just a side note:
It is not clear how ctx can be NULL here.

> +
> +    p = ngx_pnalloc(r->pool, NGX_OFF_T_LEN + 1);
> +    if (p == NULL) {
> +        return NGX_ERROR;
> +    }
> +
> +    len = ngx_sprintf(p, "%O", r->headers_out.content_length_n) -  p;

Note that r->headers_out.content_length_n may be -1 here.  Either 
because it is not known at all (this can happen for proxied 
requests), or because it was cleared/modified by a filter 
expecting to change the response.  In either case "-1" is probably 
not a correct value to print here.

It might also be a good idea to find out how to prevent filters 
from clearing the length and/or how to save it earlier.

> +
> +    if (!sizefmt_bytes) {
> +        i = len / 3;
> +        len = len % 3;
> +
> +        p[len] = ngx_http_ssi_si_prefix[i];
> +        len++;
> +    }
> +
> +    b = ngx_calloc_buf(r->pool);
> +    if (b == NULL) {
> +        return NGX_ERROR;
> +    }
> +
> +    b->memory = 1;
> +    b->pos = p;
> +    b->last = p + len;

Just a side note:
Using ngx_create_temp_buf() might be more appropriate here.

> +
> +    out= ngx_alloc_chain_link(r->pool);
> +    if (out == NULL) {
> +        return NGX_ERROR;
> +    }
> +
> +    out->buf = b;
> +    out->next = NULL;
> +
> +    return ngx_http_output_filter(r, out);

Not sure if it's ok to declare a subrequest as a header_only one, 
and then output a body for it.  I suspect it may break in 
unexpected places.

Alternative approach would be to wait for the subrequest and then 
output the result in the main request context, much like it is 
done for "include set".

> @@ -2400,6 +2529,22 @@
>          ctx->errmsg = *value;
>      }
>  
> +    value = params[NGX_HTTP_SSI_CONFIG_SIZEFMT];
> +
> +    if (value) {
> +       if (value->len == 5
> +           && ngx_strncasecmp(value->data, (u_char*)"bytes", 5) == 0) {
> +           ctx->sizefmt_bytes = 1;

Style:
The "{" should be on its own line.  And the cast needs more 
spaces,

       if (value->len == 5
           && ngx_strncasecmp(value->data, (u_char *) "bytes", 5) == 0)
       {
           ctx->sizefmt_bytes = 1;

-- 
Maxim Dounin
http://nginx.org/

From thibaultcha at fastmail.com  Tue Jan 31 01:39:13 2017
From: thibaultcha at fastmail.com (Thibault Charbonnier)
Date: Mon, 30 Jan 2017 17:39:13 -0800
Subject: [PATCH] Proxy: make timeout directives accept variables
Message-ID: <9d03643e-0798-3c7a-a793-5b739c28c2e2@fastmail.com>

Hello!

This is a proposal to allow for better granularity (eventually, on a 
per-request basis?) regarding the proxy_*_timeout directives.

Please let me know how you feel about such a feature, and if you think 
this patch should be implemented in a different way.

Thanks!

# HG changeset patch
# User Thibault Charbonnier <thibaultcha at fastmail.com>
# Date 1485823618 28800
#      Mon Jan 30 16:46:58 2017 -0800
# Branch proxy/variables-in-timeouts
# Node ID 9f27c1d40590566ae5ac55e04af473d5f8e97d87
# Parent  640f035293959b2d4b0ba5939d954bc517f57f77
Proxy: make timeout directives accept variables

We try to stay as close as possible from the original behavior regarding the
parsing of the ngx_msec_t fields, to preserve the proper parsing error logs.
As long as the timeout directives do not contain a variable, the behavior is
identical as prior to this patch.

If a timeout directive contains a variable, we consider such variable to
contain the related timeout in ms precision.

diff -r 640f03529395 -r 9f27c1d40590 
src/http/modules/ngx_http_proxy_module.c
--- a/src/http/modules/ngx_http_proxy_module.c    Fri Jan 27 19:06:35 
2017 +0300
+++ b/src/http/modules/ngx_http_proxy_module.c    Mon Jan 30 16:46:58 
2017 -0800
@@ -45,6 +45,16 @@


  typedef struct {
+    ngx_uint_t                     complex;
+
+    union {
+        ngx_msec_t                 simple;
+        ngx_http_complex_value_t  *complex;
+    } value;
+} ngx_http_proxy_timeout_t;
+
+
+typedef struct {
      ngx_array_t                   *flushes;
      ngx_array_t                   *lengths;
      ngx_array_t                   *values;
@@ -77,6 +87,10 @@
      ngx_str_t                      location;
      ngx_str_t                      url;

+    ngx_http_proxy_timeout_t      *connect_timeout;
+    ngx_http_proxy_timeout_t      *send_timeout;
+    ngx_http_proxy_timeout_t      *read_timeout;
+
  #if (NGX_HTTP_CACHE)
      ngx_http_complex_value_t       cache_key;
  #endif
@@ -171,6 +185,8 @@
  static ngx_int_t ngx_http_proxy_init_headers(ngx_conf_t *cf,
      ngx_http_proxy_loc_conf_t *conf, ngx_http_proxy_headers_t *headers,
      ngx_keyval_t *default_headers);
+static ngx_int_t ngx_http_proxy_compile_timeout(ngx_http_request_t *r,
+    ngx_http_complex_value_t *c_timeout, ngx_msec_t *u_timeout);

  static char *ngx_http_proxy_pass(ngx_conf_t *cf, ngx_command_t *cmd,
      void *conf);
@@ -182,6 +198,8 @@
      void *conf);
  static char *ngx_http_proxy_store(ngx_conf_t *cf, ngx_command_t *cmd,
      void *conf);
+static char * ngx_http_proxy_timeout(ngx_conf_t *cf, ngx_command_t *cmd,
+    void *conf);
  #if (NGX_HTTP_CACHE)
  static char *ngx_http_proxy_cache(ngx_conf_t *cf, ngx_command_t *cmd,
      void *conf);
@@ -324,16 +342,16 @@

      { ngx_string("proxy_connect_timeout"),
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
-      ngx_conf_set_msec_slot,
+      ngx_http_proxy_timeout,
        NGX_HTTP_LOC_CONF_OFFSET,
-      offsetof(ngx_http_proxy_loc_conf_t, upstream.connect_timeout),
+      offsetof(ngx_http_proxy_loc_conf_t, connect_timeout),
        NULL },

      { ngx_string("proxy_send_timeout"),
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
-      ngx_conf_set_msec_slot,
+      ngx_http_proxy_timeout,
        NGX_HTTP_LOC_CONF_OFFSET,
-      offsetof(ngx_http_proxy_loc_conf_t, upstream.send_timeout),
+      offsetof(ngx_http_proxy_loc_conf_t, send_timeout),
        NULL },

      { ngx_string("proxy_send_lowat"),
@@ -408,9 +426,9 @@

      { ngx_string("proxy_read_timeout"),
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
-      ngx_conf_set_msec_slot,
+      ngx_http_proxy_timeout,
        NGX_HTTP_LOC_CONF_OFFSET,
-      offsetof(ngx_http_proxy_loc_conf_t, upstream.read_timeout),
+      offsetof(ngx_http_proxy_loc_conf_t, read_timeout),
        NULL },

      { ngx_string("proxy_buffers"),
@@ -829,6 +847,11 @@
  };


+static ngx_http_proxy_timeout_t  ngx_http_proxy_default_timeout = {
+    0, { 60000 }
+};
+
+
  static ngx_int_t
  ngx_http_proxy_handler(ngx_http_request_t *r)
  {
@@ -1137,6 +1160,7 @@
      size_t                        len, uri_len, loc_len, body_len;
      uintptr_t                     escape;
      ngx_buf_t                    *b;
+    ngx_int_t                     rc;
      ngx_str_t                     method;
      ngx_uint_t                    i, unparsed_uri;
      ngx_chain_t                  *cl, *body;
@@ -1160,6 +1184,33 @@
      headers = &plcf->headers;
  #endif

+    if (plcf->connect_timeout->complex) {
+        rc = ngx_http_proxy_compile_timeout(r,
+ plcf->connect_timeout->value.complex,
+ &u->conf->connect_timeout);
+        if (rc != NGX_OK) {
+            return rc;
+        }
+    }
+
+    if (plcf->send_timeout->complex) {
+        rc = ngx_http_proxy_compile_timeout(r,
+ plcf->send_timeout->value.complex,
+ &u->conf->send_timeout);
+        if (rc != NGX_OK) {
+            return rc;
+        }
+    }
+
+    if (plcf->read_timeout->complex) {
+        rc = ngx_http_proxy_compile_timeout(r,
+ plcf->read_timeout->value.complex,
+ &u->conf->read_timeout);
+        if (rc != NGX_OK) {
+            return rc;
+        }
+    }
+
      if (u->method.len) {
          /* HEAD was changed to GET to cache response */
          method = u->method;
@@ -2805,6 +2856,9 @@
       *     conf->upstream.ssl_name = NULL;
       *
       *     conf->method = NULL;
+     *     conf->connect_timeout = NULL;
+     *     conf->send_timeout = NULL;
+     *     conf->read_timeout = NULL;
       *     conf->headers_source = NULL;
       *     conf->headers.lengths = NULL;
       *     conf->headers.values = NULL;
@@ -2953,14 +3007,35 @@
      ngx_conf_merge_ptr_value(conf->upstream.local,
                                prev->upstream.local, NULL);

-    ngx_conf_merge_msec_value(conf->upstream.connect_timeout,
-                              prev->upstream.connect_timeout, 60000);
-
-    ngx_conf_merge_msec_value(conf->upstream.send_timeout,
-                              prev->upstream.send_timeout, 60000);
-
-    ngx_conf_merge_msec_value(conf->upstream.read_timeout,
-                              prev->upstream.read_timeout, 60000);
+    if (conf->connect_timeout == NULL) {
+        conf->connect_timeout = (prev->connect_timeout != NULL)
+                                    ? prev->connect_timeout
+                                    : &ngx_http_proxy_default_timeout;
+    }
+
+    if (!conf->connect_timeout->complex) {
+        conf->upstream.connect_timeout = 
conf->connect_timeout->value.simple;
+    }
+
+    if (conf->send_timeout == NULL) {
+        conf->send_timeout = (prev->send_timeout != NULL)
+                                    ? prev->send_timeout
+                                    : &ngx_http_proxy_default_timeout;
+    }
+
+    if (!conf->send_timeout->complex) {
+        conf->upstream.send_timeout = conf->send_timeout->value.simple;
+    }
+
+    if (conf->read_timeout == NULL) {
+        conf->read_timeout = (prev->read_timeout != NULL)
+                                    ? prev->read_timeout
+                                    : &ngx_http_proxy_default_timeout;
+    }
+
+    if (!conf->read_timeout->complex) {
+        conf->upstream.read_timeout = conf->read_timeout->value.simple;
+    }

ngx_conf_merge_msec_value(conf->upstream.next_upstream_timeout,
prev->upstream.next_upstream_timeout, 0);
@@ -3746,6 +3821,61 @@


  static char *
+ngx_http_proxy_timeout(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
+{
+    char                               *p = conf;
+
+    ngx_msec_t                          msp;
+    ngx_str_t                          *value;
+    ngx_http_proxy_timeout_t          **t;
+    ngx_http_complex_value_t           *cv;
+    ngx_http_compile_complex_value_t    ccv;
+
+    t = (ngx_http_proxy_timeout_t **) (p + cmd->offset);
+    if (*t != NULL) {
+        return "is duplicate";
+    }
+
+    *t = ngx_palloc(cf->pool, sizeof(ngx_http_proxy_timeout_t));
+    if (*t == NULL) {
+        return NGX_CONF_ERROR;
+    }
+
+    value = cf->args->elts;
+
+    if (ngx_http_script_variables_count(&value[1]) > 0) {
+        cv = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t));
+        if (cv == NULL) {
+            return NGX_CONF_ERROR;
+        }
+
+        ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));
+
+        ccv.cf = cf;
+        ccv.value = &value[1];
+        ccv.complex_value = cv;
+
+        if (ngx_http_compile_complex_value(&ccv) != NGX_OK) {
+            return NGX_CONF_ERROR;
+        }
+
+        (*t)->complex = 1;
+        (*t)->value.complex = cv;
+
+    } else {
+        msp = ngx_parse_time(&value[1], 0);
+        if (msp == (ngx_msec_t) NGX_ERROR) {
+            return "invalid value";
+        }
+
+        (*t)->value.simple = msp;
+    }
+
+    return NGX_CONF_OK;
+}
+
+
+static char *
  ngx_http_proxy_redirect(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
  {
      ngx_http_proxy_loc_conf_t *plcf = conf;
@@ -4410,3 +4540,29 @@

      v->uri = u->uri;
  }
+
+
+static ngx_int_t
+ngx_http_proxy_compile_timeout(ngx_http_request_t *r,
+    ngx_http_complex_value_t *c_timeout, ngx_msec_t *u_timeout)
+{
+    ngx_str_t            timeout;
+    ngx_msec_t           msp;
+
+    if (ngx_http_complex_value(r, c_timeout, &timeout) != NGX_OK) {
+        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+                      "could not compile timeout value");
+        return NGX_ERROR;
+    }
+
+    msp = ngx_atoi(timeout.data, timeout.len);
+    if (msp == (ngx_msec_t) NGX_ERROR) {
+        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+                      "invalid timeout value: \"%V\"", &timeout);
+        return NGX_ERROR;
+    }
+
+    *u_timeout = msp;
+
+    return NGX_OK;
+}

From ru at nginx.com  Tue Jan 31 09:16:36 2017
From: ru at nginx.com (Ruslan Ermilov)
Date: Tue, 31 Jan 2017 09:16:36 +0000
Subject: [nginx] Implemented the "server_tokens build" option.
Message-ID: <hg.25203fc377fb.1485854196.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/25203fc377fb
branches:  
changeset: 6885:25203fc377fb
user:      Ruslan Ermilov <ru at nginx.com>
date:      Tue Jan 31 12:09:40 2017 +0300
description:
Implemented the "server_tokens build" option.

Based on a patch by Tom Thorogood.

diffstat:

 src/http/ngx_http_core_module.c          |  20 ++++++++++---
 src/http/ngx_http_core_module.h          |   7 ++++-
 src/http/ngx_http_header_filter_module.c |  26 +++++++++++++----
 src/http/ngx_http_special_response.c     |  13 ++++++++-
 src/http/v2/ngx_http_v2_filter_module.c  |  45 ++++++++++++++++++++++++++++---
 5 files changed, 92 insertions(+), 19 deletions(-)

diffs (248 lines):

diff -r 640f03529395 -r 25203fc377fb src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c	Fri Jan 27 19:06:35 2017 +0300
+++ b/src/http/ngx_http_core_module.c	Tue Jan 31 12:09:40 2017 +0300
@@ -120,6 +120,14 @@ static ngx_conf_enum_t  ngx_http_core_li
 };
 
 
+static ngx_conf_enum_t  ngx_http_core_server_tokens[] = {
+    { ngx_string("off"), NGX_HTTP_SERVER_TOKENS_OFF },
+    { ngx_string("on"), NGX_HTTP_SERVER_TOKENS_ON },
+    { ngx_string("build"), NGX_HTTP_SERVER_TOKENS_BUILD },
+    { ngx_null_string, 0 }
+};
+
+
 static ngx_conf_enum_t  ngx_http_core_if_modified_since[] = {
     { ngx_string("off"), NGX_HTTP_IMS_OFF },
     { ngx_string("exact"), NGX_HTTP_IMS_EXACT },
@@ -599,11 +607,11 @@ static ngx_command_t  ngx_http_core_comm
       NULL },
 
     { ngx_string("server_tokens"),
-      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG,
-      ngx_conf_set_flag_slot,
+      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
+      ngx_conf_set_enum_slot,
       NGX_HTTP_LOC_CONF_OFFSET,
       offsetof(ngx_http_core_loc_conf_t, server_tokens),
-      NULL },
+      &ngx_http_core_server_tokens },
 
     { ngx_string("if_modified_since"),
       NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
@@ -3576,9 +3584,9 @@ ngx_http_core_create_loc_conf(ngx_conf_t
     clcf->log_not_found = NGX_CONF_UNSET;
     clcf->log_subrequest = NGX_CONF_UNSET;
     clcf->recursive_error_pages = NGX_CONF_UNSET;
-    clcf->server_tokens = NGX_CONF_UNSET;
     clcf->chunked_transfer_encoding = NGX_CONF_UNSET;
     clcf->etag = NGX_CONF_UNSET;
+    clcf->server_tokens = NGX_CONF_UNSET_UINT;
     clcf->types_hash_max_size = NGX_CONF_UNSET_UINT;
     clcf->types_hash_bucket_size = NGX_CONF_UNSET_UINT;
 
@@ -3842,11 +3850,13 @@ ngx_http_core_merge_loc_conf(ngx_conf_t 
     ngx_conf_merge_value(conf->log_subrequest, prev->log_subrequest, 0);
     ngx_conf_merge_value(conf->recursive_error_pages,
                               prev->recursive_error_pages, 0);
-    ngx_conf_merge_value(conf->server_tokens, prev->server_tokens, 1);
     ngx_conf_merge_value(conf->chunked_transfer_encoding,
                               prev->chunked_transfer_encoding, 1);
     ngx_conf_merge_value(conf->etag, prev->etag, 1);
 
+    ngx_conf_merge_uint_value(conf->server_tokens, prev->server_tokens,
+                              NGX_HTTP_SERVER_TOKENS_ON);
+
     ngx_conf_merge_ptr_value(conf->open_file_cache,
                               prev->open_file_cache, NULL);
 
diff -r 640f03529395 -r 25203fc377fb src/http/ngx_http_core_module.h
--- a/src/http/ngx_http_core_module.h	Fri Jan 27 19:06:35 2017 +0300
+++ b/src/http/ngx_http_core_module.h	Tue Jan 31 12:09:40 2017 +0300
@@ -55,6 +55,11 @@ typedef struct ngx_thread_pool_s  ngx_th
 #define NGX_HTTP_KEEPALIVE_DISABLE_SAFARI  0x0008
 
 
+#define NGX_HTTP_SERVER_TOKENS_OFF      0
+#define NGX_HTTP_SERVER_TOKENS_ON       1
+#define NGX_HTTP_SERVER_TOKENS_BUILD    2
+
+
 typedef struct ngx_http_location_tree_node_s  ngx_http_location_tree_node_t;
 typedef struct ngx_http_core_loc_conf_s  ngx_http_core_loc_conf_t;
 
@@ -393,7 +398,7 @@ struct ngx_http_core_loc_conf_s {
     ngx_flag_t    log_not_found;           /* log_not_found */
     ngx_flag_t    log_subrequest;          /* log_subrequest */
     ngx_flag_t    recursive_error_pages;   /* recursive_error_pages */
-    ngx_flag_t    server_tokens;           /* server_tokens */
+    ngx_uint_t    server_tokens;           /* server_tokens */
     ngx_flag_t    chunked_transfer_encoding; /* chunked_transfer_encoding */
     ngx_flag_t    etag;                    /* etag */
 
diff -r 640f03529395 -r 25203fc377fb src/http/ngx_http_header_filter_module.c
--- a/src/http/ngx_http_header_filter_module.c	Fri Jan 27 19:06:35 2017 +0300
+++ b/src/http/ngx_http_header_filter_module.c	Tue Jan 31 12:09:40 2017 +0300
@@ -46,8 +46,9 @@ ngx_module_t  ngx_http_header_filter_mod
 };
 
 
-static char ngx_http_server_string[] = "Server: nginx" CRLF;
-static char ngx_http_server_full_string[] = "Server: " NGINX_VER CRLF;
+static u_char ngx_http_server_string[] = "Server: nginx" CRLF;
+static u_char ngx_http_server_full_string[] = "Server: " NGINX_VER CRLF;
+static u_char ngx_http_server_build_string[] = "Server: " NGINX_VER_BUILD CRLF;
 
 
 static ngx_str_t ngx_http_status_lines[] = {
@@ -274,8 +275,15 @@ ngx_http_header_filter(ngx_http_request_
     clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
 
     if (r->headers_out.server == NULL) {
-        len += clcf->server_tokens ? sizeof(ngx_http_server_full_string) - 1:
-                                     sizeof(ngx_http_server_string) - 1;
+        if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_ON) {
+            len += sizeof(ngx_http_server_full_string) - 1;
+
+        } else if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_BUILD) {
+            len += sizeof(ngx_http_server_build_string) - 1;
+
+        } else {
+            len += sizeof(ngx_http_server_string) - 1;
+        }
     }
 
     if (r->headers_out.date == NULL) {
@@ -436,12 +444,16 @@ ngx_http_header_filter(ngx_http_request_
     *b->last++ = CR; *b->last++ = LF;
 
     if (r->headers_out.server == NULL) {
-        if (clcf->server_tokens) {
-            p = (u_char *) ngx_http_server_full_string;
+        if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_ON) {
+            p = ngx_http_server_full_string;
             len = sizeof(ngx_http_server_full_string) - 1;
 
+        } else if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_BUILD) {
+            p = ngx_http_server_build_string;
+            len = sizeof(ngx_http_server_build_string) - 1;
+
         } else {
-            p = (u_char *) ngx_http_server_string;
+            p = ngx_http_server_string;
             len = sizeof(ngx_http_server_string) - 1;
         }
 
diff -r 640f03529395 -r 25203fc377fb src/http/ngx_http_special_response.c
--- a/src/http/ngx_http_special_response.c	Fri Jan 27 19:06:35 2017 +0300
+++ b/src/http/ngx_http_special_response.c	Tue Jan 31 12:09:40 2017 +0300
@@ -25,6 +25,13 @@ static u_char ngx_http_error_full_tail[]
 ;
 
 
+static u_char ngx_http_error_build_tail[] =
+"<hr><center>" NGINX_VER_BUILD "</center>" CRLF
+"</body>" CRLF
+"</html>" CRLF
+;
+
+
 static u_char ngx_http_error_tail[] =
 "<hr><center>nginx</center>" CRLF
 "</body>" CRLF
@@ -628,10 +635,14 @@ ngx_http_send_special_response(ngx_http_
     ngx_uint_t    msie_padding;
     ngx_chain_t   out[3];
 
-    if (clcf->server_tokens) {
+    if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_ON) {
         len = sizeof(ngx_http_error_full_tail) - 1;
         tail = ngx_http_error_full_tail;
 
+    } else if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_BUILD) {
+        len = sizeof(ngx_http_error_build_tail) - 1;
+        tail = ngx_http_error_build_tail;
+
     } else {
         len = sizeof(ngx_http_error_tail) - 1;
         tail = ngx_http_error_tail;
diff -r 640f03529395 -r 25203fc377fb src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c	Fri Jan 27 19:06:35 2017 +0300
+++ b/src/http/v2/ngx_http_v2_filter_module.c	Tue Jan 31 12:09:40 2017 +0300
@@ -148,6 +148,10 @@ ngx_http_v2_header_filter(ngx_http_reque
     static size_t nginx_ver_len = ngx_http_v2_literal_size(NGINX_VER);
     static u_char nginx_ver[ngx_http_v2_literal_size(NGINX_VER)];
 
+    static size_t nginx_ver_build_len =
+                                  ngx_http_v2_literal_size(NGINX_VER_BUILD);
+    static u_char nginx_ver_build[ngx_http_v2_literal_size(NGINX_VER_BUILD)];
+
     if (!r->stream) {
         return ngx_http_next_header_filter(r);
     }
@@ -232,7 +236,16 @@ ngx_http_v2_header_filter(ngx_http_reque
     clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
 
     if (r->headers_out.server == NULL) {
-        len += 1 + (clcf->server_tokens ? nginx_ver_len : sizeof(nginx));
+
+        if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_ON) {
+            len += 1 + nginx_ver_len;
+
+        } else if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_BUILD) {
+            len += 1 + nginx_ver_build_len;
+
+        } else {
+            len += 1 + sizeof(nginx);
+        }
     }
 
     if (r->headers_out.date == NULL) {
@@ -420,13 +433,25 @@ ngx_http_v2_header_filter(ngx_http_reque
     }
 
     if (r->headers_out.server == NULL) {
-        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, fc->log, 0,
-                       "http2 output header: \"server: %s\"",
-                       clcf->server_tokens ? NGINX_VER : "nginx");
+
+        if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_ON) {
+            ngx_log_debug1(NGX_LOG_DEBUG_HTTP, fc->log, 0,
+                           "http2 output header: \"server: %s\"",
+                           NGINX_VER);
+
+        } else if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_BUILD) {
+            ngx_log_debug1(NGX_LOG_DEBUG_HTTP, fc->log, 0,
+                           "http2 output header: \"server: %s\"",
+                           NGINX_VER_BUILD);
+
+        } else {
+            ngx_log_debug0(NGX_LOG_DEBUG_HTTP, fc->log, 0,
+                           "http2 output header: \"server: nginx\"");
+        }
 
         *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_SERVER_INDEX);
 
-        if (clcf->server_tokens) {
+        if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_ON) {
             if (nginx_ver[0] == '\0') {
                 p = ngx_http_v2_write_value(nginx_ver, (u_char *) NGINX_VER,
                                             sizeof(NGINX_VER) - 1, tmp);
@@ -435,6 +460,16 @@ ngx_http_v2_header_filter(ngx_http_reque
 
             pos = ngx_cpymem(pos, nginx_ver, nginx_ver_len);
 
+        } else if (clcf->server_tokens == NGX_HTTP_SERVER_TOKENS_BUILD) {
+            if (nginx_ver_build[0] == '\0') {
+                p = ngx_http_v2_write_value(nginx_ver_build,
+                                            (u_char *) NGINX_VER_BUILD,
+                                            sizeof(NGINX_VER_BUILD) - 1, tmp);
+                nginx_ver_build_len = p - nginx_ver_build;
+            }
+
+            pos = ngx_cpymem(pos, nginx_ver_build, nginx_ver_build_len);
+
         } else {
             pos = ngx_cpymem(pos, nginx, sizeof(nginx));
         }

From ru at nginx.com  Tue Jan 31 09:33:34 2017
From: ru at nginx.com (Ruslan Ermilov)
Date: Tue, 31 Jan 2017 12:33:34 +0300
Subject: [PATCH] HTTP: NGINX_VER_BUILD in Server header and error page
 tail.
In-Reply-To: <1485490598.2897802.861127232.499F85E0@webmail.messagingengine.com>
References: <1462703226.1314805.601366873.679532C0@webmail.messagingengine.com>
 <1485490598.2897802.861127232.499F85E0@webmail.messagingengine.com>
Message-ID: <20170131093334.GA733@lo0.su>

Hi Tom,

On Fri, Jan 27, 2017 at 02:46:38PM +1030, Tom Thorogood wrote:
> Hi,
> 
> Ping. Any chance this might be able to be merged?

I've fixed some omissions in your patch, and committed the
support for new option.

http://hg.nginx.org/nginx/rev/25203fc377fb

> On Sun, 8 May 2016, at 08:57 PM, Tom Thorogood wrote:
> > Hi,
> > 
> > This patch adds a build option to the server_tokens directive. When build is specified the Server header will have a value of NGINX_VER_BUILD. Also the tail of built in error pages will be NGINX_VER_BUILD. off and on are retained as valid values and behave as they always have.
> > 
> > This makes it possible to specify semi-custom server tokens of the form "nginx/<version> (<value>)". <value> is specified with the --build flag for ./configure.
> > 
> > A potential use case is including the name of a custom build for example with --build="super-nginx/0.1". This would give: nginx/1.9.15 (super-nginx/0.1).
> > 
> > Another potential use case of this feature is to include the git revision of a parent build with --build="`git rev-parse --short HEAD`". This would give: nginx/1.9.15 (c3ebabd). In that case the feature can be used to enhance troubleshooting and/or provide more information to end users.
> > 
> > Kind Regards,
> > Tom Thorogood.

From mdounin at mdounin.ru  Tue Jan 31 13:40:46 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:40:46 +0000
Subject: [nginx] Version bump.
Message-ID: <hg.cd02bac58c3b.1485870046.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/cd02bac58c3b
branches:  stable-1.10
changeset: 6886:cd02bac58c3b
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jan 31 16:15:31 2017 +0300
description:
Version bump.

diffstat:

 src/core/nginx.h |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff --git a/src/core/nginx.h b/src/core/nginx.h
--- a/src/core/nginx.h
+++ b/src/core/nginx.h
@@ -9,8 +9,8 @@
 #define _NGINX_H_INCLUDED_
 
 
-#define nginx_version      1010002
-#define NGINX_VERSION      "1.10.2"
+#define nginx_version      1010003
+#define NGINX_VERSION      "1.10.3"
 #define NGINX_VER          "nginx/" NGINX_VERSION
 
 #ifdef NGX_BUILD

From mdounin at mdounin.ru  Tue Jan 31 13:40:49 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:40:49 +0000
Subject: [nginx] Addition filter: set last_in_chain flag when clearing
 last_buf.
Message-ID: <hg.17000e3ba589.1485870049.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/17000e3ba589
branches:  stable-1.10
changeset: 6887:17000e3ba589
user:      Roman Arutyunyan <arut at nginx.com>
date:      Mon Oct 03 21:03:27 2016 +0300
description:
Addition filter: set last_in_chain flag when clearing last_buf.

When the last_buf flag is cleared for add_after_body to append more data from a
subrequest, other filters may still have buffered data, which should be flushed
at this point.  For example, the sub_filter may have a partial match buffered,
which will only be flushed after the subrequest is done, ending up with
interleaved data in output.

Setting last_in_chain instead of last_buf flushes the data and fixes the order
of output buffers.

diffstat:

 src/http/modules/ngx_http_addition_filter_module.c |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (11 lines):

diff --git a/src/http/modules/ngx_http_addition_filter_module.c b/src/http/modules/ngx_http_addition_filter_module.c
--- a/src/http/modules/ngx_http_addition_filter_module.c
+++ b/src/http/modules/ngx_http_addition_filter_module.c
@@ -171,6 +171,7 @@ ngx_http_addition_body_filter(ngx_http_r
     for (cl = in; cl; cl = cl->next) {
         if (cl->buf->last_buf) {
             cl->buf->last_buf = 0;
+            cl->buf->last_in_chain = 1;
             cl->buf->sync = 1;
             last = 1;
         }

From mdounin at mdounin.ru  Tue Jan 31 13:40:51 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:40:51 +0000
Subject: [nginx] Core: sockaddr lengths now respected by ngx_cmp_sockaddr().
Message-ID: <hg.70539dd7abe5.1485870051.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/70539dd7abe5
branches:  stable-1.10
changeset: 6888:70539dd7abe5
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Oct 10 16:15:41 2016 +0300
description:
Core: sockaddr lengths now respected by ngx_cmp_sockaddr().

Linux can return AF_UNIX sockaddrs with partially filled sun_path,
resulting in spurious comparison failures and failed binary upgrades.
Added proper checking of the lengths provided.

Reported by Jan Seda,
http://mailman.nginx.org/pipermail/nginx-devel/2016-September/008832.html.

diffstat:

 src/core/ngx_inet.c |  19 +++++++++++++------
 1 files changed, 13 insertions(+), 6 deletions(-)

diffs (39 lines):

diff --git a/src/core/ngx_inet.c b/src/core/ngx_inet.c
--- a/src/core/ngx_inet.c
+++ b/src/core/ngx_inet.c
@@ -1213,6 +1213,7 @@ ngx_cmp_sockaddr(struct sockaddr *sa1, s
     struct sockaddr_in6  *sin61, *sin62;
 #endif
 #if (NGX_HAVE_UNIX_DOMAIN)
+    size_t                len;
     struct sockaddr_un   *saun1, *saun2;
 #endif
 
@@ -1242,15 +1243,21 @@ ngx_cmp_sockaddr(struct sockaddr *sa1, s
 #if (NGX_HAVE_UNIX_DOMAIN)
     case AF_UNIX:
 
-        /* TODO length */
-
         saun1 = (struct sockaddr_un *) sa1;
         saun2 = (struct sockaddr_un *) sa2;
 
-        if (ngx_memcmp(&saun1->sun_path, &saun2->sun_path,
-                       sizeof(saun1->sun_path))
-            != 0)
-        {
+        if (slen1 < slen2) {
+            len = slen1 - offsetof(struct sockaddr_un, sun_path);
+
+        } else {
+            len = slen2 - offsetof(struct sockaddr_un, sun_path);
+        }
+
+        if (len > sizeof(saun1->sun_path)) {
+            len = sizeof(saun1->sun_path);
+        }
+
+        if (ngx_memcmp(&saun1->sun_path, &saun2->sun_path, len) != 0) {
             return NGX_DECLINED;
         }
 

From mdounin at mdounin.ru  Tue Jan 31 13:40:54 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:40:54 +0000
Subject: [nginx] HTTP/2: graceful shutdown of active connections (closes
 #1106).
Message-ID: <hg.09cf90250844.1485870054.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/09cf90250844
branches:  stable-1.10
changeset: 6889:09cf90250844
user:      Valentin Bartenev <vbart at nginx.com>
date:      Thu Oct 20 16:15:03 2016 +0300
description:
HTTP/2: graceful shutdown of active connections (closes #1106).

Previously, while shutting down gracefully, the HTTP/2 connections were
closed in transition to idle state after all active streams have been
processed.  That might never happen if the client continued opening new
streams.

Now, nginx sends GOAWAY to all HTTP/2 connections and ignores further
attempts to open new streams.  A worker process will quit as soon as
processing of already opened streams is finished.

diffstat:

 src/http/v2/ngx_http_v2.c |  47 ++++++++++++++++++++++++++++++++++++++---------
 src/http/v2/ngx_http_v2.h |   1 +
 2 files changed, 39 insertions(+), 9 deletions(-)

diffs (124 lines):

diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c
+++ b/src/http/v2/ngx_http_v2.c
@@ -136,6 +136,8 @@ static ngx_int_t ngx_http_v2_send_window
     ngx_uint_t sid, size_t window);
 static ngx_int_t ngx_http_v2_send_rst_stream(ngx_http_v2_connection_t *h2c,
     ngx_uint_t sid, ngx_uint_t status);
+static ngx_int_t ngx_http_v2_send_goaway(ngx_http_v2_connection_t *h2c,
+    ngx_uint_t status);
 
 static ngx_http_v2_out_frame_t *ngx_http_v2_get_frame(
     ngx_http_v2_connection_t *h2c, size_t length, ngx_uint_t type,
@@ -293,6 +295,8 @@ ngx_http_v2_init(ngx_event_t *rev)
     rev->handler = ngx_http_v2_read_handler;
     c->write->handler = ngx_http_v2_write_handler;
 
+    c->idle = 1;
+
     ngx_http_v2_read_handler(rev);
 }
 
@@ -320,6 +324,25 @@ ngx_http_v2_read_handler(ngx_event_t *re
 
     h2c->blocked = 1;
 
+    if (c->close) {
+        c->close = 0;
+        h2c->goaway = 1;
+
+        if (ngx_http_v2_send_goaway(h2c, NGX_HTTP_V2_NO_ERROR) == NGX_ERROR) {
+            ngx_http_v2_finalize_connection(h2c, 0);
+            return;
+        }
+
+        if (ngx_http_v2_send_output_queue(h2c) == NGX_ERROR) {
+            ngx_http_v2_finalize_connection(h2c, 0);
+            return;
+        }
+
+        h2c->blocked = 0;
+
+        return;
+    }
+
     h2mcf = ngx_http_get_module_main_conf(h2c->http_connection->conf_ctx,
                                           ngx_http_v2_module);
 
@@ -633,6 +656,11 @@ ngx_http_v2_handle_connection(ngx_http_v
         /* rc == NGX_OK */
     }
 
+    if (h2c->goaway) {
+        ngx_http_close_connection(c);
+        return;
+    }
+
     h2scf = ngx_http_get_module_srv_conf(h2c->http_connection->conf_ctx,
                                          ngx_http_v2_module);
     if (h2c->state.incomplete) {
@@ -640,11 +668,6 @@ ngx_http_v2_handle_connection(ngx_http_v
         return;
     }
 
-    if (ngx_terminate || ngx_exiting) {
-        ngx_http_v2_finalize_connection(h2c, NGX_HTTP_V2_NO_ERROR);
-        return;
-    }
-
     ngx_destroy_pool(h2c->pool);
 
     h2c->pool = NULL;
@@ -658,7 +681,6 @@ ngx_http_v2_handle_connection(ngx_http_v
 #endif
 
     c->destroyed = 1;
-    c->idle = 1;
     ngx_reusable_connection(c, 1);
 
     c->write->handler = ngx_http_empty_handler;
@@ -1027,6 +1049,12 @@ ngx_http_v2_state_headers(ngx_http_v2_co
         return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_SIZE_ERROR);
     }
 
+    if (h2c->goaway) {
+        ngx_log_debug0(NGX_LOG_DEBUG_HTTP, h2c->connection->log, 0,
+                       "skipping http2 HEADERS frame");
+        return ngx_http_v2_state_skip(h2c, pos, end);
+    }
+
     if ((size_t) (end - pos) < size) {
         return ngx_http_v2_state_save(h2c, pos, end,
                                       ngx_http_v2_state_headers);
@@ -4162,7 +4190,6 @@ ngx_http_v2_idle_handler(ngx_event_t *re
 #endif
 
     c->destroyed = 0;
-    c->idle = 0;
     ngx_reusable_connection(c, 0);
 
     h2scf = ngx_http_get_module_srv_conf(h2c->http_connection->conf_ctx,
@@ -4197,8 +4224,10 @@ ngx_http_v2_finalize_connection(ngx_http
 
     h2c->blocked = 1;
 
-    if (!c->error && ngx_http_v2_send_goaway(h2c, status) != NGX_ERROR) {
-        (void) ngx_http_v2_send_output_queue(h2c);
+    if (!c->error && !h2c->goaway) {
+        if (ngx_http_v2_send_goaway(h2c, status) != NGX_ERROR) {
+            (void) ngx_http_v2_send_output_queue(h2c);
+        }
     }
 
     c->error = 1;
diff --git a/src/http/v2/ngx_http_v2.h b/src/http/v2/ngx_http_v2.h
--- a/src/http/v2/ngx_http_v2.h
+++ b/src/http/v2/ngx_http_v2.h
@@ -146,6 +146,7 @@ struct ngx_http_v2_connection_s {
     unsigned                         closed_nodes:8;
     unsigned                         settings_ack:1;
     unsigned                         blocked:1;
+    unsigned                         goaway:1;
 };
 
 

From mdounin at mdounin.ru  Tue Jan 31 13:40:56 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:40:56 +0000
Subject: [nginx] HTTP/2: fixed saving preread buffer to temp file (ticket
 #1143).
Message-ID: <hg.16487f9e6665.1485870056.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/16487f9e6665
branches:  stable-1.10
changeset: 6890:16487f9e6665
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Nov 28 19:19:21 2016 +0300
description:
HTTP/2: fixed saving preread buffer to temp file (ticket #1143).

Previously, a request body bigger than "client_body_buffer_size" wasn't written
into a temporary file if it has been pre-read entirely.  The preread buffer
is freed after processing, thus subsequent use of it might result in sending
corrupted body or cause a segfault.

diffstat:

 src/http/v2/ngx_http_v2.c |  3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diffs (13 lines):

diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c
+++ b/src/http/v2/ngx_http_v2.c
@@ -3560,6 +3560,9 @@ ngx_http_v2_read_request_body(ngx_http_r
         rb->buf = ngx_create_temp_buf(r->pool, (size_t) len);
 
     } else {
+        /* enforce writing body to file */
+        r->request_body_in_file_only = 1;
+
         rb->buf = ngx_calloc_buf(r->pool);
 
         if (rb->buf != NULL) {

From mdounin at mdounin.ru  Tue Jan 31 13:40:59 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:40:59 +0000
Subject: [nginx] HTTP/2: fixed posted streams handling.
Message-ID: <hg.749bcfdf097a.1485870059.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/749bcfdf097a
branches:  stable-1.10
changeset: 6891:749bcfdf097a
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Nov 28 20:58:14 2016 +0300
description:
HTTP/2: fixed posted streams handling.

A bug was introduced by 82efcedb310b that could lead to timing out of
responses or segmentation fault, when accept_mutex was enabled.

The output queue in HTTP/2 can contain frames from different streams.
When the queue is sent, all related write handlers need to be called.
In order to do so, the streams were added to the h2c->posted queue
after handling sent frames.  Then this queue was processed in
ngx_http_v2_write_handler().

If accept_mutex is enabled, the event's "ready" flag is set but its
handler is not called immediately.  Instead, the event is added to
the ngx_posted_events queue.  At the same time in this queue can be
events from upstream connections.  Such events can result in sending
output queue before ngx_http_v2_write_handler() is triggered.  And
at the time ngx_http_v2_write_handler() is called, the output queue
can be already empty with some streams added to h2c->posted.

But after 82efcedb310b, these streams weren't processed if all frames
have already been sent and the output queue was empty.  This might lead
to a situation when a number of streams were get stuck in h2c->posted
queue for a long time.  Eventually these streams might get closed by
the send timeout.

In the worst case this might also lead to a segmentation fault, if
already freed stream was left in the h2c->posted queue.  This could
happen if one of the streams was terminated but wasn't closed, due to
the HEADERS frame or a partially sent DATA frame left in the output
queue.  If this happened the ngx_http_v2_filter_cleanup() handler
removed the stream from the h2c->waiting or h2c->posted queue on
termination stage, before the frame has been sent, and the stream
was again added to the h2c->posted queue after the frame was sent.

In order to fix all these problems and simplify the code, write
events of fake stream connections are now added to ngx_posted_events
instead of using a custom h2c->posted queue.

diffstat:

 src/http/v2/ngx_http_v2.c               |  27 +----------------
 src/http/v2/ngx_http_v2.h               |   3 +-
 src/http/v2/ngx_http_v2_filter_module.c |  50 +++++++++++++++++++++++++-------
 3 files changed, 42 insertions(+), 38 deletions(-)

diffs (183 lines):

diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c
+++ b/src/http/v2/ngx_http_v2.c
@@ -286,7 +286,6 @@ ngx_http_v2_init(ngx_event_t *rev)
                                             : ngx_http_v2_state_preface;
 
     ngx_queue_init(&h2c->waiting);
-    ngx_queue_init(&h2c->posted);
     ngx_queue_init(&h2c->dependencies);
     ngx_queue_init(&h2c->closed);
 
@@ -415,9 +414,7 @@ static void
 ngx_http_v2_write_handler(ngx_event_t *wev)
 {
     ngx_int_t                  rc;
-    ngx_queue_t               *q;
     ngx_connection_t          *c;
-    ngx_http_v2_stream_t      *stream;
     ngx_http_v2_connection_t  *h2c;
 
     c = wev->data;
@@ -452,26 +449,6 @@ ngx_http_v2_write_handler(ngx_event_t *w
         return;
     }
 
-    while (!ngx_queue_empty(&h2c->posted)) {
-        q = ngx_queue_head(&h2c->posted);
-
-        ngx_queue_remove(q);
-
-        stream = ngx_queue_data(q, ngx_http_v2_stream_t, queue);
-
-        stream->handled = 0;
-
-        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
-                       "run http2 stream %ui", stream->node->id);
-
-        wev = stream->request->connection->write;
-
-        wev->active = 0;
-        wev->ready = 1;
-
-        wev->handler(wev);
-    }
-
     h2c->blocked = 0;
 
     if (rc == NGX_AGAIN) {
@@ -2240,7 +2217,7 @@ ngx_http_v2_state_window_update(ngx_http
 
         stream = ngx_queue_data(q, ngx_http_v2_stream_t, queue);
 
-        stream->handled = 0;
+        stream->waiting = 0;
 
         wev = stream->request->connection->write;
 
@@ -4259,7 +4236,7 @@ ngx_http_v2_finalize_connection(ngx_http
                 continue;
             }
 
-            stream->handled = 0;
+            stream->waiting = 0;
 
             r = stream->request;
             fc = r->connection;
diff --git a/src/http/v2/ngx_http_v2.h b/src/http/v2/ngx_http_v2.h
--- a/src/http/v2/ngx_http_v2.h
+++ b/src/http/v2/ngx_http_v2.h
@@ -137,7 +137,6 @@ struct ngx_http_v2_connection_s {
 
     ngx_http_v2_out_frame_t         *last_out;
 
-    ngx_queue_t                      posted;
     ngx_queue_t                      dependencies;
     ngx_queue_t                      closed;
 
@@ -192,7 +191,7 @@ struct ngx_http_v2_stream_s {
 
     ngx_pool_t                      *pool;
 
-    unsigned                         handled:1;
+    unsigned                         waiting:1;
     unsigned                         blocked:1;
     unsigned                         exhausted:1;
     unsigned                         in_closed:1;
diff --git a/src/http/v2/ngx_http_v2_filter_module.c b/src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c
+++ b/src/http/v2/ngx_http_v2_filter_module.c
@@ -1122,11 +1122,11 @@ ngx_http_v2_waiting_queue(ngx_http_v2_co
     ngx_queue_t           *q;
     ngx_http_v2_stream_t  *s;
 
-    if (stream->handled) {
+    if (stream->waiting) {
         return;
     }
 
-    stream->handled = 1;
+    stream->waiting = 1;
 
     for (q = ngx_queue_last(&h2c->waiting);
          q != ngx_queue_sentinel(&h2c->waiting);
@@ -1317,20 +1317,29 @@ static ngx_inline void
 ngx_http_v2_handle_stream(ngx_http_v2_connection_t *h2c,
     ngx_http_v2_stream_t *stream)
 {
+    ngx_event_t       *wev;
     ngx_connection_t  *fc;
 
-    if (stream->handled || stream->blocked) {
+    if (stream->waiting || stream->blocked) {
         return;
     }
 
     fc = stream->request->connection;
 
-    if (!fc->error && (stream->exhausted || fc->write->delayed)) {
+    if (!fc->error && stream->exhausted) {
         return;
     }
 
-    stream->handled = 1;
-    ngx_queue_insert_tail(&h2c->posted, &stream->queue);
+    wev = fc->write;
+
+    wev->active = 0;
+    wev->ready = 1;
+
+    if (!fc->error && wev->delayed) {
+        return;
+    }
+
+    ngx_post_event(wev, &ngx_posted_events);
 }
 
 
@@ -1340,11 +1349,13 @@ ngx_http_v2_filter_cleanup(void *data)
     ngx_http_v2_stream_t *stream = data;
 
     size_t                     window;
+    ngx_event_t               *wev;
+    ngx_queue_t               *q;
     ngx_http_v2_out_frame_t   *frame, **fn;
     ngx_http_v2_connection_t  *h2c;
 
-    if (stream->handled) {
-        stream->handled = 0;
+    if (stream->waiting) {
+        stream->waiting = 0;
         ngx_queue_remove(&stream->queue);
     }
 
@@ -1378,9 +1389,26 @@ ngx_http_v2_filter_cleanup(void *data)
         fn = &frame->next;
     }
 
-    if (h2c->send_window == 0 && window && !ngx_queue_empty(&h2c->waiting)) {
-        ngx_queue_add(&h2c->posted, &h2c->waiting);
-        ngx_queue_init(&h2c->waiting);
+    if (h2c->send_window == 0 && window) {
+
+        while (!ngx_queue_empty(&h2c->waiting)) {
+            q = ngx_queue_head(&h2c->waiting);
+
+            ngx_queue_remove(q);
+
+            stream = ngx_queue_data(q, ngx_http_v2_stream_t, queue);
+
+            stream->waiting = 0;
+
+            wev = stream->request->connection->write;
+
+            wev->active = 0;
+            wev->ready = 1;
+
+            if (!wev->delayed) {
+                ngx_post_event(wev, &ngx_posted_events);
+            }
+        }
     }
 
     h2c->send_window += window;

From mdounin at mdounin.ru  Tue Jan 31 13:41:01 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:41:01 +0000
Subject: [nginx] HTTP/2: prevented creating temp files for requests without
 body.
Message-ID: <hg.169fef913184.1485870061.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/169fef913184
branches:  stable-1.10
changeset: 6892:169fef913184
user:      Valentin Bartenev <vbart at nginx.com>
date:      Sat Dec 10 13:23:38 2016 +0300
description:
HTTP/2: prevented creating temp files for requests without body.

The problem was introduced by 52bd8cc17f34.

diffstat:

 src/http/v2/ngx_http_v2.c |  6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diffs (16 lines):

diff --git a/src/http/v2/ngx_http_v2.c b/src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c
+++ b/src/http/v2/ngx_http_v2.c
@@ -3537,8 +3537,10 @@ ngx_http_v2_read_request_body(ngx_http_r
         rb->buf = ngx_create_temp_buf(r->pool, (size_t) len);
 
     } else {
-        /* enforce writing body to file */
-        r->request_body_in_file_only = 1;
+        if (stream->preread) {
+            /* enforce writing preread buffer to file */
+            r->request_body_in_file_only = 1;
+        }
 
         rb->buf = ngx_calloc_buf(r->pool);
 

From mdounin at mdounin.ru  Tue Jan 31 13:41:04 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:41:04 +0000
Subject: [nginx] Fixed trailer construction with limit on FreeBSD and macOS.
Message-ID: <hg.a3e6d660b179.1485870064.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/a3e6d660b179
branches:  stable-1.10
changeset: 6893:a3e6d660b179
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jan 20 21:12:48 2017 +0300
description:
Fixed trailer construction with limit on FreeBSD and macOS.

The ngx_chain_coalesce_file() function may produce more bytes to send then
requested in the limit passed, as it aligns the last file position
to send to memory page boundary.  As a result, (limit - send) may become
negative.  This resulted in big positive number when converted to size_t
while calling ngx_output_chain_to_iovec().

Another part of the problem is in ngx_chain_coalesce_file(): it changes cl
to the next chain link even if the current buffer is only partially sent
due to limit.

Therefore, if a file buffer was not expected to be fully sent due to limit,
and was followed by a memory buffer, nginx called sendfile() with a part
of the file buffer, and the memory buffer in trailer.  If there were enough
room in the socket buffer, this resulted in a part of the file buffer being
skipped, and corresponding part of the memory buffer sent instead.

The bug was introduced in 8e903522c17a (1.7.8).  Configurations affected
are ones using limits, that is, limit_rate and/or sendfile_max_chunk, and
memory buffers after file ones (may happen when using subrequests or
with proxying with disk buffering).

Fix is to explicitly check if (send < limit) before constructing trailer
with ngx_output_chain_to_iovec().  Additionally, ngx_chain_coalesce_file()
was modified to preserve unfinished file buffers in cl.

diffstat:

 src/core/ngx_buf.c                       |   3 +++
 src/os/unix/ngx_darwin_sendfile_chain.c  |   2 +-
 src/os/unix/ngx_freebsd_sendfile_chain.c |  22 +++++++++++++++-------
 3 files changed, 19 insertions(+), 8 deletions(-)

diffs (60 lines):

diff --git a/src/core/ngx_buf.c b/src/core/ngx_buf.c
--- a/src/core/ngx_buf.c
+++ b/src/core/ngx_buf.c
@@ -244,6 +244,9 @@ ngx_chain_coalesce_file(ngx_chain_t **in
             if (aligned <= cl->buf->file_last) {
                 size = aligned - cl->buf->file_pos;
             }
+
+            total += size;
+            break;
         }
 
         total += size;
diff --git a/src/os/unix/ngx_darwin_sendfile_chain.c b/src/os/unix/ngx_darwin_sendfile_chain.c
--- a/src/os/unix/ngx_darwin_sendfile_chain.c
+++ b/src/os/unix/ngx_darwin_sendfile_chain.c
@@ -98,7 +98,7 @@ ngx_darwin_sendfile_chain(ngx_connection
 
             send += file_size;
 
-            if (header.count == 0) {
+            if (header.count == 0 && send < limit) {
 
                 /*
                  * create the trailer iovec and coalesce the neighbouring bufs
diff --git a/src/os/unix/ngx_freebsd_sendfile_chain.c b/src/os/unix/ngx_freebsd_sendfile_chain.c
--- a/src/os/unix/ngx_freebsd_sendfile_chain.c
+++ b/src/os/unix/ngx_freebsd_sendfile_chain.c
@@ -114,16 +114,24 @@ ngx_freebsd_sendfile_chain(ngx_connectio
 
             send += file_size;
 
-            /* create the trailer iovec and coalesce the neighbouring bufs */
-
-            cl = ngx_output_chain_to_iovec(&trailer, cl, limit - send, c->log);
+            if (send < limit) {
 
-            if (cl == NGX_CHAIN_ERROR) {
-                return NGX_CHAIN_ERROR;
+                /*
+                 * create the trailer iovec and coalesce the neighbouring bufs
+                 */
+
+                cl = ngx_output_chain_to_iovec(&trailer, cl, limit - send,
+                                               c->log);
+                if (cl == NGX_CHAIN_ERROR) {
+                    return NGX_CHAIN_ERROR;
+                }
+
+                send += trailer.size;
+
+            } else {
+                trailer.count = 0;
             }
 
-            send += trailer.size;
-
             if (ngx_freebsd_use_tcp_nopush
                 && c->tcp_nopush == NGX_TCP_NOPUSH_UNSET)
             {

From mdounin at mdounin.ru  Tue Jan 31 13:41:06 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:41:06 +0000
Subject: [nginx] Upstream: fixed cache corruption and socket leaks with
 aio_write.
Message-ID: <hg.2204d71e88e5.1485870066.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/2204d71e88e5
branches:  stable-1.10
changeset: 6894:2204d71e88e5
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jan 20 21:14:19 2017 +0300
description:
Upstream: fixed cache corruption and socket leaks with aio_write.

The ngx_event_pipe() function wasn't called on write events with
wev->delayed set.  As a result, threaded writing results weren't
properly collected in ngx_event_pipe_write_to_downstream() when a
write event was triggered for a completed write.

Further, this wasn't detected, as p->aio was reset by a thread completion
handler, and results were later collected in ngx_event_pipe_read_upstream()
instead of scheduling a new write of additional data.  If this happened
on the last reading from an upstream, last part of the response was never
written to the cache file.

Similar problems might also happen in case of timeouts when writing to
client, as this also results in ngx_event_pipe() not being called on write
events.  In this scenario socket leaks were observed.

Fix is to check if p->writing is set in ngx_event_pipe_read_upstream(), and
therefore collect results of previous write operations in case of read events
as well, similar to how we do so in ngx_event_pipe_write_downstream().
This is enough to fix the wev->delayed case.  Additionally, we now call
ngx_event_pipe() from ngx_http_upstream_process_request() if there are
uncollected write operations (p->writing and !p->aio).  This also fixes
the wev->timedout case.

diffstat:

 src/event/ngx_event_pipe.c   |  13 +++++++++++++
 src/http/ngx_http_upstream.c |  15 +++++++++++++++
 2 files changed, 28 insertions(+), 0 deletions(-)

diffs (56 lines):

diff --git a/src/event/ngx_event_pipe.c b/src/event/ngx_event_pipe.c
--- a/src/event/ngx_event_pipe.c
+++ b/src/event/ngx_event_pipe.c
@@ -113,11 +113,24 @@ ngx_event_pipe_read_upstream(ngx_event_p
     }
 
 #if (NGX_THREADS)
+
     if (p->aio) {
         ngx_log_debug0(NGX_LOG_DEBUG_EVENT, p->log, 0,
                        "pipe read upstream: aio");
         return NGX_AGAIN;
     }
+
+    if (p->writing) {
+        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, p->log, 0,
+                       "pipe read upstream: writing");
+
+        rc = ngx_event_pipe_write_chain_to_temp_file(p);
+
+        if (rc != NGX_OK) {
+            return rc;
+        }
+    }
+
 #endif
 
     ngx_log_debug1(NGX_LOG_DEBUG_EVENT, p->log, 0,
diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c
+++ b/src/http/ngx_http_upstream.c
@@ -3744,9 +3744,24 @@ ngx_http_upstream_process_request(ngx_ht
     p = u->pipe;
 
 #if (NGX_THREADS)
+
+    if (p->writing && !p->aio) {
+
+        /*
+         * make sure to call ngx_event_pipe()
+         * if there is an incomplete aio write
+         */
+
+        if (ngx_event_pipe(p, 1) == NGX_ABORT) {
+            ngx_http_upstream_finalize_request(r, u, NGX_ERROR);
+            return;
+        }
+    }
+
     if (p->writing) {
         return;
     }
+
 #endif
 
     if (u->peer.connection) {

From mdounin at mdounin.ru  Tue Jan 31 13:41:08 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:41:08 +0000
Subject: [nginx] Updated zlib and PCRE used for win32 builds.
Message-ID: <hg.b409c77870d6.1485870068.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/b409c77870d6
branches:  stable-1.10
changeset: 6895:b409c77870d6
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jan 24 16:41:29 2017 +0300
description:
Updated zlib and PCRE used for win32 builds.

diffstat:

 misc/GNUmakefile |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff --git a/misc/GNUmakefile b/misc/GNUmakefile
--- a/misc/GNUmakefile
+++ b/misc/GNUmakefile
@@ -6,8 +6,8 @@ TEMP =		tmp
 
 OBJS =		objs.msvc8
 OPENSSL =	openssl-1.0.2j
-ZLIB =		zlib-1.2.8
-PCRE =		pcre-8.39
+ZLIB =		zlib-1.2.11
+PCRE =		pcre-8.40
 
 
 release: export

From mdounin at mdounin.ru  Tue Jan 31 13:41:11 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 13:41:11 +0000
Subject: [nginx] Updated OpenSSL used for win32 builds.
Message-ID: <hg.6a510ba580a6.1485870071.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/6a510ba580a6
branches:  stable-1.10
changeset: 6896:6a510ba580a6
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jan 27 19:06:35 2017 +0300
description:
Updated OpenSSL used for win32 builds.

diffstat:

 misc/GNUmakefile |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff --git a/misc/GNUmakefile b/misc/GNUmakefile
--- a/misc/GNUmakefile
+++ b/misc/GNUmakefile
@@ -5,7 +5,7 @@ NGINX =		nginx-$(VER)
 TEMP =		tmp
 
 OBJS =		objs.msvc8
-OPENSSL =	openssl-1.0.2j
+OPENSSL =	openssl-1.0.2k
 ZLIB =		zlib-1.2.11
 PCRE =		pcre-8.40
 

From mdounin at mdounin.ru  Tue Jan 31 15:04:44 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 15:04:44 +0000
Subject: [nginx] nginx-1.10.3-RELEASE
Message-ID: <hg.369b74459cf2.1485875084.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/369b74459cf2
branches:  stable-1.10
changeset: 6897:369b74459cf2
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jan 31 18:01:10 2017 +0300
description:
nginx-1.10.3-RELEASE

diffstat:

 docs/xml/nginx/changes.xml |  95 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 95 insertions(+), 0 deletions(-)

diffs (105 lines):

diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml
--- a/docs/xml/nginx/changes.xml
+++ b/docs/xml/nginx/changes.xml
@@ -5,6 +5,101 @@
 <change_log title="nginx">
 
 
+<changes ver="1.10.3" date="31.01.2017">
+
+<change type="bugfix">
+<para lang="ru">
+? ????????? add_after_body ??? ????????????? ????????? ? ?????????? sub_filter.
+</para>
+<para lang="en">
+in the "add_after_body" directive when used with the "sub_filter" directive.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+unix domain listen-?????? ????? ?? ?????????????
+??? ?????????? ???????????? ????? ?? Linux.
+</para>
+<para lang="en">
+unix domain listen sockets might not be inherited
+during binary upgrade on Linux.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??????? ?????????? ?????? ??????? ????????? ????? ???????? ??????????? ?????
+??? ????????????? HTTP/2.
+</para>
+<para lang="en">
+graceful shutdown of old worker processes might require infinite time
+when using HTTP/2.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? HTTP/2 ? ???????? limit_req ??? auth_request
+???? ??????? ????? ???? ??????????;
+?????? ????????? ? 1.10.2.
+</para>
+<para lang="en">
+when using HTTP/2 and the "limit_req" or "auth_request" directives
+client request body might be corrupted;
+the bug had appeared in 1.10.2.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? HTTP/2 ? ??????? ???????? ??? ????????? segmentation fault;
+?????? ????????? ? 1.10.2.
+</para>
+<para lang="en">
+a segmentation fault might occur in a worker process when using HTTP/2;
+the bug had appeared in 1.10.2.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ????????? sendfile ?? FreeBSD ? macOS
+??? ???????????? ???????????? ?????;
+?????? ????????? ? 1.7.8.
+</para>
+<para lang="en">
+an incorrect response might be returned
+when using the "sendfile" directive on FreeBSD and macOS;
+the bug had appeared in 1.7.8.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ????????? aio_write
+????? ??? ??????????? ? ??? ?? ?????????.
+</para>
+<para lang="en">
+a truncated response might be stored in cache
+when using the "aio_write" directive.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ????????? aio_write
+????? ??????????? ?????? ???????.
+</para>
+<para lang="en">
+a socket leak might occur
+when using the "aio_write" directive.
+</para>
+</change>
+
+</changes>
+
+
 <changes ver="1.10.2" date="18.10.2016">
 
 <change type="change">

From mdounin at mdounin.ru  Tue Jan 31 15:04:47 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 15:04:47 +0000
Subject: [nginx] release-1.10.3 tag
Message-ID: <hg.43ba2bb504fb.1485875087.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/43ba2bb504fb
branches:  stable-1.10
changeset: 6898:43ba2bb504fb
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jan 31 18:01:10 2017 +0300
description:
release-1.10.3 tag

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -401,3 +401,4 @@ 13070ecfda67397985f0e986eb9c42ecb46d05b5
 d4b7edd7fa81cbf554ee3dcfe9de53ad6d1dbc69 release-1.10.0
 7fcde9122088094471a97030161e347aa940f8ed release-1.10.1
 b98f9fdee487b00556bea6a24f8f976d7fd60795 release-1.10.2
+369b74459cf24dd35c5e1111d17c2cd4a6ecfd4b release-1.10.3

From mdounin at mdounin.ru  Tue Jan 31 16:00:42 2017
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 31 Jan 2017 19:00:42 +0300
Subject: [PATCH] Proxy: make timeout directives accept variables
In-Reply-To: <9d03643e-0798-3c7a-a793-5b739c28c2e2@fastmail.com>
References: <9d03643e-0798-3c7a-a793-5b739c28c2e2@fastmail.com>
Message-ID: <20170131160041.GK46625@mdounin.ru>

Hello!

On Mon, Jan 30, 2017 at 05:39:13PM -0800, Thibault Charbonnier wrote:

> This is a proposal to allow for better granularity (eventually, on a 
> per-request basis?) regarding the proxy_*_timeout directives.
> 
> Please let me know how you feel about such a feature, and if you think 
> this patch should be implemented in a different way.

I can't say I like the patch as it introduces a lot of code 
without obvious reasons.  Not to mention it only covers proxy, 
while there are also memcached, fastcgi, scgi, and uwsgi.

You may want to provide more details on what problem you are 
trying to solve.

[...]

> If a timeout directive contains a variable, we consider such variable to
> contain the related timeout in ms precision.

Just a side note: this is certainly wrong, it should follow the 
same rules as normal parsing of the directive.

-- 
Maxim Dounin
http://nginx.org/

From arut at nginx.com  Tue Jan 31 17:23:52 2017
From: arut at nginx.com (Roman Arutyunyan)
Date: Tue, 31 Jan 2017 17:23:52 +0000
Subject: [njs] Support of undefined return from stream body filter.
Message-ID: <hg.67d4bcb5823f.1485883432.5965299922797593991@dev.nginx.com>

details:   http://hg.nginx.org/njs/rev/67d4bcb5823f
branches:  
changeset: 308:67d4bcb5823f
user:      Roman Arutyunyan <arut at nginx.com>
date:      Tue Jan 31 20:22:01 2017 +0300
description:
Support of undefined return from stream body filter.

diffstat:

 nginx/ngx_stream_js_module.c |  38 ++++++++++++++++++++------------------
 1 files changed, 20 insertions(+), 18 deletions(-)

diffs (52 lines):

diff -r e9c89068140d -r 67d4bcb5823f nginx/ngx_stream_js_module.c
--- a/nginx/ngx_stream_js_module.c	Fri Jan 27 16:45:34 2017 +0300
+++ b/nginx/ngx_stream_js_module.c	Tue Jan 31 20:22:01 2017 +0300
@@ -503,28 +503,30 @@ ngx_stream_js_body_filter(ngx_stream_ses
             return NGX_ERROR;
         }
 
-        if (njs_vm_retval(ctx->vm, &value) != NJS_OK) {
-            return NGX_ERROR;
-        }
-
-        ngx_log_debug2(NGX_LOG_DEBUG_STREAM, c->log, 0,
-                       "js return value: \"%*s\"",
-                       value.length, value.start);
-
-        if (value.length) {
-            rc = ngx_atoi(value.start, value.length);
-
-            if (rc != NGX_OK && rc != -NGX_ERROR) {
-                ngx_log_error(NGX_LOG_ERR, c->log, 0,
-                              "unexpected js return code: \"%*s\"",
-                              value.length, value.start);
+        if (ctx->vm->retval.type != NJS_VOID) {
+            if (njs_vm_retval(ctx->vm, &value) != NJS_OK) {
                 return NGX_ERROR;
             }
 
-            rc = -rc;
+            ngx_log_debug2(NGX_LOG_DEBUG_STREAM, c->log, 0,
+                           "js return value: \"%*s\"",
+                           value.length, value.start);
 
-            if (rc == NGX_ERROR) {
-                return NGX_ERROR;
+            if (value.length) {
+                rc = ngx_atoi(value.start, value.length);
+
+                if (rc != NGX_OK && rc != -NGX_ERROR) {
+                    ngx_log_error(NGX_LOG_ERR, c->log, 0,
+                                  "unexpected js return code: \"%*s\"",
+                                  value.length, value.start);
+                    return NGX_ERROR;
+                }
+
+                rc = -rc;
+
+                if (rc == NGX_ERROR) {
+                    return NGX_ERROR;
+                }
             }
         }
 

From thibaultcha at fastmail.com  Tue Jan 31 18:17:36 2017
From: thibaultcha at fastmail.com (Thibault Charbonnier)
Date: Tue, 31 Jan 2017 10:17:36 -0800
Subject: [PATCH] Proxy: make timeout directives accept variables
In-Reply-To: <20170131160041.GK46625@mdounin.ru>
References: <9d03643e-0798-3c7a-a793-5b739c28c2e2@fastmail.com>
 <20170131160041.GK46625@mdounin.ru>
Message-ID: <dcf6e3c4-6bd1-b285-08bf-697e4d62e3d4@fastmail.com>

Hi!

On 1/31/17 8:00 AM, Maxim Dounin wrote:
> You may want to provide more details on what problem you are
> trying to solve.

The use-case was originally to use a longer proxy_read_timeout on 
certain upstream locations without duplicating too much configuration 
(our location block). I then thought it would be nice to have for other 
timeout directives as well, and eventually reuse the code in other 
modules as you suggested, so timeout directives could support variables 
anywhere.

> I can't say I like the patch as it introduces a lot of code
> without obvious reasons.  Not to mention it only covers proxy,
> while there are also memcached, fastcgi, scgi, and uwsgi.

It is a bit verbose indeed. The reasoning behind it is that 
ngx_http_proxy_timeout and ngx_http_proxy_compile_timeout could 
eventually be moved to ngx_http_script.c and reused in other locations 
like you mentioned. I can take care of it if you deem it necessary.

Another reason for the verbosity is that I originally used 
ngx_http_set_complex_value_slot, but by doing so, we would lose proper 
conf error messages when parsing a plain timeout such as 
'proxy_read_timeout foo;'. This patch tries to preserve an identical 
behavior when such directives do not contain a variable.

> Just a side note: this is certainly wrong, it should follow the
> same rules as normal parsing of the directive.

My bad, I assumed it would be more efficient to avoid ngx_parse_time 
when possible. I will change this.

Best,
Thibault

From xeioex at nginx.com  Tue Jan 31 18:22:23 2017
From: xeioex at nginx.com (Dmitry Volyntsev)
Date: Tue, 31 Jan 2017 18:22:23 +0000
Subject: [nginx] Variables: generic prefix variables.
Message-ID: <hg.d2b2ff157da5.1485886943.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/d2b2ff157da5
branches:  
changeset: 6899:d2b2ff157da5
user:      Dmitry Volyntsev <xeioex at nginx.com>
date:      Tue Jan 31 21:19:58 2017 +0300
description:
Variables: generic prefix variables.

diffstat:

 src/http/modules/ngx_http_rewrite_module.c |   13 +-
 src/http/ngx_http_core_module.h            |    3 +-
 src/http/ngx_http_upstream.c               |   14 +-
 src/http/ngx_http_upstream.h               |    5 -
 src/http/ngx_http_variables.c              |  250 ++++++++++++++--------------
 src/http/ngx_http_variables.h              |    2 +
 src/stream/ngx_stream.h                    |    3 +-
 src/stream/ngx_stream_variables.c          |  146 ++++++++++++++--
 src/stream/ngx_stream_variables.h          |    2 +
 9 files changed, 271 insertions(+), 167 deletions(-)

diffs (747 lines):

diff -r 25203fc377fb -r d2b2ff157da5 src/http/modules/ngx_http_rewrite_module.c
--- a/src/http/modules/ngx_http_rewrite_module.c	Tue Jan 31 12:09:40 2017 +0300
+++ b/src/http/modules/ngx_http_rewrite_module.c	Tue Jan 31 21:19:58 2017 +0300
@@ -917,7 +917,8 @@ ngx_http_rewrite_set(ngx_conf_t *cf, ngx
     value[1].len--;
     value[1].data++;
 
-    v = ngx_http_add_variable(cf, &value[1], NGX_HTTP_VAR_CHANGEABLE);
+    v = ngx_http_add_variable(cf, &value[1],
+                              NGX_HTTP_VAR_CHANGEABLE|NGX_HTTP_VAR_WEAK);
     if (v == NULL) {
         return NGX_CONF_ERROR;
     }
@@ -927,15 +928,7 @@ ngx_http_rewrite_set(ngx_conf_t *cf, ngx
         return NGX_CONF_ERROR;
     }
 
-    if (v->get_handler == NULL
-        && ngx_strncasecmp(value[1].data, (u_char *) "http_", 5) != 0
-        && ngx_strncasecmp(value[1].data, (u_char *) "sent_http_", 10) != 0
-        && ngx_strncasecmp(value[1].data, (u_char *) "upstream_http_", 14) != 0
-        && ngx_strncasecmp(value[1].data, (u_char *) "cookie_", 7) != 0
-        && ngx_strncasecmp(value[1].data, (u_char *) "upstream_cookie_", 16)
-           != 0
-        && ngx_strncasecmp(value[1].data, (u_char *) "arg_", 4) != 0)
-    {
+    if (v->get_handler == NULL) {
         v->get_handler = ngx_http_rewrite_var;
         v->data = index;
     }
diff -r 25203fc377fb -r d2b2ff157da5 src/http/ngx_http_core_module.h
--- a/src/http/ngx_http_core_module.h	Tue Jan 31 12:09:40 2017 +0300
+++ b/src/http/ngx_http_core_module.h	Tue Jan 31 21:19:58 2017 +0300
@@ -158,7 +158,8 @@ typedef struct {
 
     ngx_hash_t                 variables_hash;
 
-    ngx_array_t                variables;       /* ngx_http_variable_t */
+    ngx_array_t                variables;         /* ngx_http_variable_t */
+    ngx_array_t                prefix_variables;  /* ngx_http_variable_t */
     ngx_uint_t                 ncaptures;
 
     ngx_uint_t                 server_names_hash_max_size;
diff -r 25203fc377fb -r d2b2ff157da5 src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c	Tue Jan 31 12:09:40 2017 +0300
+++ b/src/http/ngx_http_upstream.c	Tue Jan 31 21:19:58 2017 +0300
@@ -162,6 +162,10 @@ static ngx_int_t ngx_http_upstream_respo
     ngx_http_variable_value_t *v, uintptr_t data);
 static ngx_int_t ngx_http_upstream_response_length_variable(
     ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data);
+static ngx_int_t ngx_http_upstream_header_variable(ngx_http_request_t *r,
+    ngx_http_variable_value_t *v, uintptr_t data);
+static ngx_int_t ngx_http_upstream_cookie_variable(ngx_http_request_t *r,
+    ngx_http_variable_value_t *v, uintptr_t data);
 
 static char *ngx_http_upstream(ngx_conf_t *cf, ngx_command_t *cmd, void *dummy);
 static char *ngx_http_upstream_server(ngx_conf_t *cf, ngx_command_t *cmd,
@@ -413,6 +417,12 @@ static ngx_http_variable_t  ngx_http_ups
 
 #endif
 
+    { ngx_string("upstream_http_"), NULL, ngx_http_upstream_header_variable,
+      0, NGX_HTTP_VAR_NOCACHEABLE|NGX_HTTP_VAR_PREFIX, 0 },
+
+    { ngx_string("upstream_cookie_"), NULL, ngx_http_upstream_cookie_variable,
+      0, NGX_HTTP_VAR_NOCACHEABLE|NGX_HTTP_VAR_PREFIX, 0 },
+
     { ngx_null_string, NULL, NULL, 0, 0, 0 }
 };
 
@@ -5391,7 +5401,7 @@ ngx_http_upstream_response_length_variab
 }
 
 
-ngx_int_t
+static ngx_int_t
 ngx_http_upstream_header_variable(ngx_http_request_t *r,
     ngx_http_variable_value_t *v, uintptr_t data)
 {
@@ -5406,7 +5416,7 @@ ngx_http_upstream_header_variable(ngx_ht
 }
 
 
-ngx_int_t
+static ngx_int_t
 ngx_http_upstream_cookie_variable(ngx_http_request_t *r,
     ngx_http_variable_value_t *v, uintptr_t data)
 {
diff -r 25203fc377fb -r d2b2ff157da5 src/http/ngx_http_upstream.h
--- a/src/http/ngx_http_upstream.h	Tue Jan 31 12:09:40 2017 +0300
+++ b/src/http/ngx_http_upstream.h	Tue Jan 31 21:19:58 2017 +0300
@@ -402,11 +402,6 @@ typedef struct {
 } ngx_http_upstream_param_t;
 
 
-ngx_int_t ngx_http_upstream_cookie_variable(ngx_http_request_t *r,
-    ngx_http_variable_value_t *v, uintptr_t data);
-ngx_int_t ngx_http_upstream_header_variable(ngx_http_request_t *r,
-    ngx_http_variable_value_t *v, uintptr_t data);
-
 ngx_int_t ngx_http_upstream_create(ngx_http_request_t *r);
 void ngx_http_upstream_init(ngx_http_request_t *r);
 ngx_http_upstream_srv_conf_t *ngx_http_upstream_add(ngx_conf_t *cf,
diff -r 25203fc377fb -r d2b2ff157da5 src/http/ngx_http_variables.c
--- a/src/http/ngx_http_variables.c	Tue Jan 31 12:09:40 2017 +0300
+++ b/src/http/ngx_http_variables.c	Tue Jan 31 21:19:58 2017 +0300
@@ -11,6 +11,9 @@
 #include <nginx.h>
 
 
+static ngx_http_variable_t *ngx_http_add_prefix_variable(ngx_conf_t *cf,
+    ngx_str_t *name, ngx_uint_t flags);
+
 static ngx_int_t ngx_http_variable_request(ngx_http_request_t *r,
     ngx_http_variable_value_t *v, uintptr_t data);
 #if 0
@@ -356,6 +359,18 @@ static ngx_http_variable_t  ngx_http_cor
       3, NGX_HTTP_VAR_NOCACHEABLE, 0 },
 #endif
 
+    { ngx_string("http_"), NULL, ngx_http_variable_unknown_header_in,
+      0, NGX_HTTP_VAR_PREFIX, 0 },
+
+    { ngx_string("sent_http_"), NULL, ngx_http_variable_unknown_header_out,
+      0, NGX_HTTP_VAR_PREFIX, 0 },
+
+    { ngx_string("cookie_"), NULL, ngx_http_variable_cookie,
+      0, NGX_HTTP_VAR_PREFIX, 0 },
+
+    { ngx_string("arg_"), NULL, ngx_http_variable_argument,
+      0, NGX_HTTP_VAR_NOCACHEABLE|NGX_HTTP_VAR_PREFIX, 0 },
+
     { ngx_null_string, NULL, NULL, 0, 0, 0 }
 };
 
@@ -384,6 +399,10 @@ ngx_http_add_variable(ngx_conf_t *cf, ng
         return NULL;
     }
 
+    if (flags & NGX_HTTP_VAR_PREFIX) {
+        return ngx_http_add_prefix_variable(cf, name, flags);
+    }
+
     cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module);
 
     key = cmcf->variables_keys->keys.elts;
@@ -402,6 +421,8 @@ ngx_http_add_variable(ngx_conf_t *cf, ng
             return NULL;
         }
 
+        v->flags &= flags | ~NGX_HTTP_VAR_WEAK;
+
         return v;
     }
 
@@ -440,6 +461,59 @@ ngx_http_add_variable(ngx_conf_t *cf, ng
 }
 
 
+static ngx_http_variable_t *
+ngx_http_add_prefix_variable(ngx_conf_t *cf, ngx_str_t *name, ngx_uint_t flags)
+{
+    ngx_uint_t                  i;
+    ngx_http_variable_t        *v;
+    ngx_http_core_main_conf_t  *cmcf;
+
+    cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module);
+
+    v = cmcf->prefix_variables.elts;
+    for (i = 0; i < cmcf->prefix_variables.nelts; i++) {
+        if (name->len != v[i].name.len
+            || ngx_strncasecmp(name->data, v[i].name.data, name->len) != 0)
+        {
+            continue;
+        }
+
+        v = &v[i];
+
+        if (!(v->flags & NGX_HTTP_VAR_CHANGEABLE)) {
+            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                               "the duplicate \"%V\" variable", name);
+            return NULL;
+        }
+
+        v->flags &= flags | ~NGX_HTTP_VAR_WEAK;
+
+        return v;
+    }
+
+    v = ngx_array_push(&cmcf->prefix_variables);
+    if (v == NULL) {
+        return NULL;
+    }
+
+    v->name.len = name->len;
+    v->name.data = ngx_pnalloc(cf->pool, name->len);
+    if (v->name.data == NULL) {
+        return NULL;
+    }
+
+    ngx_strlow(v->name.data, name->data, name->len);
+
+    v->set_handler = NULL;
+    v->get_handler = NULL;
+    v->data = 0;
+    v->flags = flags;
+    v->index = 0;
+
+    return v;
+}
+
+
 ngx_int_t
 ngx_http_get_variable_index(ngx_conf_t *cf, ngx_str_t *name)
 {
@@ -573,6 +647,8 @@ ngx_http_get_flushed_variable(ngx_http_r
 ngx_http_variable_value_t *
 ngx_http_get_variable(ngx_http_request_t *r, ngx_str_t *name, ngx_uint_t key)
 {
+    size_t                      len;
+    ngx_uint_t                  i, n;
     ngx_http_variable_t        *v;
     ngx_http_variable_value_t  *vv;
     ngx_http_core_main_conf_t  *cmcf;
@@ -610,64 +686,22 @@ ngx_http_get_variable(ngx_http_request_t
         return NULL;
     }
 
-    if (name->len >= 5 && ngx_strncmp(name->data, "http_", 5) == 0) {
-
-        if (ngx_http_variable_unknown_header_in(r, vv, (uintptr_t) name)
-            == NGX_OK)
+    len = 0;
+
+    v = cmcf->prefix_variables.elts;
+    n = cmcf->prefix_variables.nelts;
+
+    for (i = 0; i < cmcf->prefix_variables.nelts; i++) {
+        if (name->len >= v[i].name.len && name->len > len
+            && ngx_strncmp(name->data, v[i].name.data, v[i].name.len) == 0)
         {
-            return vv;
-        }
-
-        return NULL;
-    }
-
-    if (name->len >= 10 && ngx_strncmp(name->data, "sent_http_", 10) == 0) {
-
-        if (ngx_http_variable_unknown_header_out(r, vv, (uintptr_t) name)
-            == NGX_OK)
-        {
-            return vv;
-        }
-
-        return NULL;
-    }
-
-    if (name->len >= 14 && ngx_strncmp(name->data, "upstream_http_", 14) == 0) {
-
-        if (ngx_http_upstream_header_variable(r, vv, (uintptr_t) name)
-            == NGX_OK)
-        {
-            return vv;
+            len = v[i].name.len;
+            n = i;
         }
-
-        return NULL;
-    }
-
-    if (name->len >= 7 && ngx_strncmp(name->data, "cookie_", 7) == 0) {
-
-        if (ngx_http_variable_cookie(r, vv, (uintptr_t) name) == NGX_OK) {
-            return vv;
-        }
-
-        return NULL;
     }
 
-    if (name->len >= 16
-        && ngx_strncmp(name->data, "upstream_cookie_", 16) == 0)
-    {
-
-        if (ngx_http_upstream_cookie_variable(r, vv, (uintptr_t) name)
-            == NGX_OK)
-        {
-            return vv;
-        }
-
-        return NULL;
-    }
-
-    if (name->len >= 4 && ngx_strncmp(name->data, "arg_", 4) == 0) {
-
-        if (ngx_http_variable_argument(r, vv, (uintptr_t) name) == NGX_OK) {
+    if (n != cmcf->prefix_variables.nelts) {
+        if (v[n].get_handler(r, vv, (uintptr_t) name) == NGX_OK) {
             return vv;
         }
 
@@ -2502,7 +2536,6 @@ ngx_http_regex_exec(ngx_http_request_t *
 ngx_int_t
 ngx_http_variables_add_core_vars(ngx_conf_t *cf)
 {
-    ngx_int_t                   rc;
     ngx_http_variable_t        *cv, *v;
     ngx_http_core_main_conf_t  *cmcf;
 
@@ -2523,27 +2556,20 @@ ngx_http_variables_add_core_vars(ngx_con
         return NGX_ERROR;
     }
 
+    if (ngx_array_init(&cmcf->prefix_variables, cf->pool, 8,
+                       sizeof(ngx_http_variable_t))
+        != NGX_OK)
+    {
+        return NGX_ERROR;
+    }
+
     for (cv = ngx_http_core_variables; cv->name.len; cv++) {
-        v = ngx_palloc(cf->pool, sizeof(ngx_http_variable_t));
+        v = ngx_http_add_variable(cf, &cv->name, cv->flags);
         if (v == NULL) {
             return NGX_ERROR;
         }
 
         *v = *cv;
-
-        rc = ngx_hash_add_key(cmcf->variables_keys, &v->name, v,
-                              NGX_HASH_READONLY_KEY);
-
-        if (rc == NGX_OK) {
-            continue;
-        }
-
-        if (rc == NGX_BUSY) {
-            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                               "conflicting variable name \"%V\"", &v->name);
-        }
-
-        return NGX_ERROR;
     }
 
     return NGX_OK;
@@ -2553,10 +2579,11 @@ ngx_http_variables_add_core_vars(ngx_con
 ngx_int_t
 ngx_http_variables_init_vars(ngx_conf_t *cf)
 {
+    size_t                      len;
     ngx_uint_t                  i, n;
     ngx_hash_key_t             *key;
     ngx_hash_init_t             hash;
-    ngx_http_variable_t        *v, *av;
+    ngx_http_variable_t        *v, *av, *pv;
     ngx_http_core_main_conf_t  *cmcf;
 
     /* set the handlers for the indexed http variables */
@@ -2564,6 +2591,7 @@ ngx_http_variables_init_vars(ngx_conf_t 
     cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module);
 
     v = cmcf->variables.elts;
+    pv = cmcf->prefix_variables.elts;
     key = cmcf->variables_keys->keys.elts;
 
     for (i = 0; i < cmcf->variables.nelts; i++) {
@@ -2584,7 +2612,9 @@ ngx_http_variables_init_vars(ngx_conf_t 
 
                 av->index = i;
 
-                if (av->get_handler == NULL) {
+                if (av->get_handler == NULL
+                    || (av->flags & NGX_HTTP_VAR_WEAK))
+                {
                     break;
                 }
 
@@ -2592,68 +2622,34 @@ ngx_http_variables_init_vars(ngx_conf_t 
             }
         }
 
-        if (v[i].name.len >= 5
-            && ngx_strncmp(v[i].name.data, "http_", 5) == 0)
-        {
-            v[i].get_handler = ngx_http_variable_unknown_header_in;
-            v[i].data = (uintptr_t) &v[i].name;
-
-            continue;
-        }
-
-        if (v[i].name.len >= 10
-            && ngx_strncmp(v[i].name.data, "sent_http_", 10) == 0)
-        {
-            v[i].get_handler = ngx_http_variable_unknown_header_out;
-            v[i].data = (uintptr_t) &v[i].name;
-
-            continue;
-        }
-
-        if (v[i].name.len >= 14
-            && ngx_strncmp(v[i].name.data, "upstream_http_", 14) == 0)
-        {
-            v[i].get_handler = ngx_http_upstream_header_variable;
-            v[i].data = (uintptr_t) &v[i].name;
-            v[i].flags = NGX_HTTP_VAR_NOCACHEABLE;
-
-            continue;
+        len = 0;
+        av = NULL;
+
+        for (n = 0; n < cmcf->prefix_variables.nelts; n++) {
+            if (v[i].name.len >= pv[n].name.len && v[i].name.len > len
+                && ngx_strncmp(v[i].name.data, pv[n].name.data, pv[n].name.len)
+                   == 0)
+            {
+                av = &pv[n];
+                len = pv[n].name.len;
+            }
         }
 
-        if (v[i].name.len >= 7
-            && ngx_strncmp(v[i].name.data, "cookie_", 7) == 0)
-        {
-            v[i].get_handler = ngx_http_variable_cookie;
-            v[i].data = (uintptr_t) &v[i].name;
-
-            continue;
-        }
-
-        if (v[i].name.len >= 16
-            && ngx_strncmp(v[i].name.data, "upstream_cookie_", 16) == 0)
-        {
-            v[i].get_handler = ngx_http_upstream_cookie_variable;
+        if (av) {
+            v[i].get_handler = av->get_handler;
             v[i].data = (uintptr_t) &v[i].name;
-            v[i].flags = NGX_HTTP_VAR_NOCACHEABLE;
-
-            continue;
+            v[i].flags = av->flags;
+
+            goto next;
         }
 
-        if (v[i].name.len >= 4
-            && ngx_strncmp(v[i].name.data, "arg_", 4) == 0)
-        {
-            v[i].get_handler = ngx_http_variable_argument;
-            v[i].data = (uintptr_t) &v[i].name;
-            v[i].flags = NGX_HTTP_VAR_NOCACHEABLE;
-
-            continue;
+        if (v[i].get_handler == NULL) {
+            ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
+                          "unknown \"%V\" variable", &v[i].name);
+
+            return NGX_ERROR;
         }
 
-        ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
-                      "unknown \"%V\" variable", &v[i].name);
-
-        return NGX_ERROR;
-
     next:
         continue;
     }
diff -r 25203fc377fb -r d2b2ff157da5 src/http/ngx_http_variables.h
--- a/src/http/ngx_http_variables.h	Tue Jan 31 12:09:40 2017 +0300
+++ b/src/http/ngx_http_variables.h	Tue Jan 31 21:19:58 2017 +0300
@@ -30,6 +30,8 @@ typedef ngx_int_t (*ngx_http_get_variabl
 #define NGX_HTTP_VAR_NOCACHEABLE  2
 #define NGX_HTTP_VAR_INDEXED      4
 #define NGX_HTTP_VAR_NOHASH       8
+#define NGX_HTTP_VAR_WEAK         16
+#define NGX_HTTP_VAR_PREFIX       32
 
 
 struct ngx_http_variable_s {
diff -r 25203fc377fb -r d2b2ff157da5 src/stream/ngx_stream.h
--- a/src/stream/ngx_stream.h	Tue Jan 31 12:09:40 2017 +0300
+++ b/src/stream/ngx_stream.h	Tue Jan 31 21:19:58 2017 +0300
@@ -153,7 +153,8 @@ typedef struct {
 
     ngx_hash_t                     variables_hash;
 
-    ngx_array_t                    variables;   /* ngx_stream_variable_t */
+    ngx_array_t                    variables;        /* ngx_stream_variable_t */
+    ngx_array_t                    prefix_variables; /* ngx_stream_variable_t */
     ngx_uint_t                     ncaptures;
 
     ngx_uint_t                     variables_hash_max_size;
diff -r 25203fc377fb -r d2b2ff157da5 src/stream/ngx_stream_variables.c
--- a/src/stream/ngx_stream_variables.c	Tue Jan 31 12:09:40 2017 +0300
+++ b/src/stream/ngx_stream_variables.c	Tue Jan 31 21:19:58 2017 +0300
@@ -10,6 +10,8 @@
 #include <ngx_stream.h>
 #include <nginx.h>
 
+static ngx_stream_variable_t *ngx_stream_add_prefix_variable(ngx_conf_t *cf,
+    ngx_str_t *name, ngx_uint_t flags);
 
 static ngx_int_t ngx_stream_variable_binary_remote_addr(
     ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data);
@@ -137,6 +139,10 @@ ngx_stream_add_variable(ngx_conf_t *cf, 
         return NULL;
     }
 
+    if (flags & NGX_STREAM_VAR_PREFIX) {
+        return ngx_stream_add_prefix_variable(cf, name, flags);
+    }
+
     cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module);
 
     key = cmcf->variables_keys->keys.elts;
@@ -155,6 +161,8 @@ ngx_stream_add_variable(ngx_conf_t *cf, 
             return NULL;
         }
 
+        v->flags &= flags | ~NGX_STREAM_VAR_WEAK;
+
         return v;
     }
 
@@ -193,6 +201,60 @@ ngx_stream_add_variable(ngx_conf_t *cf, 
 }
 
 
+static ngx_stream_variable_t *
+ngx_stream_add_prefix_variable(ngx_conf_t *cf, ngx_str_t *name,
+    ngx_uint_t flags)
+{
+    ngx_uint_t                    i;
+    ngx_stream_variable_t        *v;
+    ngx_stream_core_main_conf_t  *cmcf;
+
+    cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module);
+
+    v = cmcf->prefix_variables.elts;
+    for (i = 0; i < cmcf->prefix_variables.nelts; i++) {
+        if (name->len != v[i].name.len
+            || ngx_strncasecmp(name->data, v[i].name.data, name->len) != 0)
+        {
+            continue;
+        }
+
+        v = &v[i];
+
+        if (!(v->flags & NGX_STREAM_VAR_CHANGEABLE)) {
+            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                               "the duplicate \"%V\" variable", name);
+            return NULL;
+        }
+
+        v->flags &= flags | ~NGX_STREAM_VAR_WEAK;
+
+        return v;
+    }
+
+    v = ngx_array_push(&cmcf->prefix_variables);
+    if (v == NULL) {
+        return NULL;
+    }
+
+    v->name.len = name->len;
+    v->name.data = ngx_pnalloc(cf->pool, name->len);
+    if (v->name.data == NULL) {
+        return NULL;
+    }
+
+    ngx_strlow(v->name.data, name->data, name->len);
+
+    v->set_handler = NULL;
+    v->get_handler = NULL;
+    v->data = 0;
+    v->flags = flags;
+    v->index = 0;
+
+    return v;
+}
+
+
 ngx_int_t
 ngx_stream_get_variable_index(ngx_conf_t *cf, ngx_str_t *name)
 {
@@ -327,6 +389,8 @@ ngx_stream_variable_value_t *
 ngx_stream_get_variable(ngx_stream_session_t *s, ngx_str_t *name,
     ngx_uint_t key)
 {
+    size_t                        len;
+    ngx_uint_t                    i, n;
     ngx_stream_variable_t        *v;
     ngx_stream_variable_value_t  *vv;
     ngx_stream_core_main_conf_t  *cmcf;
@@ -365,6 +429,28 @@ ngx_stream_get_variable(ngx_stream_sessi
         return NULL;
     }
 
+    len = 0;
+
+    v = cmcf->prefix_variables.elts;
+    n = cmcf->prefix_variables.nelts;
+
+    for (i = 0; i < cmcf->prefix_variables.nelts; i++) {
+        if (name->len >= v[i].name.len && name->len > len
+            && ngx_strncmp(name->data, v[i].name.data, v[i].name.len) == 0)
+        {
+            len = v[i].name.len;
+            n = i;
+        }
+    }
+
+    if (n != cmcf->prefix_variables.nelts) {
+        if (v[n].get_handler(s, vv, (uintptr_t) name) == NGX_OK) {
+            return vv;
+        }
+
+        return NULL;
+    }
+
     vv->not_found = 1;
 
     return vv;
@@ -1000,7 +1086,6 @@ ngx_stream_regex_exec(ngx_stream_session
 ngx_int_t
 ngx_stream_variables_add_core_vars(ngx_conf_t *cf)
 {
-    ngx_int_t                     rc;
     ngx_stream_variable_t        *cv, *v;
     ngx_stream_core_main_conf_t  *cmcf;
 
@@ -1021,27 +1106,20 @@ ngx_stream_variables_add_core_vars(ngx_c
         return NGX_ERROR;
     }
 
+    if (ngx_array_init(&cmcf->prefix_variables, cf->pool, 8,
+                       sizeof(ngx_stream_variable_t))
+        != NGX_OK)
+    {
+        return NGX_ERROR;
+    }
+
     for (cv = ngx_stream_core_variables; cv->name.len; cv++) {
-        v = ngx_palloc(cf->pool, sizeof(ngx_stream_variable_t));
+        v = ngx_stream_add_variable(cf, &cv->name, cv->flags);
         if (v == NULL) {
             return NGX_ERROR;
         }
 
         *v = *cv;
-
-        rc = ngx_hash_add_key(cmcf->variables_keys, &v->name, v,
-                              NGX_HASH_READONLY_KEY);
-
-        if (rc == NGX_OK) {
-            continue;
-        }
-
-        if (rc == NGX_BUSY) {
-            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                               "conflicting variable name \"%V\"", &v->name);
-        }
-
-        return NGX_ERROR;
     }
 
     return NGX_OK;
@@ -1051,10 +1129,11 @@ ngx_stream_variables_add_core_vars(ngx_c
 ngx_int_t
 ngx_stream_variables_init_vars(ngx_conf_t *cf)
 {
+    size_t                        len;
     ngx_uint_t                    i, n;
     ngx_hash_key_t               *key;
     ngx_hash_init_t               hash;
-    ngx_stream_variable_t        *v, *av;
+    ngx_stream_variable_t        *v, *av, *pv;
     ngx_stream_core_main_conf_t  *cmcf;
 
     /* set the handlers for the indexed stream variables */
@@ -1062,6 +1141,7 @@ ngx_stream_variables_init_vars(ngx_conf_
     cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module);
 
     v = cmcf->variables.elts;
+    pv = cmcf->prefix_variables.elts;
     key = cmcf->variables_keys->keys.elts;
 
     for (i = 0; i < cmcf->variables.nelts; i++) {
@@ -1082,7 +1162,9 @@ ngx_stream_variables_init_vars(ngx_conf_
 
                 av->index = i;
 
-                if (av->get_handler == NULL) {
+                if (av->get_handler == NULL
+                    || (av->flags & NGX_STREAM_VAR_WEAK))
+                {
                     break;
                 }
 
@@ -1090,10 +1172,32 @@ ngx_stream_variables_init_vars(ngx_conf_
             }
         }
 
-        ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
-                      "unknown \"%V\" variable", &v[i].name);
+        len = 0;
+        av = NULL;
 
-        return NGX_ERROR;
+        for (n = 0; n < cmcf->prefix_variables.nelts; n++) {
+            if (v[i].name.len >= pv[n].name.len && v[i].name.len > len
+                && ngx_strncmp(v[i].name.data, pv[n].name.data, pv[n].name.len)
+                   == 0)
+            {
+                av = &pv[n];
+                len = pv[n].name.len;
+            }
+        }
+
+        if (av) {
+            v[i].get_handler = av->get_handler;
+            v[i].data = (uintptr_t) &v[i].name;
+            v[i].flags = av->flags;
+
+            goto next;
+         }
+
+        if (v[i].get_handler == NULL) {
+            ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
+                          "unknown \"%V\" variable", &v[i].name);
+            return NGX_ERROR;
+        }
 
     next:
         continue;
diff -r 25203fc377fb -r d2b2ff157da5 src/stream/ngx_stream_variables.h
--- a/src/stream/ngx_stream_variables.h	Tue Jan 31 12:09:40 2017 +0300
+++ b/src/stream/ngx_stream_variables.h	Tue Jan 31 21:19:58 2017 +0300
@@ -30,6 +30,8 @@ typedef ngx_int_t (*ngx_stream_get_varia
 #define NGX_STREAM_VAR_NOCACHEABLE  2
 #define NGX_STREAM_VAR_INDEXED      4
 #define NGX_STREAM_VAR_NOHASH       8
+#define NGX_STREAM_VAR_WEAK         16
+#define NGX_STREAM_VAR_PREFIX       32
 
 
 struct ngx_stream_variable_s {