Hello!
On Tue, Sep 18, 2018 at 08:12:20AM -0400, Thomas Ward wrote:
> Downstream in Ubuntu, it has been proposed to demote pcre3 and
> use pcre2 instead as it is newer.
> https://trac.nginx.org/nginx/ticket/720 shows it was marked 4
> years ago that NGINX does not support pcre2. Are there any
> plans to use pcre2 instead of pcre3?
There are no immediate plans.
When we last checked, there were no problems with PCRE, but PCRE2
wasn't available in most distributions we support, making the
switch mostly meaningless.
Also, it looks like PCRE2 is still not supported even by Exim,
which is the parent project of PCRE and PCRE2:
https://bugs.exim.org/show_bug.cgi?id=1878
As such, adding PCRE2 support to nginx looks premature.
--
Maxim Dounin
http://mdounin.ru/
Hello,
following my early september attemps to introduce config-time resolved
paths, and your (Maxim) thoughts that my solution introduced
unclearness, I have come up with what I hope to be a better solution.
In essence, this introduces a {{ … }} syntax for config-time
replacements.
=== Basic use case ===
For now I only implemented {{.}} to resolve to "directory of the
currently parsed config file", and only for the 'include' directive.
But this already allows "config snippets" reusability, bringing a
simple solution to app-embedded nginx configuration snippets, e.g. given
one nginx.conf seen by "include
/var/www/whereever/you/have/put/your/*/nginx.conf;":
----------------
server
{
server_name appv1.local;
include "{{.}}/php-fpm.conf";
}
server
{
server_name appv2.local;
include "{{.}}/php-fpm.conf";
}
----------------
we can simply have a php-fpm.conf deployed next to the app's
nginx.conf, without having to go for one of the three current
approaches:
a. "hard-resolving" the full path at deploy time, resulting in
nginx.conf containing:
include /var/www/whereever/you/have/put/your/app/php-fpm.conf;
(hard to read, and error prone if wanting to hand-correct some
entries)
b. pushing the php-fpm.conf to the configuration root, to include it
from the app's nginx.conf as simply "php-fpm.conf".
But then applications do not control anymore their snippets, they
must adhere to the config root's version.
c. … or replace each include by the whole contents of the snippet.
=== Delimiter ===
Now for the drawback: I am not that fond of the {{ … }} delimiter. And
it maybe the best time to choose a better alternative.
I think we should stick to <two characters opening><var name><two
characters closing>, as opposed to <one or two chars prefix><var name>:
- because a two-characters prefix has more chances to collide with an
existing configuration file, than a 2+2 chars frame
- and because this visually distinguishes the "define-replaced" from
the "runtime-replaced" variables (just imagine a world where ${var}
meant runtime-replace var, when $[var] meant config-time replacement)
In the 2+2 characters frame, here were candidates:
directive {{.}}/php-fpm.{{php_version}}.conf;
directive $$.$$/php-fpm.$$php_version$$.conf;
directive ##.##/php-fpm.##php_version##.conf;
directive ``.``/php-fpm.``php_version``.conf;
directive ::.::/php-fpm.::php_version::.conf;
directive %%.%%/php-fpm.%%php_version%%.conf;
Other pairs too much looked like http URIs or other special characters.
The big problem with {{ … }} is that is FORCES to use quotes, or the
'{' will be seen as a block opening.
Its big advantage is that, as its opening and closing marks differ, it
is easier to read {{.}}/{{type}}.{{version}}.conf than
##.##/##type##.##version##.conf
Other than that, % is an interesting alternative, as long as in testing
we don't name our variables from any of the 5 or 6 replacements that the
testing framework harcodes.
=== Plans ===
Even with these, the solution seems far more generic than the 'nearby'
I originally proposed, and whose shortnesses were easily pointed out.
Plans are now to rely on this syntax to add block-scoped 'define's:
----------------
# Define default PHP version
define php_version 7.3;
server
{
server_name appv1.local;
include "{{.}}/php-fpm.conf";
}
server
{
server_name appv2.local;
include "{{.}}/php-fpm.conf";
}
server
{
server_name oldapp.local;
define php_version 5.6;
include "{{.}}/php-fpm.conf";
}
----------------
with php-fpm.conf containing:
----------------
…
fastcgi_pass "unix:/var/run/php{{php_version}}-fpm.sock";
…
----------------
Note that resolution is simply done at reading time, so:
include "php{{php_version}}.conf"; # Will error
define php_version 7.3;
include "php{{php_version}}.conf"; # Will include php7.3.conf, relative
to conf root
define php_version 5.6;
include "php{{php_version}}.conf"; # Will include php5.6.conf
=== Contents of the patches ===
Attached patches are:
1. definition of a new ngx_conf_complex_value() function (2 added
files: core/ngx_conf_def.h and core/ngx_conf_def.c)
This is the core of the replacer, *without* its plugging (that is,
patch 1. without patches 2. and 3. is only dead code).
It includes {{.}}.
2. plugging of 1. into 'include' directive.
3. plugging of 1. into ngx_http_compile_complex_value
This works on an opt-in basis, with a new attribute,
ccv->compile_defs, that is 0 by default
=== What's next? ===
TODO is:
- first of all, vote for a syntax!
- set compile_defs = 1 on elected ngx_http_compile_complex_value
callers (or even make it the default, inverting the logic to "resolve {{
… }}s unless compile_no_defs is 1")
- define 'define' keyword, and mechanism for ngx_conf_complex_value()
to resolve them
- document (configuration and API), test
=== Why not http script? ===
I originally thought of adding my code into
ngx_http_compile_complex_value to mutualize parsing and benefit of the
powerful variables resolution, but:
- it created a dependency from core to http
- and anyway, {{ }} resolution has to pass before http scripts parsing,
so that {{ }} resolved string can contain $ or pass through
config_prefix.
--
Guillaume
# HG changeset patch
# User Guillaume Outters <guillaume-nginx(a)outters.eu>
# Date 1572243857 -3600
# Mon Oct 28 07:24:17 2019 +0100
# Node ID 8bb356ca5a127afa4c21b57de1df950c6e059595
# Parent 89adf49fe76ada86d84e2af8f5cee9ca8c3dca19
ngx_conf_def.c: add {{ … }} syntax for configuration-time resolved
variable definitions
diff -r 89adf49fe76a -r 8bb356ca5a12 auto/sources
--- a/auto/sources Mon Oct 21 20:22:30 2019 +0300
+++ b/auto/sources Mon Oct 28 07:24:17 2019 +0100
@@ -36,6 +36,7 @@
src/core/ngx_connection.h \
src/core/ngx_cycle.h \
src/core/ngx_conf_file.h \
+ src/core/ngx_conf_def.h \
src/core/ngx_module.h \
src/core/ngx_resolver.h \
src/core/ngx_open_file_cache.h \
@@ -73,6 +74,7 @@
src/core/ngx_rwlock.c \
src/core/ngx_cpuinfo.c \
src/core/ngx_conf_file.c \
+ src/core/ngx_conf_def.c \
src/core/ngx_module.c \
src/core/ngx_resolver.c \
src/core/ngx_open_file_cache.c \
diff -r 89adf49fe76a -r 8bb356ca5a12 src/core/ngx_conf_def.c
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/core/ngx_conf_def.c Mon Oct 28 07:24:17 2019 +0100
@@ -0,0 +1,298 @@
+
+/*
+ * Copyright (C) Guillaume Outters
+ * Copyright (C) Nginx, Inc.
+ */
+
+
+#include <ngx_config.h>
+#include <ngx_core.h>
+#include <ngx_conf_def.h>
+
+
+#define NGX_CONF_SCRIPT_DELIM_LEN 2
+
+#define NGX_CONF_TYPE_TEXT 0
+#define NGX_CONF_TYPE_EXPR 1
+
+
+/* TODO: mutualize with ngx_http_script for parsing / running the mix
of
+ * strings and variables. */
+
+typedef struct {
+ ngx_str_t *value;
+ ngx_conf_t *cf;
+ ngx_array_t parts;
+ ngx_array_t part_types;
+} ngx_conf_ccv_t;
+
+int ngx_conf_ccv_compile(ngx_conf_ccv_t *ccv);
+int ngx_conf_ccv_init(ngx_conf_ccv_t *ccv, ngx_conf_t *cf, ngx_str_t
*value,
+ ngx_uint_t n);
+int ngx_conf_ccv_run(ngx_conf_ccv_t *ccv);
+int ngx_conf_ccv_resolve_expr(ngx_conf_ccv_t *ccv, ngx_str_t *expr);
+void ngx_conf_ccv_destroy(ngx_conf_ccv_t *ccv);
+
+
+int
+ngx_conf_complex_value(ngx_conf_t *cf, ngx_str_t *string)
+{
+ ngx_uint_t i, nv;
+ ngx_conf_ccv_t ccv;
+
+ nv = 0;
+
+ for (i = 0; i < string->len - 1; ++i) {
+ if (string->data[i] == '{' && string->data[i + 1] == '{') {
+ ++nv;
+ }
+ }
+
+ if (nv == 0) {
+ return NGX_OK;
+ }
+
+ if (ngx_conf_ccv_init(&ccv, cf, string, 2 * nv + 1) != NGX_OK) {
+ goto e_ccv;
+ }
+
+ if (ngx_conf_ccv_compile(&ccv) != NGX_OK) {
+ goto e_compile;
+ }
+
+ if (ngx_conf_ccv_run(&ccv) != NGX_OK) {
+ goto e_run;
+ }
+
+ ngx_conf_ccv_destroy(&ccv);
+
+ return NGX_OK;
+
+e_run:
+e_compile:
+ ngx_conf_ccv_destroy(&ccv);
+e_ccv:
+ return NGX_ERROR;
+}
+
+
+int
+ngx_conf_ccv_init(ngx_conf_ccv_t *ccv, ngx_conf_t *cf, ngx_str_t
*value,
+ ngx_uint_t n)
+{
+ ccv->value = value;
+ ccv->cf = cf;
+
+ if (ngx_array_init(&ccv->parts, cf->pool, n, sizeof(ngx_str_t)) !=
NGX_OK) {
+ goto e_alloc_parts;
+ }
+ if (ngx_array_init(&ccv->part_types, cf->pool, n,
sizeof(ngx_uint_t))
+ != NGX_OK)
+ {
+ goto e_alloc_part_types;
+ }
+
+ return NGX_OK;
+
+ ngx_array_destroy(&ccv->part_types);
+e_alloc_part_types:
+ ngx_array_destroy(&ccv->parts);
+e_alloc_parts:
+ return NGX_ERROR;
+}
+
+
+void
+ngx_conf_ccv_destroy(ngx_conf_ccv_t *ccv)
+{
+ ngx_array_destroy(&ccv->parts);
+ ngx_array_destroy(&ccv->part_types);
+}
+
+
+int
+ngx_conf_ccv_compile(ngx_conf_ccv_t *ccv)
+{
+ ngx_uint_t i, current_part_start, current_part_end;
+ ngx_uint_t current_part_type;
+
+ ccv->parts.nelts = 0;
+ ccv->part_types.nelts = 0;
+ current_part_type = NGX_CONF_TYPE_TEXT;
+
+ for (current_part_start = 0; current_part_start < ccv->value->len;
+ /* void */ )
+ {
+ switch (current_part_type) {
+
+ case NGX_CONF_TYPE_TEXT:
+
+ for (i = current_part_start;
+ i < ccv->value->len
+ && (ccv->value->data[i] != '{' || ccv->value->data[i + 1] !=
'{');
+ /* void */ )
+ {
+ ++i;
+ }
+
+ if (i > current_part_start) {
+ ((ngx_str_t *) ccv->parts.elts)[ccv->parts.nelts].data =
+ &ccv->value->data[current_part_start];
+ ((ngx_str_t *) ccv->parts.elts)[ccv->parts.nelts].len =
+ i - current_part_start;
+ ++ccv->parts.nelts;
+ ((ngx_uint_t *) ccv->part_types.elts)[ccv->part_types.nelts] =
+ current_part_type;
+ ++ccv->part_types.nelts;
+ }
+ if (i < ccv->value->len) {
+ current_part_type = NGX_CONF_TYPE_EXPR;
+ }
+ current_part_start = i;
+
+ break;
+
+ case NGX_CONF_TYPE_EXPR:
+
+ for (i = current_part_start + NGX_CONF_SCRIPT_DELIM_LEN;
+ i < ccv->value->len - 1; ++i)
+ {
+ if (ccv->value->data[i] == '}') {
+ if (ccv->value->data[i + 1] == '}') {
+ break;
+ } else {
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "forbidden \"}\" in \"%V\" at character %d",
+ ccv->value, current_part_start + 1);
+ goto e_script_parse;
+ }
+ } else if (ccv->value->data[i] == '{') {
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "forbidden character \"%c\" in \"%V\""
+ " at character %d",
+ ccv->value, current_part_start + 1);
+ goto e_script_parse;
+ }
+ }
+ if (i >= ccv->value->len - 1) {
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "unbalanced {{ in \"%V\" at character %d",
+ ccv->value, current_part_start + 1);
+ goto e_script_parse;
+ }
+
+ current_part_start += NGX_CONF_SCRIPT_DELIM_LEN;
+ while (current_part_start < ccv->value->len
+ && ccv->value->data[current_part_start] == ' ')
+ {
+ ++current_part_start;
+ }
+ for (current_part_end = i;
+ current_part_end > current_part_start
+ && ccv->value->data[current_part_end - 1] == ' ';
+ --current_part_end)
+ {
+ /* void */
+ }
+
+ if (current_part_end <= current_part_start) {
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "invalid variable name in \"%V\" at character %d",
+ ccv->value, current_part_start + 1);
+ goto e_script_parse;
+ }
+
+ ((ngx_str_t *) ccv->parts.elts)[ccv->parts.nelts].data =
+ &ccv->value->data[current_part_start];
+ ((ngx_str_t *) ccv->parts.elts)[ccv->parts.nelts].len =
+ current_part_end - current_part_start;
+ ++ccv->parts.nelts;
+ ((ngx_uint_t *) ccv->part_types.elts)[ccv->part_types.nelts] =
+ current_part_type;
+ ++ccv->part_types.nelts;
+
+ current_part_start = i + NGX_CONF_SCRIPT_DELIM_LEN;
+ current_part_type = NGX_CONF_TYPE_TEXT;
+
+ break;
+ }
+ }
+
+ return NGX_OK;
+
+e_script_parse:
+
+ return NGX_ERROR;
+}
+
+
+int
+ngx_conf_ccv_run(ngx_conf_ccv_t *ccv)
+{
+ ngx_uint_t i;
+ ngx_str_t *val;
+ size_t len;
+ unsigned char *ptr;
+
+ len = 0;
+
+ for (i = 0; i < ccv->part_types.nelts; ++i) {
+ switch (((ngx_uint_t *) ccv->part_types.elts)[i]) {
+
+ case NGX_CONF_TYPE_TEXT:
+ val = &((ngx_str_t *) ccv->parts.elts)[i];
+ len += val->len;
+ break;
+
+ case NGX_CONF_TYPE_EXPR:
+ val = &(((ngx_str_t *) ccv->parts.elts)[i]);
+ if (ngx_conf_ccv_resolve_expr(ccv, val) != NGX_OK) {
+ return NGX_ERROR;
+ }
+ len += val->len;
+ break;
+ }
+ }
+
+ ptr = ngx_pnalloc(ccv->cf->pool, len);
+ if (ptr == NULL) {
+ return NGX_ERROR;
+ }
+
+ ccv->value->len = len;
+ ccv->value->data = ptr;
+
+ for (i = 0; i < ccv->part_types.nelts; ++i) {
+ switch (((ngx_uint_t *) ccv->part_types.elts)[i]) {
+
+ case NGX_CONF_TYPE_TEXT:
+ case NGX_CONF_TYPE_EXPR:
+ val = &((ngx_str_t *) ccv->parts.elts)[i];
+ ptr = ngx_copy(ptr, val->data, val->len);
+ break;
+ }
+ }
+
+ return NGX_OK;
+}
+
+
+int
+ngx_conf_ccv_resolve_expr(ngx_conf_ccv_t *ccv, ngx_str_t *expr)
+{
+ if (expr->len == 1 && ngx_strncmp(expr->data, ".", 1) == 0) {
+ expr->len = ccv->cf->conf_file->file.name.len;
+ expr->data = ccv->cf->conf_file->file.name.data;
+ for (expr->len = ccv->cf->conf_file->file.name.len;
+ expr->data[--expr->len] != '/';
+ /* void */ )
+ { /* void */ }
+ return NGX_OK;
+ } else {
+ /* TODO: find the value of the last "define" of this context for
this
+ * variable name. */
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "not implemented: cannot resolve {{ %V }}", expr);
+ return NGX_ERROR;
+ }
+}
diff -r 89adf49fe76a -r 8bb356ca5a12 src/core/ngx_conf_def.h
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/core/ngx_conf_def.h Mon Oct 28 07:24:17 2019 +0100
@@ -0,0 +1,18 @@
+
+/*
+ * Copyright (C) Guillaume Outters
+ * Copyright (C) Nginx, Inc.
+ */
+
+
+#ifndef _NGX_CONF_DEF_H_INCLUDED_
+#define _NGX_CONF_DEF_H_INCLUDED_
+
+
+#include <ngx_core.h>
+
+
+int ngx_conf_complex_value(ngx_conf_t *cf, ngx_str_t *string);
+
+
+#endif /* _NGX_CONF_DEF_H_INCLUDED_ */
# HG changeset patch
# User Guillaume Outters <guillaume-nginx(a)outters.eu>
# Date 1572243926 -3600
# Mon Oct 28 07:25:26 2019 +0100
# Node ID cac499a2b296d34a4f7a7a861b860e8726fd0db7
# Parent 8bb356ca5a127afa4c21b57de1df950c6e059595
include: allow {{.}} to reference current file's directory
diff -r 8bb356ca5a12 -r cac499a2b296 src/core/ngx_conf_file.c
--- a/src/core/ngx_conf_file.c Mon Oct 28 07:24:17 2019 +0100
+++ b/src/core/ngx_conf_file.c Mon Oct 28 07:25:26 2019 +0100
@@ -7,6 +7,7 @@
#include <ngx_config.h>
#include <ngx_core.h>
+#include <ngx_conf_def.h>
#define NGX_CONF_BUFFER 4096
@@ -830,6 +831,9 @@
ngx_log_debug1(NGX_LOG_DEBUG_CORE, cf->log, 0, "include %s",
file.data);
+ if (ngx_conf_complex_value(cf, &file) != NGX_OK) {
+ return NGX_CONF_ERROR;
+ }
if (ngx_conf_full_name(cf->cycle, &file, 1) != NGX_OK) {
return NGX_CONF_ERROR;
}
# HG changeset patch
# User Guillaume Outters <guillaume-nginx(a)outters.eu>
# Date 1572244106 -3600
# Mon Oct 28 07:28:26 2019 +0100
# Node ID ca9d5059523d8630c63867db885cd231ca93173a
# Parent cac499a2b296d34a4f7a7a861b860e8726fd0db7
Core: allow opt-in {{ . }} syntax in
ngx_http_compile_complex_value()-parsed configuration
diff -r cac499a2b296 -r ca9d5059523d src/http/ngx_http_script.c
--- a/src/http/ngx_http_script.c Mon Oct 28 07:25:26 2019 +0100
+++ b/src/http/ngx_http_script.c Mon Oct 28 07:28:26 2019 +0100
@@ -8,6 +8,7 @@
#include <ngx_config.h>
#include <ngx_core.h>
#include <ngx_http.h>
+#include <ngx_conf_def.h>
static ngx_int_t ngx_http_script_init_arrays(ngx_http_script_compile_t
*sc);
@@ -145,6 +146,14 @@
v = ccv->value;
+ if (ccv->compile_defs) {
+ /* Compile definitions before looking for variables, so that a
+ * definition's dereference can contain a variable */
+ if (ngx_conf_complex_value(ccv->cf, v) != NGX_OK) {
+ return NGX_ERROR;
+ }
+ }
+
nv = 0;
nc = 0;
diff -r cac499a2b296 -r ca9d5059523d src/http/ngx_http_script.h
--- a/src/http/ngx_http_script.h Mon Oct 28 07:25:26 2019 +0100
+++ b/src/http/ngx_http_script.h Mon Oct 28 07:28:26 2019 +0100
@@ -83,6 +83,7 @@
unsigned zero:1;
unsigned conf_prefix:1;
unsigned root_prefix:1;
+ unsigned compile_defs:1;
} ngx_http_compile_complex_value_t;