[PATCH] {{x}} syntax for configuration-time replacements
Guillaume Outters
guillaume-nginx at outters.eu
Mon Oct 28 23:51:42 UTC 2019
Hello,
following my early september attemps to introduce config-time resolved
paths, and your (Maxim) thoughts that my solution introduced
unclearness, I have come up with what I hope to be a better solution.
In essence, this introduces a {{ … }} syntax for config-time
replacements.
=== Basic use case ===
For now I only implemented {{.}} to resolve to "directory of the
currently parsed config file", and only for the 'include' directive.
But this already allows "config snippets" reusability, bringing a
simple solution to app-embedded nginx configuration snippets, e.g. given
one nginx.conf seen by "include
/var/www/whereever/you/have/put/your/*/nginx.conf;":
----------------
server
{
server_name appv1.local;
include "{{.}}/php-fpm.conf";
}
server
{
server_name appv2.local;
include "{{.}}/php-fpm.conf";
}
----------------
we can simply have a php-fpm.conf deployed next to the app's
nginx.conf, without having to go for one of the three current
approaches:
a. "hard-resolving" the full path at deploy time, resulting in
nginx.conf containing:
include /var/www/whereever/you/have/put/your/app/php-fpm.conf;
(hard to read, and error prone if wanting to hand-correct some
entries)
b. pushing the php-fpm.conf to the configuration root, to include it
from the app's nginx.conf as simply "php-fpm.conf".
But then applications do not control anymore their snippets, they
must adhere to the config root's version.
c. … or replace each include by the whole contents of the snippet.
=== Delimiter ===
Now for the drawback: I am not that fond of the {{ … }} delimiter. And
it maybe the best time to choose a better alternative.
I think we should stick to <two characters opening><var name><two
characters closing>, as opposed to <one or two chars prefix><var name>:
- because a two-characters prefix has more chances to collide with an
existing configuration file, than a 2+2 chars frame
- and because this visually distinguishes the "define-replaced" from
the "runtime-replaced" variables (just imagine a world where ${var}
meant runtime-replace var, when $[var] meant config-time replacement)
In the 2+2 characters frame, here were candidates:
directive {{.}}/php-fpm.{{php_version}}.conf;
directive $$.$$/php-fpm.$$php_version$$.conf;
directive ##.##/php-fpm.##php_version##.conf;
directive ``.``/php-fpm.``php_version``.conf;
directive ::.::/php-fpm.::php_version::.conf;
directive %%.%%/php-fpm.%%php_version%%.conf;
Other pairs too much looked like http URIs or other special characters.
The big problem with {{ … }} is that is FORCES to use quotes, or the
'{' will be seen as a block opening.
Its big advantage is that, as its opening and closing marks differ, it
is easier to read {{.}}/{{type}}.{{version}}.conf than
##.##/##type##.##version##.conf
Other than that, % is an interesting alternative, as long as in testing
we don't name our variables from any of the 5 or 6 replacements that the
testing framework harcodes.
=== Plans ===
Even with these, the solution seems far more generic than the 'nearby'
I originally proposed, and whose shortnesses were easily pointed out.
Plans are now to rely on this syntax to add block-scoped 'define's:
----------------
# Define default PHP version
define php_version 7.3;
server
{
server_name appv1.local;
include "{{.}}/php-fpm.conf";
}
server
{
server_name appv2.local;
include "{{.}}/php-fpm.conf";
}
server
{
server_name oldapp.local;
define php_version 5.6;
include "{{.}}/php-fpm.conf";
}
----------------
with php-fpm.conf containing:
----------------
…
fastcgi_pass "unix:/var/run/php{{php_version}}-fpm.sock";
…
----------------
Note that resolution is simply done at reading time, so:
include "php{{php_version}}.conf"; # Will error
define php_version 7.3;
include "php{{php_version}}.conf"; # Will include php7.3.conf, relative
to conf root
define php_version 5.6;
include "php{{php_version}}.conf"; # Will include php5.6.conf
=== Contents of the patches ===
Attached patches are:
1. definition of a new ngx_conf_complex_value() function (2 added
files: core/ngx_conf_def.h and core/ngx_conf_def.c)
This is the core of the replacer, *without* its plugging (that is,
patch 1. without patches 2. and 3. is only dead code).
It includes {{.}}.
2. plugging of 1. into 'include' directive.
3. plugging of 1. into ngx_http_compile_complex_value
This works on an opt-in basis, with a new attribute,
ccv->compile_defs, that is 0 by default
=== What's next? ===
TODO is:
- first of all, vote for a syntax!
- set compile_defs = 1 on elected ngx_http_compile_complex_value
callers (or even make it the default, inverting the logic to "resolve {{
… }}s unless compile_no_defs is 1")
- define 'define' keyword, and mechanism for ngx_conf_complex_value()
to resolve them
- document (configuration and API), test
=== Why not http script? ===
I originally thought of adding my code into
ngx_http_compile_complex_value to mutualize parsing and benefit of the
powerful variables resolution, but:
- it created a dependency from core to http
- and anyway, {{ }} resolution has to pass before http scripts parsing,
so that {{ }} resolved string can contain $ or pass through
config_prefix.
--
Guillaume
# HG changeset patch
# User Guillaume Outters <guillaume-nginx at outters.eu>
# Date 1572243857 -3600
# Mon Oct 28 07:24:17 2019 +0100
# Node ID 8bb356ca5a127afa4c21b57de1df950c6e059595
# Parent 89adf49fe76ada86d84e2af8f5cee9ca8c3dca19
ngx_conf_def.c: add {{ … }} syntax for configuration-time resolved
variable definitions
diff -r 89adf49fe76a -r 8bb356ca5a12 auto/sources
--- a/auto/sources Mon Oct 21 20:22:30 2019 +0300
+++ b/auto/sources Mon Oct 28 07:24:17 2019 +0100
@@ -36,6 +36,7 @@
src/core/ngx_connection.h \
src/core/ngx_cycle.h \
src/core/ngx_conf_file.h \
+ src/core/ngx_conf_def.h \
src/core/ngx_module.h \
src/core/ngx_resolver.h \
src/core/ngx_open_file_cache.h \
@@ -73,6 +74,7 @@
src/core/ngx_rwlock.c \
src/core/ngx_cpuinfo.c \
src/core/ngx_conf_file.c \
+ src/core/ngx_conf_def.c \
src/core/ngx_module.c \
src/core/ngx_resolver.c \
src/core/ngx_open_file_cache.c \
diff -r 89adf49fe76a -r 8bb356ca5a12 src/core/ngx_conf_def.c
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/core/ngx_conf_def.c Mon Oct 28 07:24:17 2019 +0100
@@ -0,0 +1,298 @@
+
+/*
+ * Copyright (C) Guillaume Outters
+ * Copyright (C) Nginx, Inc.
+ */
+
+
+#include <ngx_config.h>
+#include <ngx_core.h>
+#include <ngx_conf_def.h>
+
+
+#define NGX_CONF_SCRIPT_DELIM_LEN 2
+
+#define NGX_CONF_TYPE_TEXT 0
+#define NGX_CONF_TYPE_EXPR 1
+
+
+/* TODO: mutualize with ngx_http_script for parsing / running the mix
of
+ * strings and variables. */
+
+typedef struct {
+ ngx_str_t *value;
+ ngx_conf_t *cf;
+ ngx_array_t parts;
+ ngx_array_t part_types;
+} ngx_conf_ccv_t;
+
+int ngx_conf_ccv_compile(ngx_conf_ccv_t *ccv);
+int ngx_conf_ccv_init(ngx_conf_ccv_t *ccv, ngx_conf_t *cf, ngx_str_t
*value,
+ ngx_uint_t n);
+int ngx_conf_ccv_run(ngx_conf_ccv_t *ccv);
+int ngx_conf_ccv_resolve_expr(ngx_conf_ccv_t *ccv, ngx_str_t *expr);
+void ngx_conf_ccv_destroy(ngx_conf_ccv_t *ccv);
+
+
+int
+ngx_conf_complex_value(ngx_conf_t *cf, ngx_str_t *string)
+{
+ ngx_uint_t i, nv;
+ ngx_conf_ccv_t ccv;
+
+ nv = 0;
+
+ for (i = 0; i < string->len - 1; ++i) {
+ if (string->data[i] == '{' && string->data[i + 1] == '{') {
+ ++nv;
+ }
+ }
+
+ if (nv == 0) {
+ return NGX_OK;
+ }
+
+ if (ngx_conf_ccv_init(&ccv, cf, string, 2 * nv + 1) != NGX_OK) {
+ goto e_ccv;
+ }
+
+ if (ngx_conf_ccv_compile(&ccv) != NGX_OK) {
+ goto e_compile;
+ }
+
+ if (ngx_conf_ccv_run(&ccv) != NGX_OK) {
+ goto e_run;
+ }
+
+ ngx_conf_ccv_destroy(&ccv);
+
+ return NGX_OK;
+
+e_run:
+e_compile:
+ ngx_conf_ccv_destroy(&ccv);
+e_ccv:
+ return NGX_ERROR;
+}
+
+
+int
+ngx_conf_ccv_init(ngx_conf_ccv_t *ccv, ngx_conf_t *cf, ngx_str_t
*value,
+ ngx_uint_t n)
+{
+ ccv->value = value;
+ ccv->cf = cf;
+
+ if (ngx_array_init(&ccv->parts, cf->pool, n, sizeof(ngx_str_t)) !=
NGX_OK) {
+ goto e_alloc_parts;
+ }
+ if (ngx_array_init(&ccv->part_types, cf->pool, n,
sizeof(ngx_uint_t))
+ != NGX_OK)
+ {
+ goto e_alloc_part_types;
+ }
+
+ return NGX_OK;
+
+ ngx_array_destroy(&ccv->part_types);
+e_alloc_part_types:
+ ngx_array_destroy(&ccv->parts);
+e_alloc_parts:
+ return NGX_ERROR;
+}
+
+
+void
+ngx_conf_ccv_destroy(ngx_conf_ccv_t *ccv)
+{
+ ngx_array_destroy(&ccv->parts);
+ ngx_array_destroy(&ccv->part_types);
+}
+
+
+int
+ngx_conf_ccv_compile(ngx_conf_ccv_t *ccv)
+{
+ ngx_uint_t i, current_part_start, current_part_end;
+ ngx_uint_t current_part_type;
+
+ ccv->parts.nelts = 0;
+ ccv->part_types.nelts = 0;
+ current_part_type = NGX_CONF_TYPE_TEXT;
+
+ for (current_part_start = 0; current_part_start < ccv->value->len;
+ /* void */ )
+ {
+ switch (current_part_type) {
+
+ case NGX_CONF_TYPE_TEXT:
+
+ for (i = current_part_start;
+ i < ccv->value->len
+ && (ccv->value->data[i] != '{' || ccv->value->data[i + 1] !=
'{');
+ /* void */ )
+ {
+ ++i;
+ }
+
+ if (i > current_part_start) {
+ ((ngx_str_t *) ccv->parts.elts)[ccv->parts.nelts].data =
+ &ccv->value->data[current_part_start];
+ ((ngx_str_t *) ccv->parts.elts)[ccv->parts.nelts].len =
+ i - current_part_start;
+ ++ccv->parts.nelts;
+ ((ngx_uint_t *) ccv->part_types.elts)[ccv->part_types.nelts] =
+ current_part_type;
+ ++ccv->part_types.nelts;
+ }
+ if (i < ccv->value->len) {
+ current_part_type = NGX_CONF_TYPE_EXPR;
+ }
+ current_part_start = i;
+
+ break;
+
+ case NGX_CONF_TYPE_EXPR:
+
+ for (i = current_part_start + NGX_CONF_SCRIPT_DELIM_LEN;
+ i < ccv->value->len - 1; ++i)
+ {
+ if (ccv->value->data[i] == '}') {
+ if (ccv->value->data[i + 1] == '}') {
+ break;
+ } else {
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "forbidden \"}\" in \"%V\" at character %d",
+ ccv->value, current_part_start + 1);
+ goto e_script_parse;
+ }
+ } else if (ccv->value->data[i] == '{') {
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "forbidden character \"%c\" in \"%V\""
+ " at character %d",
+ ccv->value, current_part_start + 1);
+ goto e_script_parse;
+ }
+ }
+ if (i >= ccv->value->len - 1) {
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "unbalanced {{ in \"%V\" at character %d",
+ ccv->value, current_part_start + 1);
+ goto e_script_parse;
+ }
+
+ current_part_start += NGX_CONF_SCRIPT_DELIM_LEN;
+ while (current_part_start < ccv->value->len
+ && ccv->value->data[current_part_start] == ' ')
+ {
+ ++current_part_start;
+ }
+ for (current_part_end = i;
+ current_part_end > current_part_start
+ && ccv->value->data[current_part_end - 1] == ' ';
+ --current_part_end)
+ {
+ /* void */
+ }
+
+ if (current_part_end <= current_part_start) {
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "invalid variable name in \"%V\" at character %d",
+ ccv->value, current_part_start + 1);
+ goto e_script_parse;
+ }
+
+ ((ngx_str_t *) ccv->parts.elts)[ccv->parts.nelts].data =
+ &ccv->value->data[current_part_start];
+ ((ngx_str_t *) ccv->parts.elts)[ccv->parts.nelts].len =
+ current_part_end - current_part_start;
+ ++ccv->parts.nelts;
+ ((ngx_uint_t *) ccv->part_types.elts)[ccv->part_types.nelts] =
+ current_part_type;
+ ++ccv->part_types.nelts;
+
+ current_part_start = i + NGX_CONF_SCRIPT_DELIM_LEN;
+ current_part_type = NGX_CONF_TYPE_TEXT;
+
+ break;
+ }
+ }
+
+ return NGX_OK;
+
+e_script_parse:
+
+ return NGX_ERROR;
+}
+
+
+int
+ngx_conf_ccv_run(ngx_conf_ccv_t *ccv)
+{
+ ngx_uint_t i;
+ ngx_str_t *val;
+ size_t len;
+ unsigned char *ptr;
+
+ len = 0;
+
+ for (i = 0; i < ccv->part_types.nelts; ++i) {
+ switch (((ngx_uint_t *) ccv->part_types.elts)[i]) {
+
+ case NGX_CONF_TYPE_TEXT:
+ val = &((ngx_str_t *) ccv->parts.elts)[i];
+ len += val->len;
+ break;
+
+ case NGX_CONF_TYPE_EXPR:
+ val = &(((ngx_str_t *) ccv->parts.elts)[i]);
+ if (ngx_conf_ccv_resolve_expr(ccv, val) != NGX_OK) {
+ return NGX_ERROR;
+ }
+ len += val->len;
+ break;
+ }
+ }
+
+ ptr = ngx_pnalloc(ccv->cf->pool, len);
+ if (ptr == NULL) {
+ return NGX_ERROR;
+ }
+
+ ccv->value->len = len;
+ ccv->value->data = ptr;
+
+ for (i = 0; i < ccv->part_types.nelts; ++i) {
+ switch (((ngx_uint_t *) ccv->part_types.elts)[i]) {
+
+ case NGX_CONF_TYPE_TEXT:
+ case NGX_CONF_TYPE_EXPR:
+ val = &((ngx_str_t *) ccv->parts.elts)[i];
+ ptr = ngx_copy(ptr, val->data, val->len);
+ break;
+ }
+ }
+
+ return NGX_OK;
+}
+
+
+int
+ngx_conf_ccv_resolve_expr(ngx_conf_ccv_t *ccv, ngx_str_t *expr)
+{
+ if (expr->len == 1 && ngx_strncmp(expr->data, ".", 1) == 0) {
+ expr->len = ccv->cf->conf_file->file.name.len;
+ expr->data = ccv->cf->conf_file->file.name.data;
+ for (expr->len = ccv->cf->conf_file->file.name.len;
+ expr->data[--expr->len] != '/';
+ /* void */ )
+ { /* void */ }
+ return NGX_OK;
+ } else {
+ /* TODO: find the value of the last "define" of this context for
this
+ * variable name. */
+ ngx_conf_log_error(NGX_LOG_EMERG, ccv->cf, 0,
+ "not implemented: cannot resolve {{ %V }}", expr);
+ return NGX_ERROR;
+ }
+}
diff -r 89adf49fe76a -r 8bb356ca5a12 src/core/ngx_conf_def.h
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/core/ngx_conf_def.h Mon Oct 28 07:24:17 2019 +0100
@@ -0,0 +1,18 @@
+
+/*
+ * Copyright (C) Guillaume Outters
+ * Copyright (C) Nginx, Inc.
+ */
+
+
+#ifndef _NGX_CONF_DEF_H_INCLUDED_
+#define _NGX_CONF_DEF_H_INCLUDED_
+
+
+#include <ngx_core.h>
+
+
+int ngx_conf_complex_value(ngx_conf_t *cf, ngx_str_t *string);
+
+
+#endif /* _NGX_CONF_DEF_H_INCLUDED_ */
# HG changeset patch
# User Guillaume Outters <guillaume-nginx at outters.eu>
# Date 1572243926 -3600
# Mon Oct 28 07:25:26 2019 +0100
# Node ID cac499a2b296d34a4f7a7a861b860e8726fd0db7
# Parent 8bb356ca5a127afa4c21b57de1df950c6e059595
include: allow {{.}} to reference current file's directory
diff -r 8bb356ca5a12 -r cac499a2b296 src/core/ngx_conf_file.c
--- a/src/core/ngx_conf_file.c Mon Oct 28 07:24:17 2019 +0100
+++ b/src/core/ngx_conf_file.c Mon Oct 28 07:25:26 2019 +0100
@@ -7,6 +7,7 @@
#include <ngx_config.h>
#include <ngx_core.h>
+#include <ngx_conf_def.h>
#define NGX_CONF_BUFFER 4096
@@ -830,6 +831,9 @@
ngx_log_debug1(NGX_LOG_DEBUG_CORE, cf->log, 0, "include %s",
file.data);
+ if (ngx_conf_complex_value(cf, &file) != NGX_OK) {
+ return NGX_CONF_ERROR;
+ }
if (ngx_conf_full_name(cf->cycle, &file, 1) != NGX_OK) {
return NGX_CONF_ERROR;
}
# HG changeset patch
# User Guillaume Outters <guillaume-nginx at outters.eu>
# Date 1572244106 -3600
# Mon Oct 28 07:28:26 2019 +0100
# Node ID ca9d5059523d8630c63867db885cd231ca93173a
# Parent cac499a2b296d34a4f7a7a861b860e8726fd0db7
Core: allow opt-in {{ . }} syntax in
ngx_http_compile_complex_value()-parsed configuration
diff -r cac499a2b296 -r ca9d5059523d src/http/ngx_http_script.c
--- a/src/http/ngx_http_script.c Mon Oct 28 07:25:26 2019 +0100
+++ b/src/http/ngx_http_script.c Mon Oct 28 07:28:26 2019 +0100
@@ -8,6 +8,7 @@
#include <ngx_config.h>
#include <ngx_core.h>
#include <ngx_http.h>
+#include <ngx_conf_def.h>
static ngx_int_t ngx_http_script_init_arrays(ngx_http_script_compile_t
*sc);
@@ -145,6 +146,14 @@
v = ccv->value;
+ if (ccv->compile_defs) {
+ /* Compile definitions before looking for variables, so that a
+ * definition's dereference can contain a variable */
+ if (ngx_conf_complex_value(ccv->cf, v) != NGX_OK) {
+ return NGX_ERROR;
+ }
+ }
+
nv = 0;
nc = 0;
diff -r cac499a2b296 -r ca9d5059523d src/http/ngx_http_script.h
--- a/src/http/ngx_http_script.h Mon Oct 28 07:25:26 2019 +0100
+++ b/src/http/ngx_http_script.h Mon Oct 28 07:28:26 2019 +0100
@@ -83,6 +83,7 @@
unsigned zero:1;
unsigned conf_prefix:1;
unsigned root_prefix:1;
+ unsigned compile_defs:1;
} ngx_http_compile_complex_value_t;
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bundle.hg
Type: application/x-mercurial-bundle
Size: 3543 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20191029/8b271740/attachment-0001.bin>
More information about the nginx-devel
mailing list