[PATCH 1 of 9] Upstream: re-resolvable servers
Aleksei Bavshin
a.bavshin at nginx.com
Thu Jul 11 20:32:52 UTC 2024
On 7/11/2024 12:58 PM, Mathew Heard wrote:
> On Fri, 12 Jul 2024 at 05:42, Aleksei Bavshin <a.bavshin at nginx.com> wrote:
>>
>> On 7/11/2024 12:15 PM, Mathew Heard wrote:
>>> Do you happen to know if there remains any gap in obvious capability
>>> provided by a module like jdomain compared to this?
>>
>> An obvious difference is that the jdomain directive performs resolution
>> on demand instead of periodic polling, but that's a design choice rather
>> than a feature gap. Other that that I don't see anything missing.
>>
>
> Thinking about it thats actually a surprising feature (rather than
> minor implementation detail). We don't want the service to ever fail
> to start, or to take any longer than it has to start. For this reason
> we use jdomain (and a local DNS cache). By requesting on the first
> request jdomain allows the server to start without resolution delays
> in all cases.
>
> Imagine the configuration may contain xxxx.yyyy.com which takes a
> prolonged time to resolve (for any reason) we would rather that the
> server is able to start and for requests to that server block fail
> until the resolution of xxxx.yyyy.com becomes healthy.
>
> However if the resolution of xxxx.yyyy.com resolves quickly (healthy)
> no interruption on start occurs. Beyond the time it takes to resolve
> the name, which with the aid of a DNS cache is immediately available
> on server reload.
>
> AFAIK is this a feature gap with re-resolution?
>
> I guess this is not really a feature gap as when using domains in a
> static configuration within nginx the behaviour is already like this.
`server ... resolve;` actually behaves quite similar to what you
describe: name resolution is deferred to runtime and doesn't block or
delay the startup.
The main difference is that the resolution starts as soon as the worker
processes are up instead of waiting for the first request, and we update
the expired entries in the background according to the response TTL.
>
>>>
>>> Perhaps it would be worth checking to ensure nothing obvious is not
>>> implemented?
>>>
>>> The one that I see is the ability to control if a resolution is IPv4,
>>> IPv6 or mixed. Is this something that would be useful for this feature?
>>
>> That is already implemented in the resolver directive. If the http block
>> scope is too large, one of the patches in the series allows configuring
>> resolver per upstream.
>>
>> upstream backend {
>> zone upstream_dynamic 64k;
>> resolver 127.0.0.1 ipv4=off ipv6=on;
>>
>> server example.com resolve;
>> };
>>
>
> Thank you for your eyes. I was unaware of upstream level resolver
> configuration. That does indeed look like it would provide the same
> capability.
>
>
>>>
>>> On Fri, 12 Jul 2024 at 02:40, Aleksei Bavshin <a.bavshin at nginx.com
>>> <mailto:a.bavshin at nginx.com>> wrote:
>>>
>>> On 7/9/2024 9:22 AM, Roman Arutyunyan wrote:
>>> > Hi,
>>> >
>>> > On Mon, Jul 08, 2024 at 06:20:58PM +0400, Roman Arutyunyan wrote:
>>> >> Hi,
>>> >>
>>> >> On Thu, Jun 13, 2024 at 03:28:56PM -0700, Aleksei Bavshin wrote:
>>> >>> # HG changeset patch
>>> >>> # User Ruslan Ermilov <ru at nginx.com <mailto:ru at nginx.com>>
>>> >>> # Date 1392462754 -14400
>>> >>> # Sat Feb 15 15:12:34 2014 +0400
>>> >>> # Node ID 56aeae9355df8a2ee07e21b65b6869747dd9ee45
>>> >>> # Parent 02e9411009b987f408214ab4a8b6b6093f843bcd
>>> >>> Upstream: re-resolvable servers.
>>> >>>
>>> >>> Specifying the upstream server by a hostname together with the
>>> >>> "resolve" parameter will make the hostname to be periodically
>>> >>> resolved, and upstream servers added/removed as necessary.
>>> >>>
>>> >>> This requires a "resolver" at the "http" configuration block.
>>> >>>
>>> >>> The "resolver_timeout" parameter also affects when the failed
>>> >>> DNS requests will be attempted again. Responses with NXDOMAIN
>>> >>> will be attempted again in 10 seconds.
>>> >>>
>>> >>> Upstream has a configuration generation number that is
>>> incremented each
>>> >>> time servers are added/removed to the primary/backup list.
>>> This number
>>> >>> is remembered by the peer.init method, and if peer.get detects
>>> a change
>>> >>> in configuration, it returns NGX_BUSY.
>>> >>>
>>> >>> Each server has a reference counter. It is incremented by
>>> peer.get and
>>> >>> decremented by peer.free. When a server is removed, it is
>>> removed from
>>> >>> the list of servers and is marked as "zombie". The memory
>>> allocated by
>>> >>> a zombie peer is freed only when its reference count becomes zero.
>>> >>>
>>> >>> Re-resolvable servers utilize timers that also hold a reference. A
>>> >>> reference is also held while upstream keepalive caches an idle
>>> >>> connection.
>>> >>>
>>> >>> Co-authored-by: Roman Arutyunyan <arut at nginx.com
>>> <mailto:arut at nginx.com>>
>>> >>> Co-authored-by: Sergey Kandaurov <pluknet at nginx.com
>>> <mailto:pluknet at nginx.com>>
>>> >>> Co-authored-by: Vladimir Homutov <vl at nginx.com
>>> <mailto:vl at nginx.com>>
>>> >
>>> > [..]
>>> >
>>> >>> diff --git a/src/http/ngx_http_upstream_round_robin.h
>>> b/src/http/ngx_http_upstream_round_robin.h
>>> >>> --- a/src/http/ngx_http_upstream_round_robin.h
>>> >>> +++ b/src/http/ngx_http_upstream_round_robin.h
>>> >>> @@ -14,8 +14,23 @@
>>> >>> #include <ngx_http.h>
>>> >>>
>>> >>>
>>> >>> +typedef struct ngx_http_upstream_rr_peers_s
>>> ngx_http_upstream_rr_peers_t;
>>> >>> typedef struct ngx_http_upstream_rr_peer_s
>>> ngx_http_upstream_rr_peer_t;
>>> >>>
>>> >>> +
>>> >>> +#if (NGX_HTTP_UPSTREAM_ZONE)
>>> >>> +
>>> >>> +typedef struct {
>>> >>> + ngx_event_t event; /* must be
>>> first */
>>> >>> + ngx_uint_t worker;
>>> >
>>> > Missed this last time. This field should be removed since all
>>> resolving is in
>>> > worker #0.
>>>
>>> Unfortunately, that would break the ABI compatibility between OSS and
>>> Plus. Replacing the field with yet another NGX_COMPAT_BEGIN isn't any
>>> better than leaving it in the opensource code.
>>>
>>> >
>>> >>> + ngx_str_t name;
>>> >>> + ngx_http_upstream_rr_peers_t *peers;
>>> >>> + ngx_http_upstream_rr_peer_t *peer;
>>> >>> +} ngx_http_upstream_host_t;
>>> >>> +
>>> >>> +#endif
>>> >>> +
>>> >>> +
>>> >>> struct ngx_http_upstream_rr_peer_s {
>>> >>> struct sockaddr *sockaddr;
>>> >>> socklen_t socklen;
>>> >>> @@ -46,7 +61,12 @@ struct ngx_http_upstream_rr_peer_s {
>>> >>> #endif
>>> >>>
>>> >>> #if (NGX_HTTP_UPSTREAM_ZONE)
>>> >>> + unsigned zombie:1;
>>> >>
>>> >> I suggest declaring this as in other similar places:
>>> >>
>>> >> ngx_uint_t zombie; /* unsigned
>>> zombie:1; */
>>> >>
>>> >>> +
>>> >>> ngx_atomic_t lock;
>>> >>> + ngx_uint_t id;
>>> >>
>>> >> This field is not used in open source nginx and should not be
>>> added or assigned.
>>> >>
>>> >>> + ngx_uint_t refs;
>>> >>> + ngx_http_upstream_host_t *host;
>>> >>> #endif
>>> >>>
>>> >>> ngx_http_upstream_rr_peer_t *next;
>>> >
>>> > [..]
>>> >
>>> > --
>>> > Roman Arutyunyan
>>> > _______________________________________________
>>> > nginx-devel mailing list
>>> > nginx-devel at nginx.org <mailto:nginx-devel at nginx.org>
>>> > https://mailman.nginx.org/mailman/listinfo/nginx-devel
>>> <https://mailman.nginx.org/mailman/listinfo/nginx-devel>
>>> _______________________________________________
>>> nginx-devel mailing list
>>> nginx-devel at nginx.org <mailto:nginx-devel at nginx.org>
>>> https://mailman.nginx.org/mailman/listinfo/nginx-devel
>>> <https://mailman.nginx.org/mailman/listinfo/nginx-devel>
>>>
>>>
>>> _______________________________________________
>>> nginx-devel mailing list
>>> nginx-devel at nginx.org
>>> https://mailman.nginx.org/mailman/listinfo/nginx-devel
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel at nginx.org
>> https://mailman.nginx.org/mailman/listinfo/nginx-devel
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> https://mailman.nginx.org/mailman/listinfo/nginx-devel
More information about the nginx-devel
mailing list