[PATCH 1 of 9] Upstream: re-resolvable servers
Mathew Heard
mat999 at gmail.com
Thu Jul 11 21:03:44 UTC 2024
Thank you Aleksi, I think you are right. Thats the only one of our use
cases using jdomain that I was concerned wouldn't be met by this
feature.
On Fri, 12 Jul 2024 at 06:32, Aleksei Bavshin <a.bavshin at nginx.com> wrote:
>
> On 7/11/2024 12:58 PM, Mathew Heard wrote:
> > On Fri, 12 Jul 2024 at 05:42, Aleksei Bavshin <a.bavshin at nginx.com> wrote:
> >>
> >> On 7/11/2024 12:15 PM, Mathew Heard wrote:
> >>> Do you happen to know if there remains any gap in obvious capability
> >>> provided by a module like jdomain compared to this?
> >>
> >> An obvious difference is that the jdomain directive performs resolution
> >> on demand instead of periodic polling, but that's a design choice rather
> >> than a feature gap. Other that that I don't see anything missing.
> >>
> >
> > Thinking about it thats actually a surprising feature (rather than
> > minor implementation detail). We don't want the service to ever fail
> > to start, or to take any longer than it has to start. For this reason
> > we use jdomain (and a local DNS cache). By requesting on the first
> > request jdomain allows the server to start without resolution delays
> > in all cases.
> >
> > Imagine the configuration may contain xxxx.yyyy.com which takes a
> > prolonged time to resolve (for any reason) we would rather that the
> > server is able to start and for requests to that server block fail
> > until the resolution of xxxx.yyyy.com becomes healthy.
> >
> > However if the resolution of xxxx.yyyy.com resolves quickly (healthy)
> > no interruption on start occurs. Beyond the time it takes to resolve
> > the name, which with the aid of a DNS cache is immediately available
> > on server reload.
> >
> > AFAIK is this a feature gap with re-resolution?
> >
> > I guess this is not really a feature gap as when using domains in a
> > static configuration within nginx the behaviour is already like this.
>
> `server ... resolve;` actually behaves quite similar to what you
> describe: name resolution is deferred to runtime and doesn't block or
> delay the startup.
> The main difference is that the resolution starts as soon as the worker
> processes are up instead of waiting for the first request, and we update
> the expired entries in the background according to the response TTL.
>
> >
> >>>
> >>> Perhaps it would be worth checking to ensure nothing obvious is not
> >>> implemented?
> >>>
> >>> The one that I see is the ability to control if a resolution is IPv4,
> >>> IPv6 or mixed. Is this something that would be useful for this feature?
> >>
> >> That is already implemented in the resolver directive. If the http block
> >> scope is too large, one of the patches in the series allows configuring
> >> resolver per upstream.
> >>
> >> upstream backend {
> >> zone upstream_dynamic 64k;
> >> resolver 127.0.0.1 ipv4=off ipv6=on;
> >>
> >> server example.com resolve;
> >> };
> >>
> >
> > Thank you for your eyes. I was unaware of upstream level resolver
> > configuration. That does indeed look like it would provide the same
> > capability.
> >
> >
> >>>
> >>> On Fri, 12 Jul 2024 at 02:40, Aleksei Bavshin <a.bavshin at nginx.com
> >>> <mailto:a.bavshin at nginx.com>> wrote:
> >>>
> >>> On 7/9/2024 9:22 AM, Roman Arutyunyan wrote:
> >>> > Hi,
> >>> >
> >>> > On Mon, Jul 08, 2024 at 06:20:58PM +0400, Roman Arutyunyan wrote:
> >>> >> Hi,
> >>> >>
> >>> >> On Thu, Jun 13, 2024 at 03:28:56PM -0700, Aleksei Bavshin wrote:
> >>> >>> # HG changeset patch
> >>> >>> # User Ruslan Ermilov <ru at nginx.com <mailto:ru at nginx.com>>
> >>> >>> # Date 1392462754 -14400
> >>> >>> # Sat Feb 15 15:12:34 2014 +0400
> >>> >>> # Node ID 56aeae9355df8a2ee07e21b65b6869747dd9ee45
> >>> >>> # Parent 02e9411009b987f408214ab4a8b6b6093f843bcd
> >>> >>> Upstream: re-resolvable servers.
> >>> >>>
> >>> >>> Specifying the upstream server by a hostname together with the
> >>> >>> "resolve" parameter will make the hostname to be periodically
> >>> >>> resolved, and upstream servers added/removed as necessary.
> >>> >>>
> >>> >>> This requires a "resolver" at the "http" configuration block.
> >>> >>>
> >>> >>> The "resolver_timeout" parameter also affects when the failed
> >>> >>> DNS requests will be attempted again. Responses with NXDOMAIN
> >>> >>> will be attempted again in 10 seconds.
> >>> >>>
> >>> >>> Upstream has a configuration generation number that is
> >>> incremented each
> >>> >>> time servers are added/removed to the primary/backup list.
> >>> This number
> >>> >>> is remembered by the peer.init method, and if peer.get detects
> >>> a change
> >>> >>> in configuration, it returns NGX_BUSY.
> >>> >>>
> >>> >>> Each server has a reference counter. It is incremented by
> >>> peer.get and
> >>> >>> decremented by peer.free. When a server is removed, it is
> >>> removed from
> >>> >>> the list of servers and is marked as "zombie". The memory
> >>> allocated by
> >>> >>> a zombie peer is freed only when its reference count becomes zero.
> >>> >>>
> >>> >>> Re-resolvable servers utilize timers that also hold a reference. A
> >>> >>> reference is also held while upstream keepalive caches an idle
> >>> >>> connection.
> >>> >>>
> >>> >>> Co-authored-by: Roman Arutyunyan <arut at nginx.com
> >>> <mailto:arut at nginx.com>>
> >>> >>> Co-authored-by: Sergey Kandaurov <pluknet at nginx.com
> >>> <mailto:pluknet at nginx.com>>
> >>> >>> Co-authored-by: Vladimir Homutov <vl at nginx.com
> >>> <mailto:vl at nginx.com>>
> >>> >
> >>> > [..]
> >>> >
> >>> >>> diff --git a/src/http/ngx_http_upstream_round_robin.h
> >>> b/src/http/ngx_http_upstream_round_robin.h
> >>> >>> --- a/src/http/ngx_http_upstream_round_robin.h
> >>> >>> +++ b/src/http/ngx_http_upstream_round_robin.h
> >>> >>> @@ -14,8 +14,23 @@
> >>> >>> #include <ngx_http.h>
> >>> >>>
> >>> >>>
> >>> >>> +typedef struct ngx_http_upstream_rr_peers_s
> >>> ngx_http_upstream_rr_peers_t;
> >>> >>> typedef struct ngx_http_upstream_rr_peer_s
> >>> ngx_http_upstream_rr_peer_t;
> >>> >>>
> >>> >>> +
> >>> >>> +#if (NGX_HTTP_UPSTREAM_ZONE)
> >>> >>> +
> >>> >>> +typedef struct {
> >>> >>> + ngx_event_t event; /* must be
> >>> first */
> >>> >>> + ngx_uint_t worker;
> >>> >
> >>> > Missed this last time. This field should be removed since all
> >>> resolving is in
> >>> > worker #0.
> >>>
> >>> Unfortunately, that would break the ABI compatibility between OSS and
> >>> Plus. Replacing the field with yet another NGX_COMPAT_BEGIN isn't any
> >>> better than leaving it in the opensource code.
> >>>
> >>> >
> >>> >>> + ngx_str_t name;
> >>> >>> + ngx_http_upstream_rr_peers_t *peers;
> >>> >>> + ngx_http_upstream_rr_peer_t *peer;
> >>> >>> +} ngx_http_upstream_host_t;
> >>> >>> +
> >>> >>> +#endif
> >>> >>> +
> >>> >>> +
> >>> >>> struct ngx_http_upstream_rr_peer_s {
> >>> >>> struct sockaddr *sockaddr;
> >>> >>> socklen_t socklen;
> >>> >>> @@ -46,7 +61,12 @@ struct ngx_http_upstream_rr_peer_s {
> >>> >>> #endif
> >>> >>>
> >>> >>> #if (NGX_HTTP_UPSTREAM_ZONE)
> >>> >>> + unsigned zombie:1;
> >>> >>
> >>> >> I suggest declaring this as in other similar places:
> >>> >>
> >>> >> ngx_uint_t zombie; /* unsigned
> >>> zombie:1; */
> >>> >>
> >>> >>> +
> >>> >>> ngx_atomic_t lock;
> >>> >>> + ngx_uint_t id;
> >>> >>
> >>> >> This field is not used in open source nginx and should not be
> >>> added or assigned.
> >>> >>
> >>> >>> + ngx_uint_t refs;
> >>> >>> + ngx_http_upstream_host_t *host;
> >>> >>> #endif
> >>> >>>
> >>> >>> ngx_http_upstream_rr_peer_t *next;
> >>> >
> >>> > [..]
> >>> >
> >>> > --
> >>> > Roman Arutyunyan
> >>> > _______________________________________________
> >>> > nginx-devel mailing list
> >>> > nginx-devel at nginx.org <mailto:nginx-devel at nginx.org>
> >>> > https://mailman.nginx.org/mailman/listinfo/nginx-devel
> >>> <https://mailman.nginx.org/mailman/listinfo/nginx-devel>
> >>> _______________________________________________
> >>> nginx-devel mailing list
> >>> nginx-devel at nginx.org <mailto:nginx-devel at nginx.org>
> >>> https://mailman.nginx.org/mailman/listinfo/nginx-devel
> >>> <https://mailman.nginx.org/mailman/listinfo/nginx-devel>
> >>>
> >>>
> >>> _______________________________________________
> >>> nginx-devel mailing list
> >>> nginx-devel at nginx.org
> >>> https://mailman.nginx.org/mailman/listinfo/nginx-devel
> >> _______________________________________________
> >> nginx-devel mailing list
> >> nginx-devel at nginx.org
> >> https://mailman.nginx.org/mailman/listinfo/nginx-devel
> > _______________________________________________
> > nginx-devel mailing list
> > nginx-devel at nginx.org
> > https://mailman.nginx.org/mailman/listinfo/nginx-devel
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> https://mailman.nginx.org/mailman/listinfo/nginx-devel
More information about the nginx-devel
mailing list