Periodic external HTTP exchange from NGINX for dynamic control of upstream server list

[rnmx18]

Hi,

I have a requirement to de-select or exclude one or more servers from an
upstream group, if that server is either just newborn or exhausted
(overloaded or showing reduced performance).

So, consider there are servers A, B and C in the upstream group. An external
program periodically checks A, B and C and keeps track of different
parameters (uptime, average response time etc), and assigns a "good" or
"bad" status flag to each of them. 

I would like to implement a mechanism in NGINX, in which it can periodically
(say every 5 minutes) communicate with this external program and collect the
status flag for A,B and C. This would be an HTTP communication with a
response in a parse-able text format. Assume if B is reported as "bad", I
would like to exclude B temporarily, till I get a "good" value back for it.

I am aware of the native upstream healthcheck mechanism in NGINX.

However, the requirement here is to bring some additional parameters into
consideration for the selection of upstream.

I would like to know whether this is feasible to realize in NGINX. Could
someone please explain some design insights for this problem?

Going a bit deeper, I could identify the following potential requirements or
questions?

a) Should I depend on both (a) the passive upstream check native to NGINX
and (b) the status value to be obtained from the external process? I think
since the native health check is passive in nature (like when NGINX tries to
connect to an upstream when it wants to proxy the request), it is beyond my
programmable-control. So, I needn't worry about that. What I should be
worried is about enabling NGINX to dynamically exclude a particular server,
based on a "bad" feedback from the external process.

b) How to communicate periodically with the external process over HTTP? How
can I get an HTTP client functionality in NGINX to send an external request
and process the response? Can I build this functionality as an NGINX
module?

c) Last but not the least, I would need some mechanism by which I should be
able to map the server(s) reported as "bad" to the configured upstream
servers, and then temporarily exclude them (mark down), and later include
them when the status changes to "good".

Thanks
Rajesh

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,276912,276912#msg-276912