dynamic upstream configuration

[KT Walrus]

I'm new to this mailing list, but have used nginx for simple websites for years.  Now, I am planning a more complicated website which will require multiple upstream servers and more dynamic reconfiguration events.

I see that there are several third party health check modules available, but I would like to use only "officially" supported nginx modules.  I would like to configure/reconfigure my upstream servers dynamically using a health check module (or upgraded upstream module).

Rather than editing the nginx conf file and reloading for upstream reconfiguration, I'd to do it through health checks to each upstream server.  Initially, my nginx conf file would define all possible backend servers and mark them as DOWN:

upstream backend {
  ip_hash;
  server backend1.example.com:8000 DOWN;
  server backend1.example.com:8001 DOWN;
  server backend2.example.com:8000 DOWN;
  server backend2.example.com:8001 DOWN;
  . . .
  server backendN.example.com:8000 DOWN;
  server backendN.example.com:8001 DOWN;
}

Then, I would like to have nginx do health check to each downed backend once every 5 minutes (or configurable interval).  If the health check times out, the backend stays down.  If the health check returns "UP", the server is marked up.  If check returns "STARTING", the health check will be made again in a minute (or configurable interval).  If check returns "STOPPING" or fails a configurable number of checks (like implemented now), the server is marked down.  An UPed server should be checked frequently (every couple seconds) for a status change and a DOWNed server should be checked less frequently.

As requested in my first post to this list, I also want the server to have a MAXCONN attribute and "first" load balancing.  I would like the health checks to also be able to dynamically change the MAXCONN setting which would be checked before sending new requests to the backend.  This way, the backend can "throttle" itself, by only allowing a few connections on startup to warm up the server and gradually increase the potential number of requests that it can handle.  The load balancing algorithms would need to be adjusted to honor the dynamic MAXCONN attribute when deciding to send a request to the upstream server.

I only need HTTP health checks supported and the status UP, STARTING, STOPPING, and DOWN could be http status codes instead of text.

I like using health checks in this manner since the backend server has input into when and how many requests it can server.  And, I don't have to have a backend server edit the load balancer's configuration file and reload nginx when I want to take the server offline for maintenance or to deploy a new version of the backend app (new app can start listening on port 8001 for requests while old app can mark the port 8000 server DOWN and finish handling current requests).