Best way to handle dynamic upstream backend?

[Maxim Dounin]

Hello!

On Fri, Sep 21, 2012 at 04:08:52PM +0200, Isaac Hailperin wrote:

> Hi,
> 
> I use nginx as a reverse proxy in front of about 20 apaches. Now in
> my backend, currently defined as
> 
> upstream backend_all {
> 	server 1.1.1.1;
> 	server 1.1.1.2;
> ...
> 	server 1.1.1.20;
> }
> 
> i would like to include dynamic information about the health of the
> backend nodes.
> If a backend server is down, it should be marked as down
> automatically. Or if a server is specifically busy, it should get
> less used as a backend.

Normal nginx aproach is to mark backends down atomatically based 
on max_fails/fail_timeout, see here for detail:

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server

It is proven to work well for handling server outages and gives 
system administrators enough time to either revive dead 
server or remove it from configuration.

To atomatically accomodate for unequal server load/speed (due to 
external factors) I would recommend using least_conn balancer, see 
here for details:

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#least_conn

It should also help to better detect outages when requests start 
taking a long time by routing less requests to a faulty server 
before nginx will actually able to see timeouts.

> I could implement a scrip which checks the backend and then
> generates the upstream definition accordingly. I would then have to
> reload nginx. (Would that generate timeouts for my users? - reload
> takes a few (aprox 10) seconds as there are a lot of websites.).

Configuration reload is seamless - old worker processes continue 
to serve clients while master process reads new configuration and 
spawns new worker processes.

It consumes resources though, as it spawns additional set of 
worker processes, and it takes some time for old workers to 
shutdown as they have to serve requests which are already in 
progress.  Doing configuration reloads at high frequency might not 
be a good idea, especially if there are long running requests. 

Summing up the above - it's ok to do reloads as long as they are 
initiated by hand and/or relatively rare, but you may need some 
safeguards for automatic scripts to make sure they aren't run too 
frequently.

> This seems not like the best solution to me, as I feel that nginx
> might have build in solution for this (?).

See above.

> Also I have not only one nginx server, but four, so think I need to
> take care that these four distribute evenly to the backend. For this
> I was thinking of ip_hash.

As long as each frontend distribute requests evenly to all backend 
servers - resulting backend load should be equal.  There is no 
need for ip_hash.

Maxim Dounin