Best way to handle dynamic upstream backend?

[Isaac Hailperin]

Hi,

I use nginx as a reverse proxy in front of about 20 apaches. Now in my 
backend, currently defined as

upstream backend_all {
	server 1.1.1.1;
	server 1.1.1.2;
...
	server 1.1.1.20;
}

i would like to include dynamic information about the health of the 
backend nodes.
If a backend server is down, it should be marked as down automatically. 
Or if a server is specifically busy, it should get less used as a backend.

I could implement a scrip which checks the backend and then generates 
the upstream definition accordingly. I would then have to reload nginx. 
(Would that generate timeouts for my users? - reload takes a few (aprox 
10) seconds as there are a lot of websites.). This seems not like the 
best solution to me, as I feel that nginx might have build in solution 
for this (?).

Also I have not only one nginx server, but four, so think I need to take 
care that these four distribute evenly to the backend. For this I was 
thinking of ip_hash.

Any suggestions on how to handle this?

Isaac