Best way to handle dynamic upstream backend?

[Isaac Hailperin]

Thank you Maxim for your advice!

Isaac

On 09/22/2012 04:15 PM, Maxim Dounin wrote:
> Hello!
>
> On Fri, Sep 21, 2012 at 04:08:52PM +0200, Isaac Hailperin wrote:
>
>> Hi,
>>
>> I use nginx as a reverse proxy in front of about 20 apaches. Now in
>> my backend, currently defined as
>>
>> upstream backend_all {
>> 	server 1.1.1.1;
>> 	server 1.1.1.2;
>> ...
>> 	server 1.1.1.20;
>> }
>>
>> i would like to include dynamic information about the health of the
>> backend nodes.
>> If a backend server is down, it should be marked as down
>> automatically. Or if a server is specifically busy, it should get
>> less used as a backend.
>
> Normal nginx aproach is to mark backends down atomatically based
> on max_fails/fail_timeout, see here for detail:
>
> http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server
>
> It is proven to work well for handling server outages and gives
> system administrators enough time to either revive dead
> server or remove it from configuration.
>
> To atomatically accomodate for unequal server load/speed (due to
> external factors) I would recommend using least_conn balancer, see
> here for details:
>
> http://nginx.org/en/docs/http/ngx_http_upstream_module.html#least_conn
>
> It should also help to better detect outages when requests start
> taking a long time by routing less requests to a faulty server
> before nginx will actually able to see timeouts.
>
>> I could implement a scrip which checks the backend and then
>> generates the upstream definition accordingly. I would then have to
>> reload nginx. (Would that generate timeouts for my users? - reload
>> takes a few (aprox 10) seconds as there are a lot of websites.).
>
> Configuration reload is seamless - old worker processes continue
> to serve clients while master process reads new configuration and
> spawns new worker processes.
>
> It consumes resources though, as it spawns additional set of
> worker processes, and it takes some time for old workers to
> shutdown as they have to serve requests which are already in
> progress.  Doing configuration reloads at high frequency might not
> be a good idea, especially if there are long running requests.
>
> Summing up the above - it's ok to do reloads as long as they are
> initiated by hand and/or relatively rare, but you may need some
> safeguards for automatic scripts to make sure they aren't run too
> frequently.
>
>> This seems not like the best solution to me, as I feel that nginx
>> might have build in solution for this (?).
>
> See above.
>
>> Also I have not only one nginx server, but four, so think I need to
>> take care that these four distribute evenly to the backend. For this
>> I was thinking of ip_hash.
>
> As long as each frontend distribute requests evenly to all backend
> servers - resulting backend load should be equal.  There is no
> need for ip_hash.
>
> Maxim Dounin
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>