nginx upstream problem

[Maxim Dounin]

Hello!

On Tue, Apr 13, 2010 at 04:28:35PM -0400, izghitu wrote:

[...]

> worker_processes  32;

This looks a bit too many.

[...]

>     include /etc/nginx/conf.d/*.conf;

Do you have only one file there?  Note that upstream{} blocks are 
all defined in http section, and having two with identical names 
are likely to cause confusion.

[...]

> upstream cloud {
>     server apache3 max_fails=1 fail_timeout=5;
>     server apache2 max_fails=1 fail_timeout=5;
>     server apache1 max_fails=1 fail_timeout=5;
> }

[...]

>     proxy_read_timeout 30;
>     proxy_next_upstream error timeout;
>     proxy_pass http://cloud;

[...]

> Problem 1:
> Regardless of the order of the apache servers in the upstream, 
> apache3 most of the time gets the lowest traffic. Sometimes it 
> gets the same, sometimes it gets very few traffic.

Does these servers (apache1, apache2, apache3) actually resolve 
to distinct ip addresses?

> Problem 2:
> If I take down one apache server(halt or /etc/init.d/network 
> stop) then I can see from the error log of nginx that it is 
> still sending traffic to that apache server and I see time out 
> or no route to host errors. When browsing the website I get 
> waiting pages all the time which proves it tries to send to the 
> apache server that is down.

nginx marks server as "down" after "max_fails" failures in 
"fail_timeout" seconds, and re-enables it again after 
"fail_timeout" seconds.

As it takes 60 seconds (default proxy_connect_timeout) to detect 
that server is down, and you switch it off only for 5 seconds - 
it's somewhat expected that nginx will try to use dead server most 
of the time.

Also note that statuses of upstream servers are tracked 
independantly by each worker (so with 32 workers you are likely to 
see dead server used all the time by different workers).

You may tune nginx to behave a bit better by setting bigger 
fail_timeout and/or smaller proxy_connect_timeout.

[...]

Maxim Dounin