[BUG?]fail_timeout/max_fails: code doesn't do what doc says

[Dmitry Popov]

On Tue, 21 May 2013 17:23:08 +0400
Maxim Dounin <mdounin at mdounin.ru> wrote:

> 
> This is expected behaviour.  Documentation is a bit simplified 
> here, and fail_timeout is used like session time limit - the 
> peer->fails counter is reset once there are no failures within 
> fail_timeout.
> 
> While this might be non-ideal for some use cases, it's certainly 
> not a bug.
>

Well, it really hurts. Upstreams which fail in ~1% of requests is not a rare
case, and we can't use max_fails+fail_timeout for them (because round-robin is
thrashed for them and ip_hash is completely useless). Moreover, it is very hard
to debug because of wiki. 

> Such algorithm forget everything about previous failures once per 
> fail_timeout, and won't detect bursts of failures split across 
> two fail_timeout intervals.
> 
> Consider:
> 
> - max_fails = 3, fail_timeout = 10s
> - failure at 0s
> - failure at 9s
> - at 10s peer->fails counter is reset
> - failure at 11s
> - failure at 12s
> 
> While 3 failures are only 3 seconds away from each other, this 
> is not detected due to granularity introduced by the algorithm.

Yes, I know this case, sorry, forgot to mention. However, I think it will
extend detection period to 2-3 fail_timeouts in real life (in theory up to 
max_fails fail_timeouts, yes, but it's almost improbable). If we want correct
implementation we need per-second array (with fail_timeout elements), that's an
overkill in my opinion. By the way, leaky bucket approach (like limit_req but
with fails per second) might work well here, what do you think?

-- 
Dmitry Popov
Highloadlab