No live upstreams with a single upstream
mdounin at mdounin.ru
Tue Sep 18 15:46:07 UTC 2018
On Tue, Sep 18, 2018 at 06:02:46AM -0400, domleb wrote:
> While running a load test that injects 10k TPS across 3 Nginx instances, we
> are seeing spikes of errors where Nginx returns HTTP 502 and logs the
> message 'no live upstreams while connecting to upstream'. There are no
> other errors logged e.g. connection errors.
> Also, we have a single upstream virtual IP (we use iptables to balance load
> across the backend) and according to the docs the upstream should never be
> marked as down in this case:
> 'If there is only a single server in a group, max_fails, fail_timeout and
> slow_start parameters are ignored, and such a server will never be
> considered unavailable'
> Testing locally with our config confirms this and I cannot reproduce the 'no
> live upstreams while connecting to upstream' message when simulating
> connection and read errors with a single upstream.
> To debug I tried enabling debug logs but under load that degraded
> performance too much. I also traced the worker process with strace and
> didn't find any socket or other other errors during the 502 spike.
> I was able to create this issue on Nginx 1.12.2 and 1.15.3.
> So given that we don't see any source error and we have a single upstream,
> I'm interested to know what other scenarios could result in a 502 with the
> log message 'no live upstreams while connecting to upstream'?
Could you please show the upstream configuration you are using?
With a single server in the upstream block, "no live upstreams"
error may happen if:
- the server is marked "down" in the configuration, or
- the server reached the max_conns limit.
Also note that "a single server" does not apply to cases when
there is a single hostname which resolves to multiple IP address
(this defines multiple servers at once).
More information about the nginx