Strange upstream behaviour

[abauer]

Hello,

We are using nginx as a loadbalancer in front of docker containers. Most of
the time this works without problems. But sometimes (~0.1% of the requests)
the requests are sent to the server group name instead of one of the members
of the servergroup.

upstream gateway {
     server 127.0.0.1:6000 weight=1000000;
     server 1.2.3.4:6000;
  }

 server {
    listen               <ip>:443;
    ssl                  on;
    server_name <public dns>;

    <ssl config>

    location / {
       proxy_pass http://gateway;
    }
}

Most of the time the requests are logged as expected:

03/Feb/2016:04:00:25 +0100 "/api/v1/login" 200 52 "Jersey/2.7
(HttpUrlConnection 1.8.0_51)" "time=0.192" "<public dns>"
"upstream=127.0.0.1:6000"
or
03/Feb/2016:04:00:25 +0100 "/api/v1/login" 200 52 "Jersey/2.7
(HttpUrlConnection 1.8.0_51)" "time=0.192" "<public dns>"
"upstream=1.2.3.4:6000"

But randomly this happens:

03/Feb/2016:04:00:25 +0100 "/api/v1/login" 502 52 "Jersey/2.7
(HttpUrlConnection 1.8.0_51)" "time=0.192" "<public dns>"
"upstream=gateway"

As you can see, it uses the server group name as the upstream target (which
fails of course since this is not a valid host).

What could be the cause of this behaviour?

All the best,
Armin

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,264368,264368#msg-264368