Why Nginx send traffic to the next upstream on 504 error

[stmx38]

Hello,

We have an Nginx where we configured
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream
in Nginx main config file at http level.

We don't want to send traffic to the next upstream on 504:
----
proxy_next_upstream    error timeout http_502 http_503 non_idempotent;
----

At vhost level we don't redefine this directive. But periodically we see the
following in Nginx logs:
----
141.101.69.85 - 192.168.1.10 - 59.489 - [08/Aug/2020:11:10:41.098 +0000] -
POST - /api/ - HTTP/1.1 - 499 -  - 0 - 2054 - "Java/1.8.0_242" - api-worker
- "192.168.1.11:8080, 192.168.1.12:8080" - "504, -" - "0.000, 0.000" -
"31.000, 28.489"
----

api-worker - "192.168.1.11:8080, 192.168.1.12:8080" - "504, -"

>From the logs we see that Nginx received 504 status from the first upstream
and then for some reason send traffic to the next one, despite the fact that
it should not do it on 504 http status.

We did a short test using http://httpstat.us/504
----
upstream test-504 {
  server 104.31.86.226:80;
  server 104.31.87.226:80;
}

server {
  listen      443 ssl http2;
  server_name domain.tld

  # Test 504
  location /test-504 {

    # proxy_next_upstream error timeout http_502 http_503 http_504
non_idempotent;

    proxy_pass http://test-504/504;

    proxy_set_header Host httpstat.us;
  }
}
----

If we comment 'proxy_next_upstream' it uses one defined in the main config
at http level and don't send traffic to the next upstream. If we uncomment
it, we see that Nginx send traffic to the next upstream. All works as
expected and described in the documentation.

But the question remained for our production: Why Nginx send traffic to the
next upstream on 504 error?
1. It is some misconfiguration on our side, maybe timeouts on any other
directives should be enabled/disabled?
2. It is some kind if misunderstanding how 'proxy_next_upstream' directive
works?
3. It is some kind of bug?

Thank you!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289020,289020#msg-289020