multiple php-fpm pool upstream alternating 503 & 502 errors

[Igor A. Ippolitov]

The behaviour of a single and multiple upstreams is changed due to 
'http_503' option of 'fastcgi_next_upstream' .

When a request comes to nginx, nginx forwards it to an upstream, tries 
to get a response, gets 503. http 503 is set to be a failed response 
(with proxy_next_upstream directive). So nginx marks the server as 
'failed' (because of a 'server' default of max_fails=1) and forward the 
request to the next server in the upstream.
When all of servers in an upstream returns 503 they are all marked as 
failed and nginx returns the last reply it has to a client.
For the next 'fail_timeout' seconds all servers are marked as down.
Next time a request comes within a 'fail_timeout', nginx wont proxy the 
request as all servers are marked as down. No servers alive - no request 
is made - http 502 is returned.
Note the difference: the first case is when nginx makes a request and 
returns a reply whatever it is. The second case is when no request for 
an upstream is made and 502 is returned.
If you remove 'http_503' option from 'proxy_next_upstream' nginx will 
not mark a server in an upstream as failed and you won't get http 502.

I'd say, in case a forum is suspended, you would want to return http 403 
instead of 503 (or a stub page with http 200).

Hope this helps ,
Ippolitov Igor.

On 10.11.2016 10:15, George wrote:
> Was wondering if anyone could shed some light on this issue I am
> experiencing only with multiple php-fpm pool setups but not with single
> php-fpm pool. The issue is when a forum software like Xenforo or Invision
> board uses their native forum close option to turn off the forums for guests
> but still allow forum admins access, the forum via php issue a HTTP 503
> status message. This seems to trip up and causes issues only for multiple
> php-fpm pool upstream setups causing alternating 503 and 502 bad gateway
> errors. Probably partially to do with the http_503 definition for
> fastcgi_next_upstream.
>
> The upstream settings
>
> upstream phpbackend {
> zone zone_phpbackend 64k;
> ip_hash;
> keepalive 5;
>    server 127.0.0.1:9000 weight=50;
>    server 127.0.0.1:9002 weight=50;
>    server 127.0.0.1:9003 weight=50;
>    server 127.0.0.1:9004 weight=50;
>    server 127.0.0.1:9005 weight=50;
> }
>
> and relevant php-fpm changes made were to change
>
> from single php-fpm pool
>
> fastcgi_pass   127.0.0.1:9000;
>
> to multiple php-fpm upstream pools
>
> fastcgi_next_upstream error timeout http_500 http_503;
> fastcgi_pass phpbackend;
> fastcgi_keep_conn on;
>
> I can replicate the issue with multiple php-fpm pool upstream setup by
> creating a 503.php file with contents
>
> <?php http_response_code(503) ?>
>
> and then refreshing the 503.php page and it will alternate between 503 and
> 502 errors
>
> The access.log's alternating 503 and 502 errors excerpt
>
> IPADDR - - [10/Nov/2016:06:07:07 +0000] "GET /503.php HTTP/1.1" 502 1672 "-"
> "Mozilla/5.0 snipped" "-" rt=0.000 ua="phpbackend" us="502" ut="0.000"
> ul="0" cs=-
> IPADDR - - [10/Nov/2016:06:07:03 +0000] "GET /503.php HTTP/1.1" 503 1665 "-"
> "Mozilla/5.0 snipped" "-" rt=0.000 ua="127.0.0.1:9004, 127.0.0.1:9002,
> 127.0.0.1:9005, 127.0.0.1:9003, 127.0.0.1:9000" us="502, 502, 502, 502, 503"
> ut="0.000, 0.000, 0.000, 0.000, 0.000" ul="0, 0, 0, 0, 0" cs=-
> IPADDR - - [10/Nov/2016:06:07:05 +0000] "GET /503.php HTTP/1.1" 502 1672 "-"
> "Mozilla/5.0 snipped" "-" rt=0.000 ua="phpbackend" us="502" ut="0.000"
> ul="0" cs=-
> IPADDR - - [10/Nov/2016:06:07:07 +0000] "GET /503.php HTTP/1.1" 502 1672 "-"
> "Mozilla/5.0 snipped" "-" rt=0.000 ua="phpbackend" us="502" ut="0.000"
> ul="0" cs=-
>
> using log format below
>
> log_format  main_ext '$remote_addr - $remote_user [$time_local] "$request"
> '
>                           '$status $body_bytes_sent "$http_referer" '
>                           '"$http_user_agent" "$http_x_forwarded_for" '
>                           'rt=$request_time ua="$upstream_addr" '
>                           'us="$upstream_status" ut="$upstream_response_time"
> '
>                           'ul="$upstream_response_length" '
>                           'cs=$upstream_cache_status' ;
>
> Using nginx 1.11.5 with PHP 5.6.27
>
> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,270850,270850#msg-270850
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>