multiple php-fpm pool upstream alternating 503 & 502 errors

[George]

Was wondering if anyone could shed some light on this issue I am
experiencing only with multiple php-fpm pool setups but not with single
php-fpm pool. The issue is when a forum software like Xenforo or Invision
board uses their native forum close option to turn off the forums for guests
but still allow forum admins access, the forum via php issue a HTTP 503
status message. This seems to trip up and causes issues only for multiple
php-fpm pool upstream setups causing alternating 503 and 502 bad gateway
errors. Probably partially to do with the http_503 definition for
fastcgi_next_upstream.

The upstream settings

upstream phpbackend {
zone zone_phpbackend 64k;
ip_hash;
keepalive 5;
  server 127.0.0.1:9000 weight=50;
  server 127.0.0.1:9002 weight=50;
  server 127.0.0.1:9003 weight=50;
  server 127.0.0.1:9004 weight=50;
  server 127.0.0.1:9005 weight=50;
}

and relevant php-fpm changes made were to change

from single php-fpm pool

fastcgi_pass   127.0.0.1:9000;

to multiple php-fpm upstream pools

fastcgi_next_upstream error timeout http_500 http_503;
fastcgi_pass phpbackend;
fastcgi_keep_conn on;

I can replicate the issue with multiple php-fpm pool upstream setup by
creating a 503.php file with contents

<?php http_response_code(503) ?>

and then refreshing the 503.php page and it will alternate between 503 and
502 errors

The access.log's alternating 503 and 502 errors excerpt

IPADDR - - [10/Nov/2016:06:07:07 +0000] "GET /503.php HTTP/1.1" 502 1672 "-"
"Mozilla/5.0 snipped" "-" rt=0.000 ua="phpbackend" us="502" ut="0.000"
ul="0" cs=-
IPADDR - - [10/Nov/2016:06:07:03 +0000] "GET /503.php HTTP/1.1" 503 1665 "-"
"Mozilla/5.0 snipped" "-" rt=0.000 ua="127.0.0.1:9004, 127.0.0.1:9002,
127.0.0.1:9005, 127.0.0.1:9003, 127.0.0.1:9000" us="502, 502, 502, 502, 503"
ut="0.000, 0.000, 0.000, 0.000, 0.000" ul="0, 0, 0, 0, 0" cs=-
IPADDR - - [10/Nov/2016:06:07:05 +0000] "GET /503.php HTTP/1.1" 502 1672 "-"
"Mozilla/5.0 snipped" "-" rt=0.000 ua="phpbackend" us="502" ut="0.000"
ul="0" cs=-
IPADDR - - [10/Nov/2016:06:07:07 +0000] "GET /503.php HTTP/1.1" 502 1672 "-"
"Mozilla/5.0 snipped" "-" rt=0.000 ua="phpbackend" us="502" ut="0.000"
ul="0" cs=-

using log format below

log_format  main_ext '$remote_addr - $remote_user [$time_local] "$request"
'
                         '$status $body_bytes_sent "$http_referer" '
                         '"$http_user_agent" "$http_x_forwarded_for" '
                         'rt=$request_time ua="$upstream_addr" '
                         'us="$upstream_status" ut="$upstream_response_time"
'
                         'ul="$upstream_response_length" '
                         'cs=$upstream_cache_status' ;

Using nginx 1.11.5 with PHP 5.6.27

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,270850,270850#msg-270850