Resolver not re-resolving new ip address of an AW ELB

[RKGood]

Hello,

We are trying to find a solution from past couple of days but nothing seems
to work so far. Our server in prod going down everyday when the AWS ELB
changes ip address. Our's little complex proxy (don't ask me why we have to
do this :), there is a strong reason for it), our clients will send requests
apache (legacy), apache proxies to nginx (new) and nginx decides whether
proxy back to apache or serve the request with new micro services. Resolver
re-resolves the new micro services (internal alb) ip address but fail to
re-resolve the legacy apache (has a ELB with route 53 entry in front). We
are using https endpoint to proxy apache request. The request flows thru ELB
(legacy) -> Apache -> ELB (new) -> nginx -> ELB (legacy) -> apache 

Can you please provide feedback on what are we doing wrong, this is only
happening in production. Our load is normal few fundred requests per second.
We aren't able to simulate it in test environment. 

Here is the configuration:


user nginx;
worker_processes  auto;
worker_rlimit_nofile 5120;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
  worker_connections  2048;
}

http {
  include    /etc/nginx/mime.types;
  default_type text/plain;
  
  resolver 10....2 172.16.0.23 valid=30s ipv6=off;
  resolver_timeout 5s;

  log_format main '$proxy_protocol_addr - [$status] - [$request_time] -
[$upstream_response_time] - $server_name $upstream_addr $request';
  
  access_log /var/log/nginx/error.log main;

  rewrite_log on;
  
  client_body_timeout             60s;
  client_header_timeout           30s;
  send_timeout                    60s;
  sendfile     off;

  tcp_nodelay on;
  tcp_nopush   on;
  reset_timedout_connection on;

  server_names_hash_bucket_size 128;
  client_body_buffer_size 64k;
  client_max_body_size 10m;

  server {

	listen        443 ssl proxy_protocol default_server;
    server_name  mydomain.com;
    ssl_certificate      mydomain.crt;
    ssl_certificate_key  mydomain.key;
    
    set $alb_upstream aws-internal-alb;
    set $apache_upstream legacy.domain.com;

    proxy_buffers 8 24k;
    proxy_buffer_size 2k;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header X-Real-IP       $proxy_protocol_addr;
    proxy_next_upstream off;


    location /services/(migrated1|migrated2)/ {

        proxy_set_header Host $host;
        proxy_connect_timeout 302;
        proxy_read_timeout 302;

        rewrite /services/(.*) /$1?$args break;
        proxy_pass http://alb_upstream;
    }
	
    location /services/ {
        proxy_set_header x-nginx-rejected true;
        proxy_set_header Host legacy.domain.com;
        proxy_connect_timeout 302;
        proxy_read_timeout 302;

        rewrite /services/(.*) /$1?$args break;
        proxy_pass https://$apache_upstream;
    }
    
  }  
}

Thanks in advance.
RK

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277101,277101#msg-277101