Nginx Reverse Proxy - Stale proxy_pass URL

[Sergey A. Osokin]

Hi Ben,

hope you're doing well.

On Mon, Mar 07, 2022 at 07:53:04PM +0000, Ben Mills wrote:
> Greetings nginx,
> 
> nginx version: nginx/1.18.0 running an AWS EC2 instance with an Amazon Linux 2 AMI.

I'd highly recommend to update the nginx OSS to the recent stable version, 1.20.2 [1].

[...]

> The nginx service has cached 1.2.3.4 at runtime and the fact that the https://upstream
> now resolves to different IPs has broken the proxy. Restarting the nginx service
> fixes the issue since it then resolves https://upstream to the new ELB IPs.
> 
> Question-1
> 
> Is there a directive to add to our nginx.conf server block that will force nginx to
> re-resolve its proxy_pass URL upon error? If not upon error, then perhaps at some
> configurable time interval?

It's a bit tricky, but it's possible to use the set directive [2] to set up a
variable with a backend name for dynamic resolvoing, i.e.:

  set $backend http://dyn-backned.example.com:8080;
  proxy_pass $backend;

Please note that the resolver directive [3] with a proper nameserver's IP address and
a valid parameter is a requirement and needs to be defined as well.
 
> I have my eye on proxy_cache_use_stale, but not sure if this is suited to our use
> case.
> 
> Question-2
> 
> The devices using this setup are specialized and testing is not easy. Is there a
> command line option that will allow a user with SSH access to the EC2 instance
> where nginx is running to verify what nginx currently has in its cache for
> https://upstream? (i.e. rather than having to wait for a real device to error).
> The access.log does not display this information, only the error.log does.

Sure, it's possible to use find(1) command line utility to walk through a file
hierarchy with nginx cache path, please follow the corresponding man page to get
more details.

References:
[1] http://nginx.org/en/linux_packages.html#Amazon-Linux
[2] https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#set
[3] https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver

-- 
Sergey A. Osokin