Checking multiple caches before forwarding request to upstream

[sachin.shetty@gmail.com]

Hi,

We want to define multiple caches based on certain request headers (time
stamp) so that we can put files modified in last 10 days on SSDs, last 30
days on HDDs and so on. I understand that we could use map feature to pick a
cache dynamically which is good and works for us. 

But when serving a file, we want to check in all the caches because file
modified in last 11 days could still be on SSDs, so we don't wan't to
unnecessarily pull it out from backend and put it on HDDs.

so net, net we want to check multiple caches before pulling from the
backend. Is there a way to do that? We can only define one proxy_cache in a
location block, so I figured if we some how use try files or error page
attribute to failover multiple blocks and check one cache in each block, we
could check multiple caches. 

following is my simplified config:

server {
        listen       7800 ;
        server_name  localhost;
        resolver 8.8.8.8 ipv6=off;

        location / {

            set $served_by "cache";
            set $cache_key $request_uri;
            proxy_cache cache_recent;
            proxy_cache_key $cache_key;

            error_page 418 = @go-to-cloud-storage; return 418;
            #try_files /does-not-exist @go-to-cloud-storage;

        }

        location @go-to-cloud-storage  {
            set $served_by "cloud";

            proxy_cache cache_recent;
            proxy_cache_key $cache_key;

            proxy_set_header Host $host;
            proxy_pass https://$http_host$request_uri;
        }

    }

But in the above config,   @go-to-cloud-storage storage is always executed
even when object is in cache for the / block. 

Thanks
Sachin

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273446,273446#msg-273446