Checking multiple caches before forwarding request to upstream

[Maxim Dounin]

Hello!

On Thu, Apr 06, 2017 at 06:45:23AM -0400, sachin.shetty at gmail.com wrote:

> Hi,
> 
> We want to define multiple caches based on certain request headers (time
> stamp) so that we can put files modified in last 10 days on SSDs, last 30
> days on HDDs and so on. I understand that we could use map feature to pick a
> cache dynamically which is good and works for us. 
> 
> But when serving a file, we want to check in all the caches because file
> modified in last 11 days could still be on SSDs, so we don't wan't to
> unnecessarily pull it out from backend and put it on HDDs.
> 
> so net, net we want to check multiple caches before pulling from the
> backend. Is there a way to do that? We can only define one proxy_cache in a
> location block, so I figured if we some how use try files or error page
> attribute to failover multiple blocks and check one cache in each block, we
> could check multiple caches. 
> 
> following is my simplified config:
> 
> server {
>         listen       7800 ;
>         server_name  localhost;
>         resolver 8.8.8.8 ipv6=off;
> 
>         location / {
> 
>             set $served_by "cache";
>             set $cache_key $request_uri;
>             proxy_cache cache_recent;
>             proxy_cache_key $cache_key;
> 
>             error_page 418 = @go-to-cloud-storage; return 418;
>             #try_files /does-not-exist @go-to-cloud-storage;
> 
>         }
> 
>         location @go-to-cloud-storage  {
>             set $served_by "cloud";
> 
>             proxy_cache cache_recent;
>             proxy_cache_key $cache_key;
> 
>             proxy_set_header Host $host;
>             proxy_pass https://$http_host$request_uri;
>         }
> 
>     }
> 
> But in the above config,   @go-to-cloud-storage storage is always executed
> even when object is in cache for the / block. 

That's because you unconditionally return 418 before any other 
processing.  Rewrite directives, including "return", are executed 
when nginx searches for a configuration to use, see details here:

http://nginx.org/en/docs/http/ngx_http_rewrite_module.html

Moreover, proxy_cache only works when you proxy somewhere, and 
there is no proxy_pass in your "location /".

To check multiple caches consider actual proxying instead, this 
will allow you to naturally use multiple caching layers.

-- 
Maxim Dounin
http://nginx.org/