Multiple upstream_cache_status headers in response in a dual-cache configuration

[rnmx18]

Hi,

To realize a distributed caching layer based of disk-speed and storage, I
have prepared the following configuration with an SSD-cache and HDD-cache.

http {

    add_header X-UpStream-Server-Cache-Status $upstream_cache_status;

    # proxy caching configurations
    proxy_cache_path /tmp/myssd1 keys_zone=myssd1:1m levels=1:2
inactive=60m;

    proxy_cache_path /tmp/myhdd1 keys_zone=myhdd1:10m levels=1:2
inactive=60m;

    proxy_cache_key "$request_uri";

    upstream mylocalhdd {
        server 127.0.0.1:82;
    }

    upstream myorigin {
        server 192.168.1.35:80;
    }

    # Server-block1 - will cache in SSD if we get a minimum of 5 requests.
    server {
        listen       80;
        server_name  example.com www.example.com;

        # Look up in ssd cache. If not found, goto hdd cache.
        location / {
            proxy_pass http://mylocalhdd;
            proxy_cache myssd1;
            proxy_cache_min_uses 5;
        }
    }

    # Server-block2 (acting as local upstream) - will fetch from origin and
cache in HDD
    server {
        listen       82;
        server_name  example.com www.example.com;

        # Look up in hdd cache. If not found, goto origin
        location / {
            proxy_pass http://myorigin;
            proxy_cache myhdd1;
        }
    }

}

The smaller, yet faster SSD-cache will store content only if I get at least
5 requests for the URL. On the other hand, the larger HDD cache will every
request.

The flow is straight-forward:

i) Client's first request is handled by server at port 80
ii) It looks in SSD cache. Upon a cache-miss, it proxies to local upstream
at port 82.
iii) The local server at port 82, looks in HDD cache. upon cache miss, it
fetches from origin, and adds to HDD-cache and sends the response back to
first server.
iv) Server1 does not add content yet to SSD-cache (as min_uses is not
reached), and sends response to client.
v) For the next 3 requests, the server1 would see an SSD-cache-miss, but
server2 produces an HDD-cache-hit.
vi) For the 5th request, the server1, will also add the response in
SSD-cache, as min_uses criteria is met.
vii) For the 6th request onwards, the server1 itself will serve the request
from SSD-cache itself. No request is sent to local upstream.

I have added $upstream_cache_status in the response.

For the first request, I see the header twice in the response: This seem to
correspond to MISS in both front-end SSD-cache and back-end HDD-cache.

< X-UpStream-Server-Cache-Status: MISS
< X-UpStream-Server-Cache-Status: MISS

For the next 4 requests, I see the following: This seems to correspond to
MISS in the from-end SSD-cache and HIT in the back-end HDD-cache.

< X-UpStream-Server-Cache-Status: HIT
< X-UpStream-Server-Cache-Status: MISS

For the 6th request, I see the following:

< X-UpStream-Server-Cache-Status: HIT
< X-UpStream-Server-Cache-Status: HIT

Why am I getting the header twice for the 6th request. In this case, the
request is HIT by the SSD cache itself, and there is no request sent to
local upstream also.

So, shouldn't I be getting only one instance of the header in the response?

Thanks
Rajesh

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,276727,276727#msg-276727