proxy_cache with proxy_intercept_errors bug

[ifeltsweet]

Hi,

I have a very simple proxy config:

http {
	proxy_cache_path /var/www/cache levels=1:2 keys_zone=s3-images-cache:50m
inactive=1M max_size=1000m;
	proxy_temp_path /var/www/cache/tmp;
}

server {
        listen 80;
        server_name images.example.com;

        location / {
                proxy_cache s3-images-cache;
                proxy_cache_key $scheme$proxy_host$uri$is_args$args;
                proxy_cache_bypass $http_purge_cache;
                proxy_cache_valid any 1y;
                proxy_pass http://images-example.s3.amazonaws.com;
                add_header X-Cache $upstream_cache_status;
                
                proxy_intercept_errors on;
                error_page 404 = @no_image;
        }

        location @no_image {
                return 403;
        }
}

1. Let's request /image.jpg.
2. Request is sent to proxy for /image.jpg (does not exists yet).
3. Backend responds with 404.
4. "proxy_intercept_errors on" kicks in and "error_page 404 = @no_image" is
called.
5. Nginx returns 403.
6. Do another request for same image and see that "X-Cache: HIT" is set. We
are clearly hitting cache.

But, if we check /var/www/cache/ folder at this time we will see that there
is no cache item created for this request. So does it mean Nginx keeps the
cache for it in memory and forgot to write to file?

7. Let's upload /image.jpg to backend.
8. Now do "PURGE-CACHE: 1" request to that image. We see that now we get the
image instead of 403. Good.

If we check /var/www/cache/ we will see that the cache file is now finally
created for this request. Also good.

9. Now here is the problem: lets request /image.jpg again.
10. Nginx returns 403 with "X-Cache: HIT". Why? So it's hitting the cache
but returning something else not what is in /var/www/cache folder?? How?


My only explanation to this is it seems that Nginx is caching the response
in memory and doesn't write to file when we are hitting error with our
custom error_page in the proxied responses. Furthermore when using
proxy_cache_bypass it does not overwrite in-memory cache, so that subsequent
requests to the same item will be using old cache which is stored in memory
and not the new one created in the cache folder.

Could someone please let me know if I am doing something wrong or this
really is a bug. Spent last 3 days fighting this.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,262651,262651#msg-262651