Proxying setup delivering wrong cache entry in some edge cases

[Reinis Rozitis]

> For example, a request to https://test.example.org/bla/fasel would deliver
> the content for https://foo.example.org/bla/fasel. So basically it delivers
> content for the wrong subdomain. Those occasions are very, very rare and
> totally random in regards to the subdomain from which the content gets
> delivered.

Your configuration has:
proxy_cache_key $ae$scheme://$http_host$request_uri;
and
proxy_cache_use_stale error timeout invalid_header http_502;

I would start with either disabling the proxy_cache_use_stale and/or inspecting the $http_host.

If I'm not wrong $http_host doesn't get the same treatment as $host which also comes from user request headers but if not present or empty gets set as $server_name matching the request. 
So if a client doesn't send the 'Host:'  header there might be cache entries basically just with the $request_uri which are served in some specific cases.

Nginx always stores and returns whatever the backend sent as a response. 
Since you change the Host header:

    proxy_set_header Host $upstream_endpoint;
    proxy_set_header X-Forwarded-Host $http_host;

If possible you could add some debug headers on the backends - to see if request actually landing on the nginx proxy is correctly passed on the backend (like you could again be missing the X-Forwarded-Host header).

Also you can do a simple MD5 on the problematic request (like md5 https://foo.example.org/bla/fasel -> 4DFDF87BB2FD82629ACB91BB1B1B2A1C (obviously for the gzipped content you have to add 'gzip' at the beginning) and then check if the cache file in /opt/example-org-cache/c/a1/4dfdf87bb2fd82629acb91bb1b1b2a1c actually exists and what's the content of it.

rr