NGINX stale-while-revalidate cluster

[Joan Tomàs i Buliart]

Hi,

We are implementing an stale-while-revalidate webserver cluster with NGINX.

We are using the new proxy_cache_background_update to answer request as 
soon as possible while NGINX updates the content from the origin in the 
background. This solution works perfectly when the requests for the same 
object are served by the same NGINX server (when we have only one server 
or when we have a previous load balancer that classifies the requests).

In our scenario we have a round robin load balancer (ELB) and we need to 
scale the webservers layer. So, as a consequence, only the Nginx that 
receive the request updates the cache content while the others keep the 
old version. This means that, we can send old versions of content due to 
the content not being updated on all the webservers. The problem 
accentuates when we put a CDN in front of the webservers.

We are thinking on developing something that once an Nginx instance 
updates its cache would let know all other instances to get a copy of 
the newest content. We are thinking about processing NGINX logs and, 
when it detects a MISS, EXPIRED or UPDATING cache status, it makes a 
HEAD request to the other NGINXs on the cluster to force the 
invalidation of this content.


Do any of you have dealt with this problem or a similar one?

We have also tried the post_action but it is blocking the client request 
until it completes. It is not clear for us which would be the best 
approach. The options that we are considering are:

- NGINX module
- LUA script
- External script that process syslog entries from NGINX

What would be your recommendation?

Many thanks in advance,


-- 
Joan Tomàs-Buliart
www.marfeel.com <http://www.marfeel.com>
Joan Tomàs-Buliart
+34 931 785 950
www.marfeel.com <http://www.marfeel.com>
Discover our referral program!! 
<http://blog.marfeel.com/earn-money-marfeel-referral-program/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20170707/4c07d540/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: olhgllimkponlhjm.gif
Type: image/gif
Size: 4180 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20170707/4c07d540/attachment-0001.gif>