Multi layer caching

[Joachim]

Good day,

we are trying to optimise our caching strategy on cache servers that are in
front of a very large dataset 

200 x 1MB/s x 500 days = 1MB/s x 8’640'000'000s = 8.6 PB

We  serve TV stream segments (HLS) for over 200 channels where users can
fetch time shifted content. I.e. segments up to 7 days in the past (replay)
or even 18 months in the past (recordings). Segments will be accessed much
more rarely the older they get but the actively accessed data set is still
massive. Segments never change and expire 18 months after birth.

Currently we are limited by the write rates of the SSDs in the cache
servers. We can use proxy_cache_min_uses to reduce the write rate, but that
also lowers our hit rates. What I would like to set up is a multi-stage
cache based on different cache stores:

1) RAM disk (<1TB, very fast)
2) SSD (<10TB, fast writes)
3) HD (~100 TB)

Strategy:
a) Store every reply / requested segment in the RAM disk based cache.
b) If a segment is requested >2 times in 10min/1h store it in the SSD based
cache
	this may require an upstream re-fetch if it was already purged from the RAM
disk cache
c) If a segment is requested >10 times in 1h store it to the HD
	this may require an upstream re-fetch if it was already purged from the SSD
cache

I obviously don’t want to cascade 3 instances of NGINX to pull this off..

Can anyone point me in the right direction? (Config or Code)

Thanks a lot in advance,
Joachim

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,282832,282832#msg-282832