nginx serving static files - caching?

[Gerard Flanagan]

revirii wrote:
> Hi,
>
> we want to establish some service that will deliver static files
> (images) with a size between 1-120 KB. The service should handle (at the
> moment) ~130.000.000 requests per month, so not that few and not
> extremly many requests. The to-be-served files have a size of several
> hundred GBs in total. Some of the files (5-10%) are asked more
> frequently and the bigger part less frequently.
>
> Let's say we have 16 GB Ram available for the server. We could simply
> avoid using some caching mechanism and read the files from disk. For my
> scenario: is it possible to establish some reasonable caching? No
> decision regarding the to-be-used technology is made so far. An
> associate favours varnish (with some webserver as backend) and some
> least recently used strategy, i'd give nginx+memcached a try. Any
> recommendations?
>   

I'm investigating Kyoto Cabinet/Kyoto Tycoon for a caching strategy at 
the moment and I can recommend it, though the use case is lots of small 
text files. If you don't know, Kyoto Cabinet is a file based "key-value" 
database. There are various flavours of Cabinet, one of which is a 
"directory" database, so if you create a Cabinet with a '.kcd' extension 
then the resulting database just stores each object as a unique file in 
a directory.

Kyoto Tycoon then is a HTTP server through which you can access (one or 
many) databases via GET and POST. You can access the databases directly, 
but one of the benefits of using Tycoon is auto-expiration - whenever 
you set a value you can give an expiry time and the file will be removed 
after that.

Honestly not an expert in this sort of thing, but the setup I'm looking 
at is:

    [nginx] -> [4 x Tornado] -> [16 x Kyoto Tycoon] -> [4096 x Kyoto 
Cabinet]

Tornado being the Python webserver which, as well as actually retrieving 
the data from a remote source, makes the decision of which database to 
store items based on a hash of the url. I'm thinking of a single server 
(box), but since it's all HTTP I suppose you could have a fan out among 
servers if required.

Again, not an expert, and this is all exploratory at this point, so take 
with a pinch of salt.