Nginx cache grouped by keys

[Wandenberg Peixoto]

Hi,

I work for a company which uses a lot the proxy module and the proxy cache
feature.
We always have problems on clearing the cache because it has to be done url
by url.


Thinking in a way to solve this I made the patch in attachment
(nginx_cache_with_groups.patch (http://pastebin.com/Gy6N6knq), for nginx
1.0.10 and 0001-adding-parameter-to-purge-cache-by-groups.patch (
http://pastebin.com/kegfRPUH), for ngx_cache_purge 1.4).

Basically it changes the (fastcgi|proxy|scgi|uwsgi)_cache_key directives
adding an optional value named "group key".
This value accept any variable available on nginx configuration file (query
parameters, headers,...) just like the key value.


When using this value a directory is created using the same structure of
levels to store all files which have the same group key.
Doing this we can purge the cache using a specific file, like is done
today, or all files from a group. (I change the cache_purge_module to make
this possible too)

I will give an example to be clear.
Imagine the configuration attached (nginx.conf -
http://pastebin.com/DVyeSrQT) and that "html/static" folder has some files
like test.css, test1.html and test2.html.

With that configuration the extension file will be used as the group key
and the uri as cache key.

When we execute the following commands the corresponded files are created:
command: curl -v 'http://localhost:8090/test1.html'
file:
./e/53/a/fc35fdc70d5fc69d269883a822c7a53e/1/3d/9/bcccd499fbe303221add1b6e57d693d1
-> where fc35fdc70d5fc69d269883a822c7a53e is from group key and
bcccd499fbe303221add1b6e57d693d1 from file key

command: curl -v 'http://localhost:8090/test2.html'
file:
./e/53/a/fc35fdc70d5fc69d269883a822c7a53e/4/86/6/558156b89af1ccd4844556ab699d6864
-> where fc35fdc70d5fc69d269883a822c7a53e is from group key and
558156b89af1ccd4844556ab699d6864 from file key

command: curl -v 'http://localhost:8090/test.css'
file:
./a/66/2/c7a628cba22e28eb17b5f5c6ae2a266a/a/f0/d/9a96e0fe2f89d77e756fa45ae882df0a
-> where c7a628cba22e28eb17b5f5c6ae2a266a is from group key and
9a96e0fe2f89d77e756fa45ae882df0a from file key

Now we can purge the cache from css file doing
curl -v 'localhost:8090/purge/test.css' -H 'X-Group-Clean:0'

Or purge all html files doing
curl -v 'localhost:8090/purge/_.html' -H 'X-Group-Clean:1' (the underline
in _.html is only to do a match on location)


The Group key can be as general or as specific like we want/need.

The default value for directive group_clean_cache_purge is 0, and the value
used to it must be a text which converted to a number (ngx_atoi) results in
a zero, purge only the selected file, or non zero, purge all files from
selected group.


Another example of use is we have a service to resize images, and the
desired size is passed through the uri like
http://localhost/480x360/1361234.jpg or http://localhost/x240/1361234.jpg

With that patch we can group all resized images using the image id
(1361234, in the example) as the group key and than purge all cached files
at once.

I hope this patch can be applied in future versions of Nginx and made
available to everyone.

Regards,
Wandenberg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20111205/adc5f2b2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-adding-parameter-to-purge-cache-by-groups.patch
Type: text/x-diff
Size: 24274 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20111205/adc5f2b2/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx_cache_with_groups.patch
Type: text/x-diff
Size: 19133 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20111205/adc5f2b2/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx.conf
Type: application/octet-stream
Size: 1298 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20111205/adc5f2b2/attachment-0001.obj>