Odd cache behavior

[Igor Sysoev]

On Tue, Oct 06, 2009 at 03:46:48PM +0400, Igor Sysoev wrote:

> On Sat, Oct 03, 2009 at 09:54:59AM -0400, Jim Ohlstein wrote:
> 
> > Igor Sysoev wrote:
> > > On Fri, Oct 02, 2009 at 01:07:24PM -0400, Jim Ohlstein wrote:
> > > 
> > >> Hello,
> > >>
> > >> I'm seeing odd disk usage statistics for my cache. The cache is 
> > >> primarily of small files (images, javascript, and css) and sits on a 
> > >> 128GB SSD using ext2 file system mounted with "noatime" option. The 
> > >> cache is all that the the drive is being used for at this time.
> > >>
> > >> The attached graphs were generated by Munin using df. The data from du 
> > >> correlates closely with df when run manually.
> > >>
> > >> As you can see, periodically, with no set interval, disk usage rises 
> > >> gradually, and then drops precipitously. I don't see anything out of the 
> > >> ordinary in the error log. The cache serves about 2.8-3 million requests 
> > >> per day, up from ~2.2-2.4 million one month ago.
> > >>
> > >> The cache config is:
> > >>
> > >> fastcgi_cache_path /falcon/cache levels=1:2 keys_zone=one:1024m 
> > >> inactive=1d max_size=80g;
> > >>
> > >> where the SSD is mounted as /falcon.
> > > 
> > > As I understand /falcon is used solely for nginx cache ?
> > 
> > This is correct.
> > 
> > > Could you monitor inode usage also ?
> > 
> > See attached graphs.
> > 
> > > Did you restart nginx in the times of the drops or not ?
> > 
> > No.
> > 
> > > Currently I have no idea why it may be.
> 
> I saw the similar pattern, when nginx is restarted (not reHUPed) at the cliff
> growing start point: then cache loader loads many old inactive cache items
> and sets close inactivity timers for the items.  Therefore cache size grows
> lineary and at some time all these inactive items are purged almost
> simultaneously causing precipitous drop.
> 
> Have you restarted nginx at the growing start points ?

A note: the lineary growth period is equal to an "inactive=" parameter in
fastcgi_cache_path.


-- 
Igor Sysoev
http://sysoev.ru/en/