Odd cache behavior

[Igor Sysoev]

On Sat, Oct 03, 2009 at 09:54:59AM -0400, Jim Ohlstein wrote:

> Igor Sysoev wrote:
> > On Fri, Oct 02, 2009 at 01:07:24PM -0400, Jim Ohlstein wrote:
> > 
> >> Hello,
> >>
> >> I'm seeing odd disk usage statistics for my cache. The cache is 
> >> primarily of small files (images, javascript, and css) and sits on a 
> >> 128GB SSD using ext2 file system mounted with "noatime" option. The 
> >> cache is all that the the drive is being used for at this time.
> >>
> >> The attached graphs were generated by Munin using df. The data from du 
> >> correlates closely with df when run manually.
> >>
> >> As you can see, periodically, with no set interval, disk usage rises 
> >> gradually, and then drops precipitously. I don't see anything out of the 
> >> ordinary in the error log. The cache serves about 2.8-3 million requests 
> >> per day, up from ~2.2-2.4 million one month ago.
> >>
> >> The cache config is:
> >>
> >> fastcgi_cache_path /falcon/cache levels=1:2 keys_zone=one:1024m 
> >> inactive=1d max_size=80g;
> >>
> >> where the SSD is mounted as /falcon.
> > 
> > As I understand /falcon is used solely for nginx cache ?
> 
> This is correct.
> 
> > Could you monitor inode usage also ?
> 
> See attached graphs.
> 
> > Did you restart nginx in the times of the drops or not ?
> 
> No.
> 
> > Currently I have no idea why it may be.

I saw the similar pattern, when nginx is restarted (not reHUPed) at the cliff
growing start point: then cache loader loads many old inactive cache items
and sets close inactivity timers for the items.  Therefore cache size grows
lineary and at some time all these inactive items are purged almost
simultaneously causing precipitous drop.

Have you restarted nginx at the growing start points ?


-- 
Igor Sysoev
http://sysoev.ru/en/