Odd cache behavior

[Igor Sysoev]

On Tue, Oct 06, 2009 at 03:56:46PM +0400, Igor Sysoev wrote:

> On Tue, Oct 06, 2009 at 03:46:48PM +0400, Igor Sysoev wrote:
> 
> > On Sat, Oct 03, 2009 at 09:54:59AM -0400, Jim Ohlstein wrote:
> > 
> > > Igor Sysoev wrote:
> > > > On Fri, Oct 02, 2009 at 01:07:24PM -0400, Jim Ohlstein wrote:
> > > > 
> > > >> Hello,
> > > >>
> > > >> I'm seeing odd disk usage statistics for my cache. The cache is 
> > > >> primarily of small files (images, javascript, and css) and sits on a 
> > > >> 128GB SSD using ext2 file system mounted with "noatime" option. The 
> > > >> cache is all that the the drive is being used for at this time.
> > > >>
> > > >> The attached graphs were generated by Munin using df. The data from du 
> > > >> correlates closely with df when run manually.
> > > >>
> > > >> As you can see, periodically, with no set interval, disk usage rises 
> > > >> gradually, and then drops precipitously. I don't see anything out of the 
> > > >> ordinary in the error log. The cache serves about 2.8-3 million requests 
> > > >> per day, up from ~2.2-2.4 million one month ago.
> > > >>
> > > >> The cache config is:
> > > >>
> > > >> fastcgi_cache_path /falcon/cache levels=1:2 keys_zone=one:1024m 
> > > >> inactive=1d max_size=80g;
> > > >>
> > > >> where the SSD is mounted as /falcon.
> > > > 
> > > > As I understand /falcon is used solely for nginx cache ?
> > > 
> > > This is correct.
> > > 
> > > > Could you monitor inode usage also ?
> > > 
> > > See attached graphs.
> > > 
> > > > Did you restart nginx in the times of the drops or not ?
> > > 
> > > No.
> > > 
> > > > Currently I have no idea why it may be.
> > 
> > I saw the similar pattern, when nginx is restarted (not reHUPed) at the cliff
> > growing start point: then cache loader loads many old inactive cache items
> > and sets close inactivity timers for the items.  Therefore cache size grows
> > lineary and at some time all these inactive items are purged almost
> > simultaneously causing precipitous drop.
> > 
> > Have you restarted nginx at the growing start points ?

This may be not a restart, but an online upgrade too.


-- 
Igor Sysoev
http://sysoev.ru/en/