Odd cache behavior

[Jim Ohlstein]

Igor Sysoev wrote:
> On Tue, Oct 06, 2009 at 03:56:46PM +0400, Igor Sysoev wrote:
> 
>> On Tue, Oct 06, 2009 at 03:46:48PM +0400, Igor Sysoev wrote:
>>
>>> On Sat, Oct 03, 2009 at 09:54:59AM -0400, Jim Ohlstein wrote:
>>>
>>>> Igor Sysoev wrote:
>>>>> On Fri, Oct 02, 2009 at 01:07:24PM -0400, Jim Ohlstein wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm seeing odd disk usage statistics for my cache. The cache is 
>>>>>> primarily of small files (images, javascript, and css) and sits on a 
>>>>>> 128GB SSD using ext2 file system mounted with "noatime" option. The 
>>>>>> cache is all that the the drive is being used for at this time.
>>>>>>
>>>>>> The attached graphs were generated by Munin using df. The data from du 
>>>>>> correlates closely with df when run manually.
>>>>>>
>>>>>> As you can see, periodically, with no set interval, disk usage rises 
>>>>>> gradually, and then drops precipitously. I don't see anything out of the 
>>>>>> ordinary in the error log. The cache serves about 2.8-3 million requests 
>>>>>> per day, up from ~2.2-2.4 million one month ago.
>>>>>>
>>>>>> The cache config is:
>>>>>>
>>>>>> fastcgi_cache_path /falcon/cache levels=1:2 keys_zone=one:1024m 
>>>>>> inactive=1d max_size=80g;
>>>>>>
>>>>>> where the SSD is mounted as /falcon.
>>>>> As I understand /falcon is used solely for nginx cache ?
>>>> This is correct.
>>>>
>>>>> Could you monitor inode usage also ?
>>>> See attached graphs.
>>>>
>>>>> Did you restart nginx in the times of the drops or not ?
>>>> No.
>>>>
>>>>> Currently I have no idea why it may be.
>>> I saw the similar pattern, when nginx is restarted (not reHUPed) at the cliff
>>> growing start point: then cache loader loads many old inactive cache items
>>> and sets close inactivity timers for the items.  Therefore cache size grows
>>> lineary and at some time all these inactive items are purged almost
>>> simultaneously causing precipitous drop.
>>>
>>> Have you restarted nginx at the growing start points ?
> 
> This may be not a restart, but an online upgrade too.
> 
> 

I had error log level set for "crit" so I can't be sure. The binary was 
built on September 29 @ 12:39 which was after the the climb started but 
it's possible there was a restart earlier. I generally don't restart 
except when upgrading or rebooting but it is possible.

I will set error logging to "notice" and upgrade the binary today and 
watch what happens.

-- 
Jim Ohlstein