Cache questions

[Maxim Dounin]

Hello!

On Mon, Jul 13, 2009 at 04:02:06PM -0400, Jim Ohlstein wrote:

>
>
> Igor Sysoev wrote:
>> On Mon, Jul 13, 2009 at 03:33:21PM +0400, Maxim Dounin wrote:
>>
>>   
>>> Hello!
>>>
>>> On Fri, Jul 10, 2009 at 07:39:46PM -0400, Jim Ohlstein wrote:
>>>
>>>     
>>>> I'm using the fastcgi cache for static files (images, 
>>>> javascript,css)  and just found multiple lines in the error log 
>>>> like this one:
>>>>
>>>> 2009/07/10 10:22:54 [crit] 22476#0: ngx_slab_alloc() failed: no 
>>>> memory  in cache keys zone "one"
>>>>
>>>> So I increased the memory available for the zone and reloaded 
>>>> nginx. It  took over five hours to go through the cache but these 
>>>> are the relevant  entries:
>>>>
>>>> 2009/07/10 12:11:03 [notice] 21038#0: start cache manager process 32730
>>>> 2009/07/10 12:11:04 [notice] 21038#0: cache manager process 22480 
>>>> exited  with code 0
>>>>
>>>> and finally
>>>>
>>>> 2009/07/10 17:43:27 [notice] 32730#0: http file cache:   
>>>> /usr/local/nginx/cache 11638.289M, bsize: 4096
>>>>
>>>> My questions are:
>>>>
>>>> Is that simply the total (11638.289MB or 11.4GB) of all of the file 
>>>>  sizes, or is that the actual disk space consumed taking into 
>>>> account  total number of blocks used multiplied by the block size? 
>>>> The number        
>>> It's size on disk (i.e. number of blocks * block size), but for  
>>> files only (it doesn't take directories into account).
>>>     
>>
>> Just note: nginx rounds a file size to the bsize.
>> bsize is f_bsize from statfs() or f_frsize from statvfs().
>> I'm not sure that bsize matches always a filesystem allocation unit.
>>   
>
> OK, Thanks.
>
> In trying to tune this, if I set fastcgi_cache_min_uses to 2, does that  
> mean that a file will only be written to the cache the second time that  
> it is requested? Google translate did not give me a clear answer to this  
> from the Russian documentation. I think that I could improve efficiency  
> greatly if I didn't cache files on the first request.

Yes, file will be written only after second request.  Note that 
shared memory will be allocated anyway.

> I would know  
> better if I could get some statistics. I know the last time I asked the  
> answer was "not yet". Do you have any idea when this might be  
> implemented even on a rudimentary basis?

Cache status was implemented in 0.8.3:

    *) Feature: the $upstream_cache_status variable.

You may log it to access_log and collect stats you want.

>>>> could be quite different given what I estimate are nearly one 
>>>> million  mostly small files in the cache at this point.
>>>>
>>>> When I next upgrade nginx (I'm running 0.8.4), and I attempt a 
>>>> "graceful  upgrade" will it have to go through this entire process 
>>>> again?
>>>>       
>>> Yes.  Binary upgrade doesn't preserve shared memory zones, so cache 
>>> will be rescanned again.
>>>
>>> Note that nginx uses cache even before it was completely scanned, so 
>>> it shouldn't be a problem.
>>>     
>
> Yes, but it uses a fair bit of system resources scanning a large cache.  
> Nothing much to be done about it though.

Actually, nginx tries to be nice and pauses scanning for 200ms 
if takes more than 200ms to scan 100 files.

Maxim Dounin