Cache questions

[Jim Ohlstein]

Maxim Dounin wrote:
> Hello!
>
> On Mon, Jul 13, 2009 at 08:45:03PM -0400, Jim Ohlstein wrote:
>
>   
>> Igor Sysoev wrote:
>>     
>>> On Mon, Jul 13, 2009 at 04:02:06PM -0400, Jim Ohlstein wrote:
>>>
>>>   
>>>       
>>>> Igor Sysoev wrote:
>>>>     
>>>>         
>>>>> On Mon, Jul 13, 2009 at 03:33:21PM +0400, Maxim Dounin wrote:
>>>>>
>>>>>        
>>>>>           
>>>>>> Hello!
>>>>>>
>>>>>> On Fri, Jul 10, 2009 at 07:39:46PM -0400, Jim Ohlstein wrote:
>>>>>>
>>>>>>            
>>>>>>             
>>>>>>> I'm using the fastcgi cache for static files (images, 
>>>>>>> javascript,css)  and just found multiple lines in the error log 
>>>>>>> like this one:
>>>>>>>
>>>>>>> 2009/07/10 10:22:54 [crit] 22476#0: ngx_slab_alloc() failed: no 
>>>>>>> memory  in cache keys zone "one"
>>>>>>>
>>>>>>> So I increased the memory available for the zone and reloaded 
>>>>>>> nginx. It  took over five hours to go through the cache but 
>>>>>>> these are the relevant  entries:
>>>>>>>
>>>>>>> 2009/07/10 12:11:03 [notice] 21038#0: start cache manager process 32730
>>>>>>> 2009/07/10 12:11:04 [notice] 21038#0: cache manager process 
>>>>>>> 22480 exited with code 0
>>>>>>>
>>>>>>> and finally
>>>>>>>
>>>>>>> 2009/07/10 17:43:27 [notice] 32730#0: http file cache:   
>>>>>>> /usr/local/nginx/cache 11638.289M, bsize: 4096
>>>>>>>
>>>>>>> My questions are:
>>>>>>>
>>>>>>> Is that simply the total (11638.289MB or 11.4GB) of all of the 
>>>>>>> file  sizes, or is that the actual disk space consumed taking 
>>>>>>> into account  total number of blocks used multiplied by the 
>>>>>>> block size? The number                 
>>>>>>>               
>>>>>> It's size on disk (i.e. number of blocks * block size), but for  
>>>>>> files only (it doesn't take directories into account).
>>>>>>            
>>>>>>             
>>>>> Just note: nginx rounds a file size to the bsize.
>>>>> bsize is f_bsize from statfs() or f_frsize from statvfs().
>>>>> I'm not sure that bsize matches always a filesystem allocation unit.
>>>>>        
>>>>>           
>>>> OK, Thanks.
>>>>
>>>> In trying to tune this, if I set fastcgi_cache_min_uses to 2, does 
>>>> that mean that a file will only be written to the cache the second 
>>>> time that it is requested? Google translate did not give me a clear 
>>>> answer to this from the Russian documentation. I think that I could 
>>>> improve efficiency greatly if I didn't cache files on the first 
>>>> request.
>>>>     
>>>>         
>>> Yes, a file will be cached only on the second request made during period
>>> set in by "inactive" option of fastcgi_cache_path directive.
>>>
>>>   
>>>       
>>>> I would know better if I could get some statistics. I know the last 
>>>> time I asked the answer was "not yet". Do you have any idea when this 
>>>> might be implemented even on a rudimentary basis?
>>>>     
>>>>         
>>> You may log $upstream_cache_status.
>>>   
>>>       
>> Can you give me an example of how to do this?
>>
>> I have tried various permutations of:
>>
>> location ~ (jpg|jpeg|png|gif|ico|js|css)$ {
>>    fastcgi_pass unix:/tmp/my.sock;
>>    fastcgi_cache one;
>>    fastcgi_cache_key unix:/tmp/my.sock.1$request_uri;
>>    fastcgi_ignore_headers  Cache-Control  Expires;
>>    fastcgi_cache_valid 200 302 1d;
>>    fastcgi_cache_valid 301 7d;
>>    fastcgi_cache_valid any 10m;
>>    fastcgi_cache_min_uses 2;
>>    include /usr/local/nginx/conf/fastcgi_params;
>>    fastcgi_param  SCRIPT_NAME myscript;
>>    fastcgi_buffers 64 8k;
>>    access_log  logs/my-cache.access.log;
>>    log_format '$remote_addr - $remote_user [$time_local] '
>>                          '"$request" $status $body_bytes_sent '
>>                          '"$http_referer" "$http_user_agent"' '  
>> $upstream_cache_status ';
>> }
>>
>> The logs give me all above with exception of "$upstream_cache_status".  
>> Every entry ends with the user agent.
>>     
>
> log_format  cachestatus  '$remote_addr - $remote_user [$time_local] '
>                          '"$request" $status $body_bytes_sent '
>                          '"$http_referer" "$http_user_agent" '
>                          '$upstream_cache_status';
>
> access_log  /path/to/log  cachestatus;
>
> Note that log_format's first argument is format name, and second 
> argument of access_log is format name too (optional, defaults 
> to 'combined' predefined format).
>
> See http://wiki.nginx.org/NginxHttpLogModule for details.
>   

Thank you Maxim. Very much appreciated. I had read that page but I guess 
I didn't quite "get it". I've always used the standard "combined" format 
as it had suited my needs until now.

It is now logging "HIT" or "MISS" for each request in that location block.

It is safe to assume that with fastcgi_cache_min_uses set at 2 that for 
every first miss shared memory is allocated, that the file is written to 
the cache on the second miss, and that each logged "hit" is an actual 
serve from the cache (third and subsequent requests for the file)?

> Maxim Dounin
>
>
>   
Jim