nginx cache mounted on tmpf getting fulled

[steve]

Hi,


On 01/17/2017 02:37 PM, Peter Booth wrote:
> I'm curious, why are you using tmpfs for your cache store? With fast local storage bring so cheap, why don't you devote a few TB to your cache?
>
> When I look at the techempower benchmarks I see that openresty (an nginx build that comes with lots of lua value add) can serve 440,000 JSON responses per sec with 3ms latency. That's on five year old E7-4850 Westmere hardware at 2.0GHz, with 10G NICs. The min latency to get a packet from nginx through the kernel stack and onto the wire is about 4uS for a NIC of that vintage, dropping to 2uS with openonload (sokarflare's kernel bypass).
>
> As ippolitiv suggests, your cache already has room for 1.6M items- that's a huge amount. What kind of hit rate are you seeing for your cache?
>
> One way to manage cache size is to only cache popular items- if you set proxy_cache_min_uses =4 then only objects that are requested four times will be cached, which will increase your hit rates and reduce the space needed for the cache.
>
> Peter
>
> Sent from my iPhone
>
>> On Jan 16, 2017, at 4:41 AM, Igor A. Ippolitov <iippolitov at nginx.com> wrote:
>>
>> Hello,
>>
>> Your cache have 200m space for keys. This is around 1.6M items, isn't it?
>> How much files do you have in your cache? May we have a look at
>> `df -i ` and `du -s /cache/123` output, please?
>>
>>> On 06.01.2017 08:40, omkar_jadhav_20 wrote:
>>> Hi,
>>>
>>> I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
>>> access we have mounted cache of nginx of different application on RAM.But
>>> even after giving enough buffer of size , now and then cache is getting
>>> filled , below are few details of files for your reference :
>>> maximum size given in nginx conf file is 500G , while mouting we have given
>>> 600G of space i.e. 100G of buffer.But still it is getting filled 100%.
>>>
>>> fstab entries :
>>> tmpfs                   /cache/123            tmpfs   defaults,size=600G
>>>     0 0
>>> tmpfs                   /cache/456            tmpfs   defaults,size=60G
>>>    0 0
>>> tmpfs                   /cache/789            tmpfs   defaults,size=110G
>>>     0 0
>>>
>>> cache getting filled , df output:
>>>
>>> tmpfs                              tmpfs      60G   17G   44G  28%
>>> /cache/456
>>> tmpfs                              tmpfs     110G  323M  110G   1%
>>> /cache/789
>>> tmpfs                              tmpfs     600G  600G     0 100%
>>> /cache/123
>>>
>>> nginx conf details :
>>>
>>> proxy_cache_path    /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
>>> inactive=3d;
>>>
>>> server{
>>> listen 80;
>>> server_name   dvr.catchup.com;
>>> location  ~.*.m3u8 {
>>> access_log /var/log/nginx/access_123.log access;
>>> proxy_cache off;
>>> root    /xyz/123;
>>> if (!-e $request_filename) {
>>> #origin url will be used if content is not available on DS
>>>       proxy_pass http://10.10.10.1X;
>>> }
>>> }
>>> location     / {
>>> access_log /var/log/nginx/access_123.log access;
>>> proxy_cache_valid    3d;
>>> proxy_cache   a123;
>>> root    /xyz/123;
>>> if (!-e $request_filename) {
>>> #origin url will be used if content is not available on server
>>>       proxy_pass http://10.10.10.1X;
>>> }
>>> proxy_cache_key    $proxy_host$uri;
>>> }
>>> }
>>>
>>> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx at nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>> _______________________________________________
>> nginx mailing list
>> nginx at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

So that's a total of 1TB of memory allocated to caches. Do you have that 
much spare on your server? Linux will allocate *up  to* the specified 
amount *as long as it's spare*. It would be worth looking at your server 
to ensure that 1TB memory is spare before blaming nginx.

You can further improve performance and safety by mounting them

nodev,noexec,nosuid,noatime,async,size=xxxM,mode=0755,uid=xx,gid=xx

To answer this poster... memory is even faster!

Steve

-- 
Steve Holdoway BSc(Hons) MIITP
http://www.greengecko.co.nz
Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa