cache manager process - i/o perf

[Andy]

I hit similar problem ...

Can I know what is the ingest Gbps into the SSDs when you hit the problem?
and How many cached file nodes in cache-manager? i have millions ...


On Wed, May 21, 2014 at 7:57 PM, ixos <nginx-forum at nginx.us> wrote:

> I'm having problem with I/O performance. I'm running nginx as caching
> reverse proxy server.
> When cache size on disk exceeds max_size cache manager starts working, but
> it causes two problems occur:
>
> 1) I/O %util reach 100% and nginx starts dropping connections
> 2) cache manager process dosen't unlink files speed enough to delete old
> file. So cache becomes bigger util the space on disk ends.
>
> Can you give me an idea how can I solve those problems. Below are some
> details.
>
> #build on 20x 300GB SAS disks with 2 SSDs for Cachecade.
>
> # storcli64 /c0 show
> VD LIST :
> =======
>
> ----------------------------------------------------------------
> DG/VD TYPE   State Access Consist Cache Cac sCC       Size Name
> ----------------------------------------------------------------
> 1/2   RAID60 Optl  RW     Yes     RaWBC R   ON    4.357 TB
> 2/1   Cac0   Optl  RW     Yes     RaWTD -   ON  557.875 GB
> ----------------------------------------------------------------
>
> # mount
> /dev/sdb1 on /cache type ext4 (rw,noatime,data=ordered)
>
> # df -h /dev/sdb1
> /dev/sdb1       4.3T  3.2T  828G  80% /cache
>
>
> # for pid in `pgrep nginx `;do ionice -p $pid ;done
> unknown: prio 4 <- master
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0
> best-effort: prio 0 <- workers
> idle <- cache manager
>
> # grep proxy_cache_path nginx.conf
> proxy_cache_path /cache zone=my-cache:20000msize=3355443m
>
> # netstat -sp|grep -i drop
>     6335115 SYNs to LISTEN sockets dropped
>
> # iostat -dx 1 /dev/sdb |grep ^sdb | awk '{print $14}'
> 24.40
> 31.20
> 26.80
> 23.60
> 26.80
> 16.00
> 34.80
> 35.20
> 29.60
> ...
> 14.40
> 15.60
> 11.60
> 16.00
> 17.20
> 18.00
> 17.20
> 42.00
> 90.80 <- cache manager process starts
> 100.00
> 100.00
> 29.20
> 100.00
> 100.00
> 100.00
> 52.00
> 100.00
> 100.00
> 100.00
>
> Posted at Nginx Forum:
> http://forum.nginx.org/read.php?2,250247,250247#msg-250247
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20140522/9e8c064a/attachment.html>