cache manager process - i/o perf

[ixos]

I'm having problem with I/O performance. I'm running nginx as caching
reverse proxy server.
When cache size on disk exceeds max_size cache manager starts working, but
it causes two problems occur:

1) I/O %util reach 100% and nginx starts dropping connections
2) cache manager process dosen't unlink files speed enough to delete old
file. So cache becomes bigger util the space on disk ends.

Can you give me an idea how can I solve those problems. Below are some
details.

#build on 20x 300GB SAS disks with 2 SSDs for Cachecade.

# storcli64 /c0 show
VD LIST :
=======

----------------------------------------------------------------
DG/VD TYPE   State Access Consist Cache Cac sCC       Size Name 
----------------------------------------------------------------
1/2   RAID60 Optl  RW     Yes     RaWBC R   ON    4.357 TB      
2/1   Cac0   Optl  RW     Yes     RaWTD -   ON  557.875 GB      
----------------------------------------------------------------

# mount
/dev/sdb1 on /cache type ext4 (rw,noatime,data=ordered)

# df -h /dev/sdb1
/dev/sdb1       4.3T  3.2T  828G  80% /cache


# for pid in `pgrep nginx `;do ionice -p $pid ;done
unknown: prio 4 <- master
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0
best-effort: prio 0 <- workers
idle <- cache manager

# grep proxy_cache_path nginx.conf
proxy_cache_path /cache zone=my-cache:20000msize=3355443m

# netstat -sp|grep -i drop
    6335115 SYNs to LISTEN sockets dropped

# iostat -dx 1 /dev/sdb |grep ^sdb | awk '{print $14}'
24.40
31.20
26.80
23.60
26.80
16.00
34.80
35.20
29.60
...
14.40
15.60
11.60
16.00
17.20
18.00
17.20
42.00
90.80 <- cache manager process starts
100.00
100.00
29.20
100.00
100.00
100.00
52.00
100.00
100.00
100.00

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,250247,250247#msg-250247