Uneven High Load on the Nginx Server

[anish10dec]

Thanks  Maxim 

We enabled the upstream_request_time on both the server which shows response
time less than a sec for Upstream request.
It doesn't seems to be issue with Upstream Server . 
Even for the request which are HIT response time on the server on which
"Writing" is more varies from 10 sec to 60 sec and even more while on other
its less than 2 sec either its MISS or HIT.
Once we restart the nginx service the load reduces and so the response time,
but same comes back in 5 min duration. 
And if we stop the Nginx service on that server load decreases and same load
gets shifted to other with high "Writing" value.

We have the Cache Directory Mounted on SSD as well as RAM .
We are using CentOS 6.5 with both IPV4 and IPV6 enabled on the server.
Application/User Traffic directly connect to Server over IPV4/IPV6 IP . 

Output of iostat: 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.17    0.00    0.60    0.00    0.00   99.22

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read        
Blk_wrtn
sda              11.75        24.82       402.35      659063678      
10683772144
sdb             454.37      2180.70      3663.85   57905655098    
97288688008
sdc             565.74      1771.80      4512.49  47047922584   
119823249968
dm-0              0.00         0.00         0.00       5136                 
0
dm-1              0.05         0.40         0.30       10621106          
7853376
dm-2           1040.29      3952.19      8176.34 104945070786  217111937976
dm-3              1.01         1.40         8.06        37053274        
214028160
dm-4             33.97         2.39       271.65       63438578       
7213330456
dm-5             15.55        20.55       122.34     545791730        
3248559664

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.58    0.00    4.52    0.02    0.00   94.88

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00   181.00    0.00   34.60     0.00  1724.80    49.85  
  0.01    0.17   0.09   0.30
sdb               0.00    68.40   46.40  781.40 10710.40  6798.40    21.15  
  0.31    0.37   0.03   2.88
sdc               0.00    26.00   59.80  552.60 13425.60  4628.80    29.48  
  0.28    0.46   0.05   3.22
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00  
  0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00  
  0.00    0.00   0.00   0.00
dm-2              0.00     0.00  106.20 1428.40 24136.00 11427.20    23.17  
  0.79    0.52   0.04   6.00
dm-3              0.00     0.00    0.00    0.20     0.00     1.60     8.00  
  0.00    3.00   3.00   0.06
dm-4              0.00     0.00    0.00  212.00     0.00  1696.00     8.00  
  0.13    0.61   0.00   0.08
dm-5              0.00     0.00    0.00    3.40     0.00    27.20     8.00  
  0.00    0.47   0.47   0.16


"

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,269874,269878#msg-269878