<div dir="ltr"> Without knowing everything in the mix my first thought would be the NFS head node is being tapped out and can't keep up. Generally you'd solve this with some type of caching, either at a CDN level or you could look at the SlowFS module. I've not checked to see if it still compiles against the current releases but if you're dealing with short-life hot data or a consistent group of commonly accessed files, either solution would make a significant impact in reducing NFS load without having to resort to other potentially dodgy solutions.</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><b style="color:rgb(25,25,25)"><div style="color:rgb(0,0,0);font-weight:normal"><b style="color:rgb(25,25,25)"><font face="Lucida Grande" size="1"><div style="color:rgb(0,0,0);font-weight:normal"><span style="color:rgb(25,25,25);font-weight:bold">__________________</span></div><div style="color:rgb(0,0,0);font-weight:normal"><div style="margin:0px"><font color="#191919"><b><br></b></font></div><div style="margin:0px"><font color="#191919"><b>Scott Larson</b></font></div><div style="margin:0px"><div style="margin:0px"><font color="#007EFD"><span style="color:rgb(0,0,0)"><div style="margin:0px"><font color="#191919"><b><div style="margin:0px;font-weight:normal;color:rgb(120,120,120)"><span style="color:rgb(25,25,25)"><b><div style="margin:0px;font-weight:normal;color:rgb(120,120,120)">Systems Administrator</div></b></span></div><div style="margin:0px;font-weight:normal;min-height:8px"><br></div><div style="margin:0px;font-weight:normal"><b>Wiredrive/LA</b></div><div style="margin:0px;font-weight:normal"><a value="+13108238238" style="color:rgb(17,85,204)">310 823 8238 ext. 1106</a></div><div style="margin:0px;font-weight:normal"><a value="+13109432078" style="color:rgb(17,85,204)">310 943 2078</a> fax</div></b></font></div><div style="margin:0px"><font color="#2498FC"><a href="http://www.wiredrive.com/" style="color:rgb(17,85,204)" target="_blank">www.wiredrive.com</a></font></div><div style="margin:0px"><font color="#2498FC"><span style="color:rgb(0,0,0)"><div style="margin:0px;color:rgb(120,120,120)"><div style="margin:0px"><a href="http://www.twitter.com/wiredrive" style="color:rgb(17,85,204)" target="_blank"><font color="#2498FC">www.twitter.com/wiredrive</font></a></div><div style="margin:0px"><font color="#2498FC"><a href="http://www.wiredrive.com/facebook" style="color:rgb(17,85,204)" target="_blank">www.facebook.com/wiredrive</a></font></div></div></span></font></div></span></font></div></div></div></font></b></div></b></div></div></div>
<br><div class="gmail_quote">On Mon, Dec 29, 2014 at 12:52 PM, erankor2 <span dir="ltr"><<a href="mailto:nginx-forum@nginx.us" target="_blank">nginx-forum@nginx.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
In our production environment, we have several nginx servers running on<br>
Ubuntu that serve files from a very large (several PBs) NFS mounted storage.<br>
<br>
Usually the storage responds pretty fast, but occasionally it can be very<br>
slow. Since we are using aio, when a file read operation runs slow, it's not<br>
that bad - the specific HTTP request will just take a longer time to<br>
complete.<br>
However, we have seen cases in which the file open itself can take a long<br>
time to complete. Since the open is synchronous, when the open is slow all<br>
active requests on the same nginx worker process are delayed.<br>
One way to mitigate the problem may be to increase the number of nginx<br>
workers, to some number well above the number of CPU cores. This will make<br>
each worker handle less requests and therefore less requests will be delayed<br>
due to a slow open, but this solution sounds far from ideal.<br>
Another possibility (that requires some development) may be to create a<br>
thread that will perform the task of opening the file. The main thread will<br>
wait on the completion of the open-thread asynchronously, and will be<br>
available to handle other requests until the open completes. The child<br>
thread can either be created per request (many requests probably won't even<br>
need to open a file, thanks to the caching of open file handles), or<br>
alternatively some fancier thread pooling mechanism could be developed.<br>
<br>
I'd love to hear any thoughts / ideas on this subject<br>
<br>
Thanks,<br>
<br>
Eran<br>
<br>
Posted at Nginx Forum: <a href="http://forum.nginx.org/read.php?2,255847,255847#msg-255847" target="_blank">http://forum.nginx.org/read.php?2,255847,255847#msg-255847</a><br>
<br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote></div><br></div>