Serving files from a slow NFS storage

[erankor2]

Hi all,

In our production environment, we have several nginx servers running on
Ubuntu that serve files from a very large (several PBs) NFS mounted storage.

Usually the storage responds pretty fast, but occasionally it can be very
slow. Since we are using aio, when a file read operation runs slow, it's not
that bad - the specific HTTP request will just take a longer time to
complete. 
However, we have seen cases in which the file open itself can take a long
time to complete. Since the open is synchronous, when the open is slow all
active requests on the same nginx worker process are delayed. 
One way to mitigate the problem may be to increase the number of nginx
workers, to some number well above the number of CPU cores. This will make
each worker handle less requests and therefore less requests will be delayed
due to a slow open, but this solution sounds far from ideal.
Another possibility (that requires some development) may be to create a
thread that will perform the task of opening the file. The main thread will
wait on the completion of the open-thread asynchronously, and will be
available to handle other requests until the open completes. The child
thread can either be created per request (many requests probably won't even
need to open a file, thanks to the caching of open file handles), or
alternatively some fancier thread pooling mechanism could be developed.

I'd love to hear any thoughts / ideas on this subject

Thanks,

Eran

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,255847,255847#msg-255847