Serving files from a slow NFS storage

[Steve Holdoway]

On Mon, 2014-12-29 at 15:52 -0500, erankor2 wrote:
> Hi all,
> 
> In our production environment, we have several nginx servers running on
> Ubuntu that serve files from a very large (several PBs) NFS mounted storage.
> 
> Usually the storage responds pretty fast, but occasionally it can be very
> slow. Since we are using aio, when a file read operation runs slow, it's not
> that bad - the specific HTTP request will just take a longer time to
> complete. 
> However, we have seen cases in which the file open itself can take a long
> time to complete. Since the open is synchronous, when the open is slow all
> active requests on the same nginx worker process are delayed. 
> One way to mitigate the problem may be to increase the number of nginx
> workers, to some number well above the number of CPU cores. This will make
> each worker handle less requests and therefore less requests will be delayed
> due to a slow open, but this solution sounds far from ideal.
> Another possibility (that requires some development) may be to create a
> thread that will perform the task of opening the file. The main thread will
> wait on the completion of the open-thread asynchronously, and will be
> available to handle other requests until the open completes. The child
> thread can either be created per request (many requests probably won't even
> need to open a file, thanks to the caching of open file handles), or
> alternatively some fancier thread pooling mechanism could be developed.
> 
> I'd love to hear any thoughts / ideas on this subject
> 
> Thanks,
> 
> Eran

As a generic SysAdmin, I would say the first place to start is to look
into installing something like cachefs, which will keep local copies of
the remote files, so once filled, the problem should go away.

There are also options to tune kernel and mount options that can help a
bit.

I would/do use this approach in preference to the (still too new IMO)
alternatives like GlusterFS.

In my experience, serving any volume of files over NFS will always be a
bottleneck.

Cheers,

Steve

-- 
Steve Holdoway BSc(Hons) MIITP
http://www.greengecko.co.nz
Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa