Weird timeouts, not sure if I've set the right threshholds

[Denis S. Filimonov]

On Friday 02 May 2008 19:56:45 Cliff Wells wrote:
> On Fri, 2008-05-02 at 16:44 -0400, Denis S. Filimonov wrote:
> > Hi guys,
> >
> > Can anyone explain the prejudice against NFS?
>
> I hadn't noticed any.  It's more a question of using the wrong tool for
> this job.  NFS is great if you need normal filesystem access for a
> workgroup.  It's the wrong tool for serving web content.
>
That's exactly the prejudice I was talking about :-)
I've yet to see any proof of this statement while my experience suggests that 
NFS suits serving web content very well (at least in some cases).

> > Specifically, why would additional proxy hop be faster than serving files
> > from NFS?
>
> Because I'm betting Nginx is faster than NFS.
>
> > I can see two points in favor of NFS:
> > - NFS client caches files while Nginx doesn't (yet)
>
> But you *must* use Nginx if you want your files available on the web (or
> more exactly, some HTTP server).  Caching is almost wholly irrelevant if
> you go through a layer that doesn't cache.  The only overhead being
> eliminated by NFS caching is that of NFS itself.
>
That's not true.
If I understand the proposed schemes correctly, they are
proxy <-(1)-> nginx/fcgi <-(2)-> NFS server
vs.
proxy <-(1)-> nginx <-(2)-> nginx/fcgi (over local fs)

Even though all traffic goes through link (1) anyway, we still want to reduce 
the latency in the link (2). And I maintain that the scheme with NFS can do 
better, partly because of caching.

> > - Nginx doesn't support keepalive connections to upstream, hence
> > additional latencies and traffic for TCP handshake/finalization. NFS
> > doesn't have this issue since it typically works over UDP.
>
> Except that you ignore all the other overhead NFS imposes at a different
> layer.  NFS may use a fairly efficient network protocol, however it's
> also a fairly complete virtual filesystem, and this emulation imposes
> significant overhead that a simple HTTP server doesn't have.
>
The only significant overhead NFS imposes is due to default consistency 
requirements which are excessive for most web applications. Relaxing those 
requirements goes a long way in improving NFS performance but for some reason 
it's usually neglected. Seriously, having disabled close-to-open consistency 
(totally irrelevant to a typical web application) I have observed *multifold* 
reduction of NFS packets and huge drop in latency.

> > I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS.
> > It works just fine, though it did take some NFS tuning.
>
> I'm sure apps would run just fine off NFS, since they are typically
> read-once, run-many.  Serving a few static files from NFS would probably
> be okay too, because the caching would help.  
I agree.

> Serving lots of static 
> files over NFS is probably going to suck badly unless your NFS server
> has tons of RAM for cache and even then I doubt it will perform very
> well.
True, it can perform poorly in extreme circumstances but I'm sure a web server 
would do even worse under the same conditions.

> At the end of the day, I suggest the OP *try* both methods and let the
> tests decide.  In the time it took to write this reply the OP could have
> tested both cases and this discussion would have been moot.
>
Agreed. Hope he posts the results, I did my research on this subject years 
ago. Much has changed, it'd be nice to have an update.

-- 
Denis.