Serving static file dynamically ...

[Marcus Clyne]

Hi Brice,

Firstly, I'm not talking about a full-blown database that's serving 
files, but a lightweight front to the files (as PBMS is).  PBMS is just 
an HTTP front to a storage engine for MySQL, and doesn't deal with SQL 
or anything like that.  It's really like serving the content directly 
out of a large file, in a pretty similar way to many caches do.

Ordinarily, the filesystem will be much faster than even the lightest of 
fronts to a db, but if you have milllions of files, then each file will 
have metadata associated with it (which takes up space - usually at 
least 4KB) and the filesystem has to cope with all the files, and many 
filesystems struggle when you start getting to large numbers of files, 
and it slows things down.

If you have billions of files, then you couldn't even serve them off a 
normal 32-bit fs, because you'd run out of inodes (I believe).

For thousands of files, or tens of thousands, you'd be fine, though, and 
the filesystem will definitely be quicker than PBMS.

When I did my tests with PBMS, I created 2M objects, and put them in the 
database with MySQL, then served them using PBMS.  My benchmarks showed 
that the number of req/s I could serve was similar to Apache 2 at best, 
and about 20% slower at worst (it depended on the index of the file).  
That might not seem great, but that was serving from 1 file vs 2M.  I 
tried creating files in a hierarchical structure, and after I got to 
around 400k (? I can't quite remember), my system almost completely 
stalled, so I stopped trying to add files.

In most scenarios, the filesystem will be quicker, but not always.

Cheers,

Marcus.


Brice Leroy wrote:
>
> On Mar 24, 2009, at 1:55 PM, Marcus Clyne wrote:
>
>> Michael Shadle wrote:
>>> not to mention that this is only useful if the OP is storing files in
>>> the database to begin with, I believe.
>>>
>>> if it's filesystem, then X-Accel-Redirect is the way to go.
>>>
>> Yes, if the filesystem is the storage mechanism, then I'd agree.  If 
>> you have a very large number of files, though, storing them in the DB 
>> can be more efficient than using the filesystem (depending on 
>> platform, file number, directory hierarchy etc).
>
> How can the filesystem can be slower than DB to serve huge video files 
> ? That's completely on the opposite to my culture ! Can you explain me 
> how this is possible ?
> My situation will be serving thousand(maybe more later) of different 
> big files(between 200MB and 4GB) to different users.
>
> Brice
>
>