Serving static file dynamically ...

[Brice Leroy]

Thank you Marcus. I never think about those kind of FS issues. files  
should not exceed hundred thousand in the worse case scenario so I  
think file system should be the better solution for me.

On Mar 25, 2009, at 11:07 AM, Marcus Clyne wrote:

> Hi Brice,
>
> Firstly, I'm not talking about a full-blown database that's serving  
> files, but a lightweight front to the files (as PBMS is).  PBMS is  
> just an HTTP front to a storage engine for MySQL, and doesn't deal  
> with SQL or anything like that.  It's really like serving the  
> content directly out of a large file, in a pretty similar way to  
> many caches do.
>
> Ordinarily, the filesystem will be much faster than even the  
> lightest of fronts to a db, but if you have milllions of files, then  
> each file will have metadata associated with it (which takes up  
> space - usually at least 4KB) and the filesystem has to cope with  
> all the files, and many filesystems struggle when you start getting  
> to large numbers of files, and it slows things down.
>
> If you have billions of files, then you couldn't even serve them off  
> a normal 32-bit fs, because you'd run out of inodes (I believe).
>
> For thousands of files, or tens of thousands, you'd be fine, though,  
> and the filesystem will definitely be quicker than PBMS.
>
> When I did my tests with PBMS, I created 2M objects, and put them in  
> the database with MySQL, then served them using PBMS.  My benchmarks  
> showed that the number of req/s I could serve was similar to Apache  
> 2 at best, and about 20% slower at worst (it depended on the index  
> of the file).  That might not seem great, but that was serving from  
> 1 file vs 2M.  I tried creating files in a hierarchical structure,  
> and after I got to around 400k (? I can't quite remember), my system  
> almost completely stalled, so I stopped trying to add files.
>
> In most scenarios, the filesystem will be quicker, but not always.
>
> Cheers,
>
> Marcus.
>
>
> Brice Leroy wrote:
>>
>> On Mar 24, 2009, at 1:55 PM, Marcus Clyne wrote:
>>
>>> Michael Shadle wrote:
>>>> not to mention that this is only useful if the OP is storing  
>>>> files in
>>>> the database to begin with, I believe.
>>>>
>>>> if it's filesystem, then X-Accel-Redirect is the way to go.
>>>>
>>> Yes, if the filesystem is the storage mechanism, then I'd agree.   
>>> If you have a very large number of files, though, storing them in  
>>> the DB can be more efficient than using the filesystem (depending  
>>> on platform, file number, directory hierarchy etc).
>>
>> How can the filesystem can be slower than DB to serve huge video  
>> files ? That's completely on the opposite to my culture ! Can you  
>> explain me how this is possible ?
>> My situation will be serving thousand(maybe more later) of  
>> different big files(between 200MB and 4GB) to different users.
>>
>> Brice
>>
>>
>
>