Serving static file dynamically ...

Brice Leroy bbrriiccee at
Wed Mar 25 21:49:11 MSK 2009

Thank you Marcus. I never think about those kind of FS issues. files  
should not exceed hundred thousand in the worse case scenario so I  
think file system should be the better solution for me.

On Mar 25, 2009, at 11:07 AM, Marcus Clyne wrote:

> Hi Brice,
> Firstly, I'm not talking about a full-blown database that's serving  
> files, but a lightweight front to the files (as PBMS is).  PBMS is  
> just an HTTP front to a storage engine for MySQL, and doesn't deal  
> with SQL or anything like that.  It's really like serving the  
> content directly out of a large file, in a pretty similar way to  
> many caches do.
> Ordinarily, the filesystem will be much faster than even the  
> lightest of fronts to a db, but if you have milllions of files, then  
> each file will have metadata associated with it (which takes up  
> space - usually at least 4KB) and the filesystem has to cope with  
> all the files, and many filesystems struggle when you start getting  
> to large numbers of files, and it slows things down.
> If you have billions of files, then you couldn't even serve them off  
> a normal 32-bit fs, because you'd run out of inodes (I believe).
> For thousands of files, or tens of thousands, you'd be fine, though,  
> and the filesystem will definitely be quicker than PBMS.
> When I did my tests with PBMS, I created 2M objects, and put them in  
> the database with MySQL, then served them using PBMS.  My benchmarks  
> showed that the number of req/s I could serve was similar to Apache  
> 2 at best, and about 20% slower at worst (it depended on the index  
> of the file).  That might not seem great, but that was serving from  
> 1 file vs 2M.  I tried creating files in a hierarchical structure,  
> and after I got to around 400k (? I can't quite remember), my system  
> almost completely stalled, so I stopped trying to add files.
> In most scenarios, the filesystem will be quicker, but not always.
> Cheers,
> Marcus.
> Brice Leroy wrote:
>> On Mar 24, 2009, at 1:55 PM, Marcus Clyne wrote:
>>> Michael Shadle wrote:
>>>> not to mention that this is only useful if the OP is storing  
>>>> files in
>>>> the database to begin with, I believe.
>>>> if it's filesystem, then X-Accel-Redirect is the way to go.
>>> Yes, if the filesystem is the storage mechanism, then I'd agree.   
>>> If you have a very large number of files, though, storing them in  
>>> the DB can be more efficient than using the filesystem (depending  
>>> on platform, file number, directory hierarchy etc).
>> How can the filesystem can be slower than DB to serve huge video  
>> files ? That's completely on the opposite to my culture ! Can you  
>> explain me how this is possible ?
>> My situation will be serving thousand(maybe more later) of  
>> different big files(between 200MB and 4GB) to different users.
>> Brice

More information about the nginx mailing list