mod_mogilefs for nginx was (A hardware question)

[Ezra Zygmuntowicz]

	If some enterprising C hacker wanted to work on this Engine Yard  
would pay a bounty to make it happen as long as it is released open  
source. Contact me if you are interested in working on this project.

Cheers-
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra at engineyard.com
-- EngineYard.com


On Apr 28, 2008, at 12:27 PM, mike wrote:
> I think it would be something worthwhile myself.
>
> Just needs parameters - the tracker database info, and the domain
> right? Then you just use the domain + key to get the file. Maybe
> domain isn't even required.
>
> However, wouldn't you need some sort of logic to determine how to
> create the key?
>
> Some people use an md5 of the URI I guess, so it would be something  
> like
>
> foo.com/media/abc72849482bc8398af98effd9g843
>
> /media/ would be mapped to something like
>
> option #1 would be nginx do the mysql connection itself.
> option #2 would be nginx just connects to the tracker (this is
> probably best) since it's a simple socket connection
>
> location /media/ {
> mogilefs on;
> mogilefs_domain images;
>
> option #1
> mogilefs_tracker_username mogilefs;
> mogilefs_tracker_password foo;
> mogilefs_tracker_host 192.168.1.203;
> mogilefs_tracker_database mogilefs;
>
> option #2
> mogilefs_tracker_host 192.168.1.202:7001;
>
> and perhaps something like mogilefs_key_style hash; ? or something...
> (used to determine how the keys are actually setup)
> }
>
> I think the biggest missing piece of the puzzle is how to determine
> the key structure. Otherwise I am sure it's quite simple, and it uses
> dav/HTTP to retrieve the files anyway, so the C code needed would be:
>
> option #1:
> 1) mysql client
> 2) something which can do efficient HTTP GETs with offset support
> (probably already in nginx)
>
> option #2:
> 1) something that can do a simple socket connect, ask the tracker for
> info, and then pass it on to step 2 (probably already in nginx)
> 2) something which can do efficient HTTP GETs with offset support
> (probably already in nginx)
>
> If I knew anything worthwhile in C I would try it myself. I am sure
> someone even with basic C skills could hack it up really quick. It
> just needs one socket to talk to the tracker, then depending on the
> reply, open a second connection to grab the file...
>
>
> On 4/28/08, kingler <kingler at 72pines.com> wrote:
>> I know there are MogileFS client libraries for Perl/PHP/Ruby, but if
>> someone can code a C library for MogileFS and it is then possible to
>> make a Nginx module to support. I guess this will be more efficient
>> than going down the Perl/PHP/Ruby route.
>>
>> Does it make sense at all? Would the benefits outweigh the extra
>> efforts to code the nginx module?
>>
>> -Liang
>>
>> kingler from 72pines
>>
>> On Mon, Apr 28, 2008 at 11:46 AM, mike <mike503 at gmail.com> wrote:
>>> I am actually thinking about making a MogileFS passthrough
>>>
>>> (I posted an email about it, looks like someone in Japan figured  
>>> it out)
>>>
>>> Would allow for minimal PHP/perl/python/whatever language to be
>>> involved, just enough to look up where a file is. Then pass that  
>>> back
>>> to nginx with "file is on $server" and then use X-Accel-Redirect to
>>> bounce to that server...
>>>
>>> Haven't done it yet, but I think it would work quite amazing!
>>>
>>>
>>>
>>> On 4/28/08, Igor Sysoev <is at rambler-co.ru> wrote:
>>>> On Mon, Apr 28, 2008 at 12:16:59PM +0200, Chavelle Vincent wrote:
>>>>
>>>>> Igor Sysoev wrote:
>>>>>
>>>>>> We use the proxying in this case instead of NFS:
>>>>>>
>>>>>> client   >      nginx (1)   >      nginx
>>>>>>
>>>>>> On nginx (1) it's better to set "proxy_max_temp_file_size 0" for
>>>>>> the proxied location.
>>>>>
>>>>> You recommend to use proxying for mass file serving. I am in  
>>>>> this case,
>>>>> I use nginx and a specific module that I have coded. (to simplify
>>>>> things, it control access files and redirect client to
>>>>> ngx_http_internal_redirect)
>>>>>
>>>>> I have think to use NFS for serve the files. I don't understand  
>>>>> how to
>>>>> use proxying instead of that.
>>>>
>>>> Set up nginx on NFS server and proxy to it instead of getting  
>>>> files via NFS:
>>>>
>>>>       location /files/ {
>>>>            proxy_pass                 http://nfs_server_nginx;
>>>>            proxy_max_temp_file_size   0;
>>>>       }
>>>>
>>>>
>>>> --
>>>> Igor Sysoev
>>>> http://sysoev.ru/en/
>>>>
>>>>
>>>
>>>
>>
>>
>