mod_mogilefs for nginx was (A hardware question)

[mike]

I would also contribute a few hundred bucks if it was developed and
ran an open source license (so we could use and other people could
enhance, etc!)


On 4/29/08, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
>
>        If some enterprising C hacker wanted to work on this Engine Yard
> would pay a bounty to make it happen as long as it is released open source.
> Contact me if you are interested in working on this project.
>
> Cheers-
> - Ezra Zygmuntowicz
> -- Founder & Software Architect
> -- ezra at engineyard.com
> -- EngineYard.com
>
>
> On Apr 28, 2008, at 12:27 PM, mike wrote:
> > I think it would be something worthwhile myself.
> >
> > Just needs parameters - the tracker database info, and the domain
> > right? Then you just use the domain + key to get the file. Maybe
> > domain isn't even required.
> >
> > However, wouldn't you need some sort of logic to determine how to
> > create the key?
> >
> > Some people use an md5 of the URI I guess, so it would be something like
> >
> > foo.com/media/abc72849482bc8398af98effd9g843
> >
> > /media/ would be mapped to something like
> >
> > option #1 would be nginx do the mysql connection itself.
> > option #2 would be nginx just connects to the tracker (this is
> > probably best) since it's a simple socket connection
> >
> > location /media/ {
> > mogilefs on;
> > mogilefs_domain images;
> >
> > option #1
> > mogilefs_tracker_username mogilefs;
> > mogilefs_tracker_password foo;
> > mogilefs_tracker_host 192.168.1.203;
> > mogilefs_tracker_database mogilefs;
> >
> > option #2
> > mogilefs_tracker_host 192.168.1.202:7001;
> >
> > and perhaps something like mogilefs_key_style hash; ? or something...
> > (used to determine how the keys are actually setup)
> > }
> >
> > I think the biggest missing piece of the puzzle is how to determine
> > the key structure. Otherwise I am sure it's quite simple, and it uses
> > dav/HTTP to retrieve the files anyway, so the C code needed would be:
> >
> > option #1:
> > 1) mysql client
> > 2) something which can do efficient HTTP GETs with offset support
> > (probably already in nginx)
> >
> > option #2:
> > 1) something that can do a simple socket connect, ask the tracker for
> > info, and then pass it on to step 2 (probably already in nginx)
> > 2) something which can do efficient HTTP GETs with offset support
> > (probably already in nginx)
> >
> > If I knew anything worthwhile in C I would try it myself. I am sure
> > someone even with basic C skills could hack it up really quick. It
> > just needs one socket to talk to the tracker, then depending on the
> > reply, open a second connection to grab the file...
> >
> >
> > On 4/28/08, kingler <kingler at 72pines.com> wrote:
> >
> > > I know there are MogileFS client libraries for Perl/PHP/Ruby, but if
> > > someone can code a C library for MogileFS and it is then possible to
> > > make a Nginx module to support. I guess this will be more efficient
> > > than going down the Perl/PHP/Ruby route.
> > >
> > > Does it make sense at all? Would the benefits outweigh the extra
> > > efforts to code the nginx module?
> > >
> > > -Liang
> > >
> > > kingler from 72pines
> > >
> > > On Mon, Apr 28, 2008 at 11:46 AM, mike <mike503 at gmail.com> wrote:
> > >
> > > > I am actually thinking about making a MogileFS passthrough
> > > >
> > > > (I posted an email about it, looks like someone in Japan figured it
> out)
> > > >
> > > > Would allow for minimal PHP/perl/python/whatever language to be
> > > > involved, just enough to look up where a file is. Then pass that back
> > > > to nginx with "file is on $server" and then use X-Accel-Redirect to
> > > > bounce to that server...
> > > >
> > > > Haven't done it yet, but I think it would work quite amazing!
> > > >
> > > >
> > > >
> > > > On 4/28/08, Igor Sysoev <is at rambler-co.ru> wrote:
> > > >
> > > > > On Mon, Apr 28, 2008 at 12:16:59PM +0200, Chavelle Vincent wrote:
> > > > >
> > > > >
> > > > > > Igor Sysoev wrote:
> > > > > >
> > > > > >
> > > > > > > We use the proxying in this case instead of NFS:
> > > > > > >
> > > > > > > client   >      nginx (1)   >      nginx
> > > > > > >
> > > > > > > On nginx (1) it's better to set "proxy_max_temp_file_size 0" for
> > > > > > > the proxied location.
> > > > > > >
> > > > > >
> > > > > > You recommend to use proxying for mass file serving. I am in this
> case,
> > > > > > I use nginx and a specific module that I have coded. (to simplify
> > > > > > things, it control access files and redirect client to
> > > > > > ngx_http_internal_redirect)
> > > > > >
> > > > > > I have think to use NFS for serve the files. I don't understand
> how to
> > > > > > use proxying instead of that.
> > > > > >
> > > > >
> > > > > Set up nginx on NFS server and proxy to it instead of getting files
> via NFS:
> > > > >
> > > > >      location /files/ {
> > > > >           proxy_pass                 http://nfs_server_nginx;
> > > > >           proxy_max_temp_file_size   0;
> > > > >      }
> > > > >
> > > > >
> > > > > --
> > > > > Igor Sysoev
> > > > > http://sysoev.ru/en/
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>
>
>
>