Serve *only* from cache for particular user-agents

[Ajay k]

you can use  http://nginx.org/en/docs/http/ngx_http_map_module.html

Ex:

map $http_user_agent $mobile {
     ~* msn-bot  'http://devnull';

    default 'http://productionrupal';

}



Thanks,

Ajay K



On Fri, Feb 21, 2014 at 10:16 PM, rge3 <nginx-forum at nginx.us> wrote:

> Maxim Dounin Wrote:
> -------------------------------------------------------
> > Hello!
> >
> > On Fri, Feb 21, 2014 at 10:25:58AM -0500, rge3 wrote:
> >
> > > I havne't found any ideas for this and thought I might ask here.  We
> > have a
> > > fairly straightforward proxy_cache setup with a proxy_pass backend.
> > We
> > > cache documents for different lengths of time or go the backend for
> > what's
> > > missing.  My problem is we're getting overrun with bot and spider
> > requests.
> > > MSN in particular started hitting us exceptionally hard yesterday
> > and
> > > started bringing our backend servers down.  Because they're crawling
> > the
> > > site from end to end our cache is missing a lot of those pages and
> > nginx has
> > > to pass the request on through.
> > >
> > > I'm looking for a way to match on User-Agent and say that if it
> > matches
> > > certain bots to *only* serve out of proxy_cache.  Ideally I'd like
> > the logic
> > > to be:  if it's in the cache, serve it.  If it's not, then return
> > some 4xx
> > > error.  But in the case of those user-agents, *don't* go to the
> > backend.
> > > Only give them cache.  My first thought was something like...
> > >
> > > if ($http_user_agent ~* msn-bot) {
> > >       proxy_pass http://devnull;
> > >  }
> > >
> > > by making a bogus backend.  But in nginx 1.4.3 (that's what we're
> > running) I
> > > get
> > > nginx: [emerg] "proxy_pass" directive is not allowed here
> > >
> > > Does anyone have another idea?
> >
> > The message suggests you are trying to write the snippet above at
> > server{} level.  Moving things into a location should do the
> > trick.
> >
> > Please make sure to read http://wiki.nginx.org/IfIsEvil though.
>
> That seems to have done it!  With a location block I now have...
>
>                        location / {
>                                 proxy_cache_valid  200 301 302  30m;
>
>                                 if ($http_user_agent ~* msn-bot) {
>                                         proxy_pass http://devnull;
>                                 }
>
>                                 if ($http_user_agent !~* msn-bot) {
>                                         proxy_pass http://productionrupal;
>                                 }
>                         }
>
> That seems to work perfectly.  But is it a safe use of "if"?  Is there a
> safer way to do it without an if?
>
> Thanks for the help!
> -R
>
> Posted at Nginx Forum:
> http://forum.nginx.org/read.php?2,247837,247845#msg-247845
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



-- 
Thanks,
Ajay K
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20140221/84c16060/attachment-0001.html>