GeoIP Module

[Igor Sysoev]

On Wed, Dec 01, 2010 at 05:05:56AM -0500, Nam wrote:

> Hey Guys, I have run into a problem with the geo module. I have set up a
> geo list containing a LARGE list of IPs which we need to have
> "whitelisted" for getting through to the upstream. These IPs are for
> search engines. Currently we have the list set up via the following
> way...
> 
> geo $remote_addr $search  {
>         default          0;
>         include          geoip-search.conf;
> }
> 
> The geoip-search.conf file contains a the list of IPs in the following
> format...
> 
> 114.111.36.26/32  search;
> 114.111.36.28/32  search;
> 114.111.36.29/32  search;
> 114.111.36.30/32  search;
> 114.111.36.31/32  search;
> 114.111.36.32/32  search;
> 119.63.193.100/32  search;
> 119.63.193.101/32  search;
> 119.63.193.102/32  search;
> 119.63.193.103/32  search;
> 
> Then inside of the configurations, we do the following... which was
> based on recommendations from Igor...
> 
> if ( $search = search ) {
>             proxy_pass              http://LB_HTTP_UPSTREAM;
>             break;
> }
> 
> Then under that we also have some stuff for security which checks for a
> cookie and stuff serving them a different page if no cookie is present.
> We want the search engine IPs to be able to make it through to the
> upstream, but it appears that this is no longer occurring. We had no
> problems in the past... Perhaps it is due to something in 0.8.53 as we
> had upgraded to that a while ago, and then after a while we got
> complaints of google bots not getting through. Our list contains about
> 40,000 lines which covers well over 100,000 IPs. Anyone have any ideas
> on what could be causing this?

It should work. Could you create debug log of the request ?
BTW, you may compress geo file using this script:

------
#!/usr/bin/perl -w

use Net::CIDR::Lite;
use strict;
use warnings;

my %cidr;

while (<>) {
    if (/^(\S+)\s+(\S+);/) {
        my($net, $region) = ($1, $2);
        if (!defined $cidr{$region}) {
            $cidr{$region} = Net::CIDR::Lite->new;
        }
        $cidr{$region}->add($net);
    }
}

for my $region (sort { $a cmp $b } keys %cidr) {
    print((join " $region;\n", $cidr{$region}->list), " $region;\n");
}
------

For example, the 10 above lines are compressed to just 4:
------
114.111.36.26/32 search;
114.111.36.28/30 search;
114.111.36.32/32 search;
119.63.193.100/30 search;
------

Also, if you use an original client $remote_addr, then this

-geo $remote_addr $search  {
+geo $search  {
         default          0;
         include          geoip-search.conf;
}

will work slightly faster.

Also, you may avoid "if":

geo $search {
    default  usual_upstream;
    ...      search_upstream;
    ...      search_upstream;
    ...      search_upstream;
    ...      search_upstream;
    ...
}

upstream search_upstream {
    ...
}

upstream usual_upstream {
    ...
}

server {
    location / {
        proxy_pass  http://$search;
    }


-- 
Igor Sysoev
http://sysoev.ru/en/