GeoIP Question - Speed & efficiency
chrissavery at gmail.com
Sat Aug 16 12:05:56 MSD 2008
Thanks Maxim. Sounds cool, fast but perhaps a bit of a memory hog. For
loading time I would think the way to improve that is to compile a
binary representation on disk that can be loaded as a "pre-made tree"
into memory so that no insertion scan need be done. Or pre-sort the data
to insert with minimum searches.
Anyway, I may write a small script to see if I can amalgamate countries
into big blocks as that would help both speed and memory.
I gather that being at the http level config this means it is "always
on". I could see it being useful to be able to put it in a location
specifier so that only certain requests go through the lookup. For
example, I've no need for static images to get country codes but my
index page would be great as I would set a "best choice" value for
serving the user for all further requests in the session. It sounds like
it doesn't use much cpu time but I expect to be serving vasts amounts
of small thumbnails so reducing cycles on that is always a good thing.
(25 thumbs/page/user ad nauseum photo app).
Cheers, for excellent info.
Maxim Dounin wrote:
> On Sat, Aug 16, 2008 at 07:43:45AM +0700, Chris Savery wrote:
>> I am thinking of using the GeoIP module with input from the maxmind
>> database converted with the perl script as described through the link
>> on the nginx site.
>> I'm curious if the country-ip pairs are managed efficiently so that
>> the lookup/conversion is very fast or not? That is, does the module
>> do something like sort the list and then use a binary tree to
>> quickly locate the country? Is the whole thing loaded in memory?
> Geo module builds in-memory radix tree when loading configs. This is
> the same data structure as used in routing, and lookups are really fast.
>> This country database is quite huge and if this process happens on
>> every hit or even on only a selected entry page then it could be
>> very slow. Does anyone here have experience with this?
> The only inconvinience of using really large geobases is config
> reading time. My currently takes about 30 seconds to load - but
> that's for more than 30 Mb of data, and not only countries.
>> For my purposes I only really need to detect continents for deciding
>> if visitors should pull from one of a few server locations. So
>> presumably it may be possible to combine many countries into larger
>> blocks so that there are fewer steps in the lookup. Any input on how
>> speedy or efficient this has shown to be would be super helpful here.
> Aggregating blocks is good thinks to do if you don't need detailed
> information, but you'll hardly notice any difference.
> Maxim Dounin
More information about the nginx