Update Geoip Database Weekly.

[EidolonHost]

So. Here's a script for updating your GeoIP database.

#!/bin/bash

# cd to directory where the MaxMind database is to be downloaded.
if ! cd /home/example/public_html/billing/system/; then
echo "Can't find work directory" >&2
exit 1
fi

# Remove existing files so we start off with a clean set of updated data from Maxmind.

rm -f GeoLite*
rm -f md5sum.txt

# Download databases and if applicable, their md5s.

curl -L https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz |gunzip |tee GeoLite2-City.mmdb|md5sum |awk '{print $1}' > GeoLite2-City.mmdb.md5-loc$
curl -L http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.md5 > GeoLite2-City.mmdb.md5-remote.md5

# Localized copies of our working data.
local=`cat GeoLite2-City.mmdb.md5-local.md5`
remote=`cat GeoLite2-City.mmdb.md5-remote.md5`

echo L=$local R=$remote

# DO THE THING! ie, compare the files!
if [ "$local" != "$remote" ]; then
mail -s "Results of GeoLite Updates" example@domain.com <<< "md5sum for GeoLite2-City failed. Please check the md5sum. File may possibly be corrupted."
else
fi

 

 

Works for me so far.

 

My cronjob looks like this:

* 20 * * 6 /home/example/geoip.sh

Runs every Saturday at 8PM.

 

Haven't figured out how to automatically overwrite... oh wait.

[Blesta Addons]

So. Here's a script for updating your GeoIP database.

#!/bin/bash
echo "GeoLiteCity update beginning at `date`" > /home/example/logs/geoip.txt
cd /home/example/public_html/exampledomain.com/billing/system/
rm -rf GeoLiteCity.dat
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz
echo "GeoLiteCity update completed at `date`" > /home/example/logs/geoip.txt
 

Works for me so far.

My cronjob looks like this:

* 20 * * 6 /home/example/geoip.sh

Runs every Saturday at 8PM.

Haven't figured out how to automatically overwrite... oh wait.

Thanks . Usefull .

I will add this to my admin tools plugin

[Michael]

Thanks . Usefull .

I will add this to my admin tools plugin

 

What's it supposed to do?

[EidolonHost]

What's it supposed to do?

It's supposed to update your GeoIP database. Put it in a shell script, set it executable with chmod +x geoip.sh or something, however you want to have the file named. Edit paths according to your billing system install.

Add to cronjob and decide how often you want it to run. You're set, then.

[Michael]

It's supposed to update your GeoIP database. Put it in a shell script, set it executable with chmod +x geoip.sh or something, however you want to have the file named. Edit paths according to your billing system install.

Add to cronjob and decide how often you want it to run. You're set, then.

 

but why does it need to be updated? the countries don't change nor do the IPs?

[EidolonHost]

but why does it need to be updated? the countries don't change nor do the IPs?

 

Because MaxMind said themselves that they do update their GeoIP database content every so often. Says so when you go to download it, at any rate from their site.

 

Besides, better to know you have a fresh copy. I think they update their stuff weekly or once a month? I'm not sure but WHT has threads about this.

[activa]

but why does it need to be updated? the countries don't change nor do the IPs?

 

No ; the database is chaging from time to time .

[Adam]

So. Here's a script for updating your GeoIP database.

#!/bin/bash
echo "GeoLiteCity update beginning at `date`" > /home/example/logs/geoip.txt
cd /home/example/public_html/exampledomain.com/billing/system/
rm -rf GeoLiteCity.dat
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz
echo "GeoLiteCity update completed at `date`" > /home/example/logs/geoip.txt

If you run this in a cronjob, no need to log each message since you log the whole script (email or pipe redirection). I would also make the suggestion you use https as opposed to http for the transfer (geolite supports this).

You could get a small performance boost if you used named pipes and curl. Something like this would work:

 

curl -L https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz | gunzip > GeoLiteCity.dat

Works for me so far.

 

My cronjob looks like this:

* 20 * * 6 /home/example/geoip.sh

Runs every Saturday at 8PM.

 

Haven't figured out how to automatically overwrite... oh wait.

That is pretty agressive. The Maxmind website says the database is updated on the first Tuesday of the month. Taking into account the various number of timezones and not knowing when on Tuesday the database is updated, lets run this on Wednesdays instead.

 

0 9 * * 3 [ `date +\%d` -le 7 ] && curl -L https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz | gunzip > /srv/uploads/system/GeoLiteCity.dat

 

Runs on the first Wednesday of every month at 9AM server time.

 

but why does it need to be updated? the countries don't change nor do the IPs?

It is more then just countries, cities are accounted for. With IPv4 shortage, v4 addresses are being reallocated, reassigned more fequrently then ever before.

Note to Blesta Devs: Should switch to the new GeoLite2 database format. The current one is legacy and will be deprecated at some point. The new GeoLite2 also includes md5 sums which can be added to the script above.

http://dev.maxmind.com/geoip/geoip2/geolite2/

-Adam

[EidolonHost]

If you run this in a cronjob, no need to log each message since you log the whole script (email or pipe redirection). I would also make the suggestion you use https as opposed to http for the transfer (geolite supports this).

You could get a small performance boost if you used named pipes and curl. Something like this would work:

 

curl -L https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz | gunzip > GeoLiteCity.dat

That is pretty agressive. The Maxmind website says the database is updated on the first Tuesday of the month. Taking into account the various number of timezones and not knowing when on Tuesday the database is updated, lets run this on Wednesdays instead.

 

0 9 * * 3 [ `date +\%d` -le 7 ] && curl -L https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz | gunzip > /srv/uploads/system/GeoLiteCity.dat

 

Runs on the first Wednesday of every month at 9AM server time.

 

It is more then just countries, cities are accounted for. With IPv4 shortage, v4 addresses are being reallocated, reassigned more fequrently then ever before.

Note to Blesta Devs: Should switch to the new GeoLite2 database format. The current one is legacy and will be deprecated at some point. The new GeoLite2 also includes md5 sums which can be added to the script above.

http://dev.maxmind.com/geoip/geoip2/geolite2/

-Adam

 

 

Ah, thanks! I'll update the script then, with your suggestions. I'll also update to GeoLite2 when Blesta shifts to supporting GeoLite2. I suppose I could have the script download GeoLite 2 in preparation for Blesta switching to using GeoLite 2. In fact... I'll do that. Check the OP, the script will be updated shortly.

 

Edit: This bit here: 0 9 * * 3 [ `date +\%d` -le 7 ], is that to be in the cronjob or the script itself?

 

Edit2: Suggestions on MD5 verifying signature? Should it e-mail you on mismatch of the signature? Also, does the script look good now?

[Adam]

Ah, thanks! I'll update the script then, with your suggestions. I'll also update to GeoLite2 when Blesta shifts to supporting GeoLite2. I suppose I could have the script download GeoLite 2 in preparation for Blesta switching to using GeoLite 2. In fact... I'll do that. Check the OP, the script will be updated shortly.

Script does not work -- did you try it out in your shell? By default curl streams the data to standard output. You are not capturing the output. Every line after that is incorrect. Your logic on checking for MD5 is backwords -- if you ask me. Read the man page for the command md5sum (man md5sum).

 

Edit: This bit here: 0 9 * * 3 [ `date +\%d` -le 7 ], is that to be in the cronjob or the script itself?

Yes, that goes in your crontab file (/var/spool/cron/) Do not forget the &&. The following does everything you need:

 

0 9 * * 3 [ `date +\%d` -le 7 ] && cd /path/to/uploads/system/ && curl -L https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz | gunzip > /srv/uploads/system/GeoLiteCity.dat

Edit2: Suggestions on MD5 verifying signature? Should it e-mail you on mismatch of the signature? Also, does the script look good now?

No, the script is not good. Please test before posting. Depending on what philosophy you subscribe too, I am of the idea that I should only be email when something bad happens. Since you plan on making this into a cronjob, cron can email you the output of a script. So only print to standard output (or error) in your bash script when it fails. Get ride of the verbose printing would be what I suggest.

-Adam

[EidolonHost]

Script does not work -- did you try it out in your shell? By default curl streams the data to standard output. You are not capturing the output. Every line after that is incorrect. Your logic on checking for MD5 is backwords -- if you ask me. Read the man page for the command md5sum (man md5sum).

Yes, that goes in your crontab file (/var/spool/cron/) Do not forget the &&. The following does everything you need:

 

0 9 * * 3 [ `date +\%d` -le 7 ] && curl -L https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz | gunzip > /srv/uploads/system/GeoLiteCity.dat

No, the script is not good. Please test before posting. Depending on what philosophy you subscribe too, I am of the idea that I should only be email when something bad happens. Since you plan on making this into a cronjob, cron can email you the output of a script. So only print to standard output (or error) in your bash script when it fails. Get ride of the verbose printing would be what I suggest.

-Adam

 

 

Ack. Sorry about that, I'll need to shellcheck more carefully.

 

Edit: OK, I think I've got it... it works for me now that I've corrected the issues. Can I have someone else test and confirm? It should work now. I'm not the greatest at bash scripting. :c

 

Edit2: Er... looks like it works fine but not quite in the way I intended. I just got an e-mail from the server about the md5sum results... it was supposed to e-mail me on failure of the md5sum test. Looks like I need to fine-tune it a bit more.

 

Edit3: This is a bit frustrating, trying to fine-tune the script to e-mail on md5sum error failure.

[Cody]

Note to Blesta Devs: Should switch to the new GeoLite2 database format. The current one is legacy and will be deprecated at some point. The new GeoLite2 also includes md5 sums which can be added to the script above.

 

Yup, it's on todo list.

[EidolonHost]

 

 

Yup, it's on todo list.

 

 

Nice. When you guys do that, will you guys be including the functionality to update the databases as mandated by MaxMind? Or is this script the preferred way to do it?

[Cody]

Nice. When you guys do that, will you guys be including the functionality to update the databases as mandated by MaxMind? Or is this script the preferred way to do it?

 

I think that's sort of a gray area, in terms of licensing. We can't distribute MaxMind DB with Blesta, and some users use a shared MaxMind DB. But we'll evaluate when we get there. If nothing else, a plugin could be created to auto update MaxMind DB.

[EidolonHost]

I think that's sort of a gray area, in terms of licensing. We can't distribute MaxMind DB with Blesta, and some users use a shared MaxMind DB. But we'll evaluate when we get there. If nothing else, a plugin could be created to auto update MaxMind DB.

 

I don't think that'll be an issue. I meant having the ability to have Blesta download the DB for you on an automated basis. As you said, you just can't distribute the DB with Blesta but you can set it up so that it'll download it for you on an automated basis.

[Cody]

I don't think that'll be an issue. I meant having the ability to have Blesta download the DB for you on an automated basis. As you said, you just can't distribute the DB with Blesta but you can set it up so that it'll download it for you on an automated basis.

 

Yeah, that's the gray area, and I'm not a lawyer, but my interpretation is that if it is completely optional and not enabled by default (e.g. the download does not happen automatically unless and until someone checks a box), then it should be ok.

[EidolonHost]

Yeah, that's the gray area, and I'm not a lawyer, but my interpretation is that if it is completely optional and not enabled by default (e.g. the download does not happen automatically unless and until someone checks a box), then it should be ok.

 

Ahhh, I gotcha now. Herp. Yeah, I believe you're right on this. Hopefully that's all it'll take.

[EidolonHost]

Script in OP is now updated to work properly, for the most part.

 

The only remaining issue now is that you still get an e-mail regarding the failure of the md5sums. But if you go to actually check the md5sums for the GeoLite2-City database, you find they're the exact same.

 

I'm not sure why this is, just yet... but by and large, you should be able to just throw this into a script, set it up in cron to fire when MaxMind updates their stuff each month.

[EidolonHost]

Alright, another friend of mine has noted that I wasn't actually properly comparing. This final script should properly compare, now. It also now correctly e-mails on bad md5sums.