Monitoring update ipsets

update-ipsets maintains the site iplists.firehol.org by itself.

With update-ipsets, you can have the same site, with your IP lists, running on your web server! Having this site with your data allows you to monitor the ipsets as they are updated on your server.

The site is a single static web page, but it needs some data. There is no database server involved. All the data exist as static JSON, CSV and XML documents.

Here are the data needed:

• the evolution data (size over time) for each ipset, in CSV (example)
• the changesets data (changes over time) for each ipset, in CSV (example)
• the country maps, for each ipset and each geolocation provider, in JSON (example)
• the age of IPs currently listed and the retention policy of past IPs (a double histogram), for each ipset, in JSON (example)
• the overlaps between all ipsets for each ipset, in JSON (example)
• a description of each ipset, in JSON (example)
• a list of all available ipsets, in JSON (example)
• a sitemap, in XML (example)

update-ipsets can provide all these!

---

How to enable it?

You have to configure a directory and download two files from iplists.firehol.org.

WEB_DIR

WEB_DIR is the directory where the web files will be saved. It should be under the directory structure of your web server, so that they can be served.

If you run update-ipsets as root:

# create /etc/firehol/update-ipsets.conf
cat >>/etc/firehol/update-ipsets.conf <<EOF
WEB_DIR="/var/www/localhost/htdocs/iplists"
WEB_OWNER="apache:apache"
EOF

# create the web directory
mkdir /var/www/localhost/htdocs/iplists

If you run update-ipsets as any other user and you use the USERDIR module of apache:

# create /etc/firehol/update-ipsets.conf
echo "WEB_DIR=${HOME}/public_html/iplists" >>~/.update-ipsets/firehol/update-ipsets.conf

# create the web directory
mkdir -p ${HOME}/public_html/iplists

---

Download site static files

Step into WEB_DIR and do these:

# get the index.html page
curl -o index.html 'http://iplists.firehol.org/index.html'

# get typeahead.css
curl -o typeahead.css 'http://iplists.firehol.org/typeahead.css'

These two files are the whole of it. The entire site.

---

Enable geolocation providers

Run this to enable the geolocation providers (used for the country maps):

update-ipsets enable geolite2_country ipdeny_country ip2location_country

---

Enable the firehol_level1 IP list

Run this to enable the firehol_level1 IP lists (used as the default ipset for the site):

update-ipsets enable bambenek_c2 cta_cryptowall dshield feodo fullbogons palevo spamhaus_drop spamhaus_edrop sslbl zeus_badips firehol_level1

---

Run it!

The last step is to run update-ipsets to populate all information in these directories. It will do it automatically on every run from now on, but the first time we have to tell it to rebuild the web files.

We do this with the -r option.

update-ipsets -r

That's all! Now go to your web server and check it. You have the site too!

You can keep your update-ipsets command running as before (e.g. in cron). You will notice that update-ipsets now makes a few more passes on the downloaded ipsets to calculate all the data needed for the site.

---

OPTIONAL: Custom content per IP list

The site supports custom information per IP list to be shown the About section.

To use this feature, you create simple HTML fragment files (without any header, footer, etc) in WEB_DIR with the content you like.

The content for firehol_level1 is ${WEB_DIR}/firehol_level1.html. Similarly for all other IP lists.

---

OPTIONAL: Publish it!

If you want to publish it at github-pages, update-ipsets can help in this area too!

Step into WEB_DIR and do:

git init

Configure your git to automatically (without password) push to the place you need to (google it, this is done by having ssh keys on your github account and switching the git url to use ssh) and then edit your update-ipsets configuration and add this:

PUSH_TO_GIT=1

Now, update-ipsets will:

• download the ipsets from their maintainers
• convert them / process them
• update the ipsets that are active in your kernel
• calculate all the data for the site
• commit the updated web files to git
• and push the git repo for you, to publish the site

This is how I run the site too.

---

Goodies...

The site you will host, will share the discussions with iplists.firehol.org, so everyone installing update-ipsets is connected to each other via the site comments engine.