Introduction

A few months ago I setup a private DNS resolver with ad blocking. At that time I decided to manually update the block file whenever I installed server updates. This works, but it very quickly became tedious. I’m not sure why I initially thought it wouldn’t. I’ve since decided to automate the process.

Block File Script

The previous script I wrote would download a hosts file based block list and convert it into a format that Unbound can load. It sets DNS entires for both IPv4 and IPv6. The whole process I need to go though includes generating the file plus checking the it’s valid, and putting it into the Unbound configuration directory. Checking and installing the file are easily things the script can handle.

/usr/local/bin/adblock_loader.py

#!/usr/bin/env python

import argparse
import subprocess
import sys
import tempfile
import urllib.request

# https://github.com/StevenBlack/hosts
URL = 'https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
OUTPUT_FILENAME = '/etc/unbound/unbound.conf.d/adblock.conf'

def parse_args():
    parser = argparse.ArgumentParser(description='Parse hosts file into unbound config file')
    parser.add_argument('--url', '-u', default=URL, help='URL of hosts file to download from the internet. Default is "{URL}"'.format(URL=URL))
    parser.add_argument('--output', '-o', default=OUTPUT_FILENAME, help='File to write the parsed hosted data to in unbound configuration file format. Default is "{OUTPUT_FILENAME}"'.format(OUTPUT_FILENAME=OUTPUT_FILENAME))
    parser.add_argument('--no-verify', action='store_true', help='Don\'t call unbound-checkconf to verify the generated file before writing')
    return parser.parse_args()

def dl_blocklist(url):
    data = ''
    with urllib.request.urlopen(url) as f:
        if f.code != 200:
            raise Exception('Could not download')
        data = f.read().decode('utf-8')
    return data

def parse_blocklist(data):
    # Use a set to ensure we don't get any duplicate entries
    domains = set()

    for line in data.splitlines():
        # Lines that don't start with an ip address are other things
        # like comments and we can ignore those
        if not line.startswith('0.0.0.0'):
            continue
        _, _, domain = line.partition(' ')
        # A few entires in the hosts file have end of line comments which
        # we need to remove
        domain = domain.partition('#')[0].strip()
        # Sanity check to verify it's a good domain to add to the list
        if not domain or domain == '0.0.0.0' or domain.lower() == 'localhost':
            continue
        domains.add(domain)

    # The block file has nearly 200k domains. Check that we have at last someting
    # reasonable to ensure we didn't get a badly truncated file
    if len(domains) < 17000:
        raise Exception('Too few domains to be valid')

    # Write the unbound config
    buf = [ 'server:' ]
    for domain in sorted(domains):
        buf.append('\tlocal-zone: "{domain}" redirect'.format(domain=domain))
        buf.append('\tlocal-data: "{domain} A 0.0.0.0"'.format(domain=domain))
        buf.append('\tlocal-data: "{domain} AAAA ::"'.format(domain=domain))

    buf.append('')
    return '\n'.join(buf)

def save_unbound_block(output_filename, data, verify=True):
    # Write unbound config data to a temporary file so we can run
    # unbound-checkconf on it and verify it's proper. If there is a
    # bug in the file that could prevent unbound from starting.
    if verify:
        with tempfile.NamedTemporaryFile() as tf:
            tf.file.write(data.encode())
            # check parameter will raise an exception if process return non-zero
            subprocess.run(['unbound-checkconf', tf.name], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT, check=True)
    # Write the conf data to the real config file
    with open(output_filename, 'wb') as f:
        f.write(data.encode())

def main():
    args = parse_args()

    try:
        data = dl_blocklist(args.url)
        data = parse_blocklist(data)
        save_unbound_block(args.output, data, not args.no_verify)
    except Exception as e:
        print('Failure: {e}'.format(e=e))
        return 1

    return 0

if __name__ == '__main__':
    sys.exit(main())

The script was pretty much overhauled and made more generic so it can be reused in the future without necessarily needing to modify the code. I figured that if I’m going to enhance the script, I should structure it as a “proper” application instead of it being one off use.

I still have the block list I’m using as a default if not specified but since I wanted to enhance this script, I also make it an option so this can be overridden. I also made an option to specify what file to write the unbound configuration to instead of dumping it to stdout.

Real error handling is present in this version and exceptions are checked. Any error will result in the process exiting with a return code of 1 to indicate a failure. I don’t have proper logging and instead only write the failure reason stdout. Since this is such as simple script I decided if there is an error that isn’t transient, running the script manually is enough to determine why failures are occurring.

The script calls unbound-checkconf in order to verify the generated config won’t break unbound. This uses a named temporary file because unbound-checkconf will only read data from a file. I want to check the file before writing it to the output location, not after. I want to prevent saving the file if there is a problem with it. Also, my Mac doesn’t have Unbound installed so there is an option to skip this check. This is very handy for testing locally.

Auto Running Every Week

At this point I can generate the updated block list by running one command. The next step is to have systemd run the script automatically on a regular interval.

Service File

/etc/systemd/system/adblock_loader.service

[Unit]
Description=Update Unbound ad block list
After=unbound.service

[Service]
Type=oneshot
ExecStart=python /usr/local/bin/adblock_loader.py
ExecStartPost=systemctl restart unbound

[Install]
WantedBy=unbound.service

The service file does the heavy lifting of calling the script. However, running the script isn’t enough. Unbound needs to load the new configuration file in order to use the updated block list. The key to make this happen is the line ExecStartPost=systemctl restart unbound which restarts Unbound after the update script runs.

Timer File

Service files only instruct systemd what to run. An additional timer file is needed to tell systemd when to run a given service. systemd will run the service with the same base name as the timer.

/etc/systemd/system/adblock_loader.timer

[Unit]
Description=Update ad block list every week

[Timer]
OnCalendar=weekly

[Install]
WantedBy=timers.target

This is a basic weekly timer.

Now that the service and timer files have been created, we can use the following to enable the timer to run on boot.

sudo systemctl enable adblock_loader.timer

Also, the following is needed to start the timer otherwise, you’ll need to reboot to start it.

sudo systemctl start adblock_loader.timer

Conclusion

I shouldn’t have been lazy and I should have done this when I first setup Unbound. This is going to save me a lot of hassle and streamline updates. Moral of the story, don’t be lazy so you can be lazier later.