Introduction
A few months ago I setup a private DNS resolver with ad blocking. At that time I decided to manually update the block file whenever I installed server updates. This works, but it very quickly became tedious. I’m not sure why I initially thought it wouldn’t. I’ve since decided to automate the process.
Block File Script
The previous script I wrote would download a hosts file based block list and convert it into a format that Unbound can load. It sets DNS entires for both IPv4 and IPv6. The whole process I need to go though includes generating the file plus checking the it’s valid, and putting it into the Unbound configuration directory. Checking and installing the file are easily things the script can handle.
/usr/local/bin/adblock_loader.py
#!/usr/bin/env python
import argparse
import subprocess
import sys
import tempfile
import urllib.request
# https://github.com/StevenBlack/hosts
URL = 'https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
OUTPUT_FILENAME = '/etc/unbound/unbound.conf.d/adblock.conf'
def parse_args():
parser = argparse.ArgumentParser(description='Parse hosts file into unbound config file')
parser.add_argument('--url', '-u', default=URL, help='URL of hosts file to download from the internet. Default is "{URL}"'.format(URL=URL))
parser.add_argument('--output', '-o', default=OUTPUT_FILENAME, help='File to write the parsed hosted data to in unbound configuration file format. Default is "{OUTPUT_FILENAME}"'.format(OUTPUT_FILENAME=OUTPUT_FILENAME))
parser.add_argument('--no-verify', action='store_true', help='Don\'t call unbound-checkconf to verify the generated file before writing')
return parser.parse_args()
def dl_blocklist(url):
data = ''
with urllib.request.urlopen(url) as f:
if f.code != 200:
raise Exception('Could not download')
data = f.read().decode('utf-8')
return data
def parse_blocklist(data):
# Use a set to ensure we don't get any duplicate entries
domains = set()
for line in data.splitlines():
# Lines that don't start with an ip address are other things
# like comments and we can ignore those
if not line.startswith('0.0.0.0'):
continue
_, _, domain = line.partition(' ')
# A few entires in the hosts file have end of line comments which
# we need to remove
domain = domain.partition('#')[0].strip()
# Sanity check to verify it's a good domain to add to the list
if not domain or domain == '0.0.0.0' or domain.lower() == 'localhost':
continue
domains.add(domain)
# The block file has nearly 200k domains. Check that we have at last someting
# reasonable to ensure we didn't get a badly truncated file
if len(domains) < 17000:
raise Exception('Too few domains to be valid')
# Write the unbound config
buf = [ 'server:' ]
for domain in sorted(domains):
buf.append('\tlocal-zone: "{domain}" redirect'.format(domain=domain))
buf.append('\tlocal-data: "{domain} A 0.0.0.0"'.format(domain=domain))
buf.append('\tlocal-data: "{domain} AAAA ::"'.format(domain=domain))
buf.append('')
return '\n'.join(buf)
def save_unbound_block(output_filename, data, verify=True):
# Write unbound config data to a temporary file so we can run
# unbound-checkconf on it and verify it's proper. If there is a
# bug in the file that could prevent unbound from starting.
if verify:
with tempfile.NamedTemporaryFile() as tf:
tf.file.write(data.encode())
# check parameter will raise an exception if process return non-zero
subprocess.run(['unbound-checkconf', tf.name], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT, check=True)
# Write the conf data to the real config file
with open(output_filename, 'wb') as f:
f.write(data.encode())
def main():
args = parse_args()
try:
data = dl_blocklist(args.url)
data = parse_blocklist(data)
save_unbound_block(args.output, data, not args.no_verify)
except Exception as e:
print('Failure: {e}'.format(e=e))
return 1
return 0
if __name__ == '__main__':
sys.exit(main())
The script was pretty much overhauled and made more generic so it can be reused in the future without necessarily needing to modify the code. I figured that if I’m going to enhance the script, I should structure it as a “proper” application instead of it being one off use.
I still have the block list I’m using as a default if not specified but since I wanted to enhance this script,
I also make it an option so this can be overridden. I also made an option to specify what file to write the
unbound configuration to instead of dumping it to stdout
.
Real error handling is present in this version and exceptions are checked. Any error will result in the process
exiting with a return code of 1
to indicate a failure. I don’t have proper logging and instead only write
the failure reason stdout
. Since this is such as simple script I decided if there is an error that
isn’t transient, running the script manually is enough to determine why failures are occurring.
The script calls unbound-checkconf
in order to verify the generated config
won’t break unbound. This uses a named temporary file because
unbound-checkconf
will only read data from a file. I want to check the file
before writing it to the output location, not after. I want to prevent saving the file if
there is a problem with it. Also, my Mac doesn’t have Unbound installed so there is
an option to skip this check. This is very handy for testing locally.
Auto Running Every Week
At this point I can generate the updated block list by running one command. The next step is to have systemd run the script automatically on a regular interval.
Service File
/etc/systemd/system/adblock_loader.service
[Unit]
Description=Update Unbound ad block list
After=unbound.service
[Service]
Type=oneshot
ExecStart=python /usr/local/bin/adblock_loader.py
ExecStartPost=systemctl restart unbound
[Install]
WantedBy=unbound.service
The service file does the heavy lifting of calling the script. However, running the script isn’t enough.
Unbound needs to load the new configuration file in order to use the updated block list. The key
to make this happen is the line ExecStartPost=systemctl restart unbound
which restarts Unbound after
the update script runs.
Timer File
Service files only instruct systemd what to run. An additional timer file is needed to tell systemd when to run a given service. systemd will run the service with the same base name as the timer.
/etc/systemd/system/adblock_loader.timer
[Unit]
Description=Update ad block list every week
[Timer]
OnCalendar=weekly
[Install]
WantedBy=timers.target
This is a basic weekly timer.
Now that the service and timer files have been created, we can use the following to enable the timer to run on boot.
sudo systemctl enable adblock_loader.timer
Also, the following is needed to start the timer otherwise, you’ll need to reboot to start it.
sudo systemctl start adblock_loader.timer
Conclusion
I shouldn’t have been lazy and I should have done this when I first setup Unbound. This is going to save me a lot of hassle and streamline updates. Moral of the story, don’t be lazy so you can be lazier later.