Python Http Server

Introduction

Quite often I find that I need to serve some files for viewing in a web browser. Most recently, I needed to do this with an in progress OpenAPI document as rendered by ReDoc.

All I needed was something that can serve static files. I really didn’t want to take the time to setup and configure something like Apache or Nginx. These are overkill for static files on a developer machine. All I needed was a simple development web server that can be started and stopped quickly and easily. Oh and something without configuration.

Simple Python http server

The first thing I went to is Python’s built in mini web server. It’s really nice and can set the port, the interface to listen on, and allows you to specify which directory to serve.

python -m http.server 8000 -b 127.0.0.1 -d files/

99% of the time this will get the job done and this command is super useful.

Over kill Python http(s) server

While directly invoking the Python http server works, it doesn’t support TLS connections. It would be really nice to have TLS support. This is my main motivation to writing something else and not using the build in one.

Considerations

First of all I wanted to keep similar functionality as the built in call.

Port configuration
Listen local or global
Specify a directory to serve

Additionally, I want to allow specifying a TLS cert and key to use for TLS connections. That said, I wanted to make it super easy to enable TLS. So, another feature I must have.

Generate and use a self signed cert

The built in one supports CGI request handling but I really don’t need that. I’m going to leave CGI out for now. I’ll leave adding it as an exercise for any anyone interested.

The code

import argparse
import functools
import http.server
import os
import random
import socket
import ssl
import sys

from tempfile import gettempdir

from self_sign_cert import gen_self_signed_cert

def parse_args():
    parser = argparse.ArgumentParser(description='Simple static content HTTP(S) server')
    parser.add_argument('--listen_global', '-g', action='store_true', help='Listen globally')
    parser.add_argument('--port', '-p', type=int, help='Port to listen on. Default 8080 or 8443 (TLS)')
    parser.add_argument('--directory', '-d', help='Directory to serve')
    parser.add_argument('--tls', '-t', action='store_true', help='Enable TLS. Will genreate and use self signed cert if cert and key are not specified')
    parser.add_argument('--tls_crt', '-c', help='TLS cert to use. Corresponding key must be specified. Implies --tls')
    parser.add_argument('--tls_key', '-k', help='TLS cert key to use. Corresponding cert must be specified. Implies --tls')

    return parser.parse_args()

def setup_tls(args):
    args.self_cert = False
    if not args.tls and not args.tls_crt and not args.tls_key:
        return True

    args.tls = True
    if args.tls_crt and args.tls_key:
        return True

    args.self_cert = True
    tls_crt, tls_key = gen_self_signed_cert()
    if not tls_crt:
        print('Failed to generate self signed certificate')
        return False

    args.tls_crt = os.path.join(gettempdir(), 'serv_self_cert_%d.crt' % random.randrange(10000))
    args.tls_key = os.path.join(gettempdir(), 'serv_self_cert_%d.key' % random.randrange(10000))
    with open(args.tls_crt, 'wb') as f:
        f.write(tls_crt)

    with open(args.tls_key, 'wb') as f:
        f.write(tls_key)

    return True

def setup_port(args):
    if args.port:
        return

    args.port = 8080
    if args.tls:
        args.port = 8443

def serv(httpd):
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    except Exception as e:
        print(e)

def main():
    args = parse_args()

    if not setup_tls(args):
        print('Failed to setup TLS')
        return 2
    setup_port(args)

    listen = 'localhost'
    if args.listen_global:
        listen = ''

    print('Listening {how} on port {port} (TLS {have_tls})'.format(
        how='globally' if args.listen_global else 'locally',
        port=args.port,
        have_tls='enabled' if args.tls else 'disabled'))

    handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=args.directory)
    with http.server.ThreadingHTTPServer((listen, args.port), handler) as httpd:
        if args.tls:
            context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
            context.load_cert_chain(certfile=args.tls_crt, keyfile=args.tls_key)
            if args.self_cert:
                os.remove(args.tls_crt)
                os.remove(args.tls_key)
            with context.wrap_socket(httpd.socket, server_side=True) as sock:
                httpd.socket = sock
                serv(httpd)
        else:
            serv(httpd)

    return 0

if __name__ == '__main__':
    sys.exit(main())

The code explained

I’m going skip talking about the parse_args function because it should be self explanatory. The same goes for setup_port which determines which default port to use if one wasn’t specified.

setup_tls will kick off generating a self signed cert if necessary. It uses the Self signed cert python module for the heavy lifting.

    args.tls_crt = os.path.join(gettempdir(), 'serv_self_cert_%d.crt' % random.randrange(10000))
    args.tls_key = os.path.join(gettempdir(), 'serv_self_cert_%d.key' % random.randrange(10000))
    with open(args.tls_crt, 'wb') as f:
        f.write(tls_crt)

    with open(args.tls_key, 'wb') as f:
        f.write(tls_key)

Once the module has generated the cert, we need to write the cert and key to disk. We’ll store the filenames for these in the args object. We’ll ensure args.self_cert is set appropriately because we’ll need it later. We’re using a random file name in the user’s temp directory instead of something static like ‘self.cert/.key’ so we don’t accidentally overwrite anything that might have been legitimately created.

I’ll explain more about why we’re writing the cert and key to disk when when we get to the place they’re being used.

Starting the server

Now let’s skip the setup code in main and go right to the good stuff.

        handler = partial(http.server.SimpleHTTPRequestHandler, directory=args.directory)

We’re using the default request handler because we don’t need anything fancy, just simple file serving. The class takes a directory argument which allows us to set the directory where files will be severed from. However, when we pass off the handler we pass the class not an initialized object.

We need to use functools.partial to wrap the class with the argument so when it’s invoked directory will be set to what was passed into the script. If a directory wasn’t specified args.directory will be None which is fine. That’s the default value for SimpleHTTPRequestHandler if the argument wasn’t passed in.

    with http.server.ThreadingHTTPServer((listen, args.port), handler) as httpd:

One thing to note is the ThreadingHTTPServer class is being used. This will prevent the server from blocking on a single request. Useful if the files being served are massive and multiple people are trying to access them.

I’ve seen people use socketserver.TCPServer in place of [Threading]HTTPServer. If you’re going to use socketserver I high recommend using ThreadingTCPServer.

You could do something like this instead of the ThreadingHTTPServer line above:

    with socketserver.ThreadingTCPServer((listen, args.port), handler) as httpd:

It ultimately doesn’t matter which one is used. HTTPServer is a sub classes of TCPServer. Inspecting the code for HTTPServer it sets allow_reuse_address and a stores a few bits of info that can be pulled back out later. Otherwise HTTPServer doesn’t do anything that makes it absolutely required.

That said, it’s still better to use the right class for what we’re doing. Even though we’ll never access the cached data but allow_reuse_address is good to have set.

        if args.tls:
            context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
            context.load_cert_chain(certfile=args.tls_crt, keyfile=args.tls_key)
            if args.self_cert:
                os.remove(args.tls_crt)
                os.remove(args.tls_key)
            with context.wrap_socket(httpd.socket, server_side=True) as sock:
                httpd.socket = sock
                serve(httpd)
        else:
            serv(httpd)

If we’re using TLS we need to create a context and load the cert and key files. There is no way to load a cert into the context from memory. Using a temporary file is unfortunately necessary. Which is why we wrote the self signed certs (if requested) to disk earlier.

If we’re using self signed certs we can delete them once they’ve been loaded. Once loaded they’re been read into memory and the files on disk won’t be accessed again.

Once loaded, we wrap and replace the httpd’s socket with one that is a server side TLS connection. Everything will be routed through this socket which will encrypt and decrypt the data.

Finally, whether we’re using TLS or not, we start serving files by calling the serv function.

def serv(httpd):
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    except Exception as e:
        print(e)

This gives is just a wrapper so we don’t have to duplicate the try..except block.

The httpd.serve_forever() call will not return. Instead we wait for a KeyboardInterrupt exception, Ctrl+c, and capture it. This will breaks out of the serve_forever() function. We’re treating this event as a graceful shutdown. This way doesn’t show a nasty message in the console.

We’re capturing all other exceptions and printing them out. We want to do this so the app won’t exit due to an uncaught exception.

Capturing exceptions here allows all the cleanup from the with statements to run. We want the network sockets to be cleanly closed so if we start again the ports will be accessible for binding.

Conclusion

I’m very surprised that in memory TLS contexts aren’t built into Python. I’m also surprised that the http.server module doesn’t natively support TLS. While it’s pretty easy to add TLS by wrapping the socket, it would be nice to have the python -m http.server ... call support it too.

Overall this slightly over engineered development web server was an interesting little project.

Introduction#

Simple Python http server#

Over kill Python http(s) server#

Considerations#

The code#

The code explained#

Starting the server#

Conclusion#