Private tor network on kubernetes

I recently came across someone running a private tor network with docker and immediately decided I'd have to do similar but in Kubernetes. I also followed another useful blog post about this subject.

This seemed like a great opportunity to learn about the inner workings of the tor network and flex my kubernetes muscles. Here are some of the tricky bits I encountered for anyone trying to do something similar.

Testing mode

To get a chance of running our own tor network we must enable TestingTorNetwork, this tweaks a number of settings, such as not totally banning private IPs and reducing delays in voting.

Directory Authorities

A fundamental part of a tor network is the Directory Authority. When connecting to the network the client will connect to one of these to find out a list of relays to further connect to. These are hardcoded in to the tor source code*.

Fortunately there are config options we can use to override these values (DirAuthority). This config needs to have not just the address but the fingerprint of the authority (so we know we can trust it).

So from initial research it sounded like all we need to do was:

• Generate certificates and signatures for 3 directory authorities
• Create directory authorities (configured with their certificates)
• Configure 10 relays to talk to directory authorities
• Create 10 relays

ConfigMaps and directories

When trying to get the directory authorities running I had issues poking the certificates in. tor is kind of specific about the structure it expects (an id and keys dir). Because ConfigMaps don't do subdirectories (ref) I ended up using a flat structure in the ConfigMap and using my docker-entrypoint.sh to set up symlinks to achieve the desired structure.

DirtAuthority address

For the DirAuthority line we're expected to use an IP address (mailing list discussion). From a kubernetes point of view this is a bit annoying. Using a Service we can easily know the hostname upfront but an IP is more tricky. We could set the ClusterIP but that leaves config bound to a particular cluster setup.

The solution is not so bad - when we generate each DirAuthority line we just make sure we've already created the Services and use their IP addresses. We can use jsonpath to get the IP:

kubectl get svc da1 -o 'jsonpath={.spec.clusterIP}'

Works, but it makes our setup a bit less elegant - we have to generate config files based upon the state of the kubernetes cluster.

Relay address

On start, if not provided with one, tor will search for an IP address to use. As we don't know our pod IP up front, this sounds ideal. Unfortunately, tor will not pick a private IP address (ref) unless explicitly given that address.

This means we have to have add another trick - a docker-entrypoint.sh to append an Address line to our torrc with the pod's IP . Again, not awful, but not pretty.

Running it

With all these pieces in place I was able to successfully run a private tor network. I can route internet traffic through it (and see it hopping between servers) and scale the number of relays up and down.

Conclusion

These are the main problems I had to overcome to get tor running inside kubernetes. The resulting set of scripts is on github: andrewmichaelsmith/private-tor-network-kube.

I'm reasonably happy with my final product, it produces a fully operational tor network. There is a certain amount of bash scaffolding which I'm not a huge fan of. It might be interesting to try and do this project again but as an Operator.

** I'm lying here to keep things simple. There are also Fallback mirrors that tor will connect to first. These are also hardcoded in to the tor source code.