No Route to Host: How Docker Hijacked My Work VPN
Imagine waking up to find out your work VPN no longer gives you access to the resources you had just yesterday. You try restarting and reconnecting, and everything seems fine - at first. You’re able to SSH into internal servers, but as soon as you start up your work environment, things begin to break…
Your SSH sessions start timing out. Internal services are suddenly unreachable. You try reconnecting to the same internal servers, only to see:
ssh: connect to host 172.x.x.x port 22: No route to host
I’ve had my fair share of VPN connection problems in the past when I was setting up a WireGuard server for my home lab:
- Incorrect port forwarding on the router
- Incorrect container port mapping
- Misconfigured
AllowedIPs - Unavailable (ie. shut off) internal services
But clearly I was able to access internal services with my work VPN briefly after booting, so what changed? Was someone/something actively breaking my VPN connection as I was using it?
After consulting a rubber ducky (read: LLM), it suggests to check the routing table with ip route show, after which I noticed something peculiar:
~ $ ip route show
default via 192.168.1.1 dev wlp0s20f3 proto dhcp src 192.168.1.x metric 600
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown # Docker default bridge
172.18.0.0/16 dev br-95858a7d8dfb proto kernel scope link src 172.18.0.1 # Docker network #01
..
172.28.0.0/16 dev br-d76afceb7da5 proto kernel scope link src 172.28.0.1 # Docker network #11
172.30.0.0/16 dev br-c9796d65c24d proto kernel scope link src 172.30.0.1 # Docker network #12
172.30.0.0/16 dev tun0 proto static scope link metric 50 # VPN
172.30.0.0/16 via 172.30.x.x dev tun0 proto static metric 50 # VPN
192.168.1.0/24 dev wlp0s20f3 proto kernel scope link src 192.168.1.x metric 600
192.168.1.1 dev wlp0s20f3 proto static scope link metric 50
Why are there multiple routes for 172.30.0.0/16? Isn’t that range used for Docker?
Oh… A Docker network has an overlapping subnet with my target VPN network, meaning any packet I intended to direct to the VPN network ends up going to the Docker network.
Fortunately, it is a relatively easy fix. One could modify the VPN subnet and reconfigure the whole network, or alternatively make Docker use different IP ranges for its networks by simply adding this into /etc/docker/daemon.json:
{
"bip": "10.10.0.1/24",
"default-address-pools": [
{
"base": "10.11.0.0/16",
"size": 24
}
]
}
And restart Docker with sudo systemctl restart docker. Be sure the new subnet does not conflict with other networks you are working with. eg. LAN, VPN.
You may also need to remove all existing Docker networks to properly resolve the subnet overlap by running docker network prune.
~ $ ip route show
default via 192.168.1.1 dev wlp0s20f3 proto dhcp src 192.168.1.x metric 600 # Wi-Fi
10.10.0.0/24 dev docker0 proto kernel scope link src 10.10.0.5 linkdown # Docker default bridge (new)
172.18.0.0/16 dev br-95858a7d8dfb proto kernel scope link src 172.18.0.1 # Docker network #01
..
172.28.0.0/16 dev br-d76afceb7da5 proto kernel scope link src 172.28.0.1 # Docker network #11
172.30.0.0/16 dev br-c9796d65c24d proto kernel scope link src 172.30.0.1 # Docker network #12
192.168.1.0/24 dev wlp0s20f3 proto kernel scope link src 192.168.1.x metric 600
~ $ docker network prune
WARNING! This will remove all custom networks not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Networks:
A-network
..
L-network
~ $ ip route show
default via 192.168.1.1 dev wlp0s20f3 proto dhcp src 192.168.1.x metric 600 # Wi-Fi
10.10.0.0/24 dev docker0 proto kernel scope link src 10.10.0.5 linkdown # Docker default bridge (new)
172.23.0.0/16 dev br-5c82828d7884 proto kernel scope link src 172.23.0.1 # Docker network #06
192.168.1.0/24 dev wlp0s20f3 proto kernel scope link src 192.168.1.x metric
If you have remaining Docker networks after pruning, you have to first remove the associated containers before removing the network.
~ $ docker network list
NETWORK ID NAME DRIVER SCOPE
7cbb42004af6 bridge bridge local
5c82828d7884 <project>_default bridge local
bdb30656a96d host host local
36768aff74cc none null local
~ $ docker network rm 5c82828d7884
Error response from daemon: error while removing network: network <project>_default has active endpoints (name:"<project>-pgadmin-1" id:"de8c3f53304e")
exit status 1
~ $ docker rm <project>-pgadmin-1
Error response from daemon: cannot remove container "<project>-pgadmin-1": container is running: stop the container before removing or force remove
~ $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
12fe07e41d9d dpage/pgadmin4 "/entrypoint.sh" 6 weeks ago Up 2 minutes .. <project>-pgadmin-1
~ $ docker stop <project>-pgadmin-1
<project>-pgadmin-1
~ $ docker rm <project>-pgadmin-1
<project>-pgadmin-1
~ $ docker network rm 5c82828d7884
5c82828d7884
You should now be able to have Docker containers and your VPN as the same time.
~ $ ip route show
default via 192.168.1.1 dev wlp0s20f3 proto dhcp src 192.168.1.x metric 600 # Wi-Fi
10.10.0.0/24 dev docker0 proto kernel scope link src 10.10.0.5 linkdown # Docker default bridge (new)
10.11.0.0/24 dev br-202f7b10f672 proto kernel scope link src 10.11.0.1 # Docker network #01 (new)
w.x.y.z via 192.168.1.1 dev wlp0s20f3 proto static metric 50
172.30.0.0/16 dev tun0 proto static scope link metric 50 # VPN
172.30.0.0/16 via 172.30.x.x dev tun0 proto static metric 50 # VPN
192.168.1.0/24 dev wlp0s20f3 proto kernel scope link src 192.168.1.x metric 600
192.168.1.1 dev wlp0s20f3 proto static scope link metric 50
tl:dr
- If your VPN suddenly stops working, and you’re using Docker, check for subnet overlaps with
ip route