Hostile Private Networks in the Cloud

Update 2018-07: As of July 2018 DigitalOcean has begun rolling out true private networking that is isolated per-account. This is a great step forward, and I sincerely hope other VPS providers follow suit. Until then, this post is still relevant for basically every other provider.

A while ago I deployed a blog using DigitalOcean's infrastructure. I used separate droplets for the database and web tier, and used the "private IP" feature to only allow MySQL traffic over that interface. I noticed the private network seemed larger than necessary for just hosting my own stuff, so I did a little digging.

DigitalOcean's "private networking" is quite literally the opposite of private. Your droplet's private IP will be in a shared subnet with every other customer's if they end up on the same network as you. Do not think for a minute you can "trust the LAN" or anything like that; I'm sure if I scanned deeper I could find dozens of unauthenticated Redis and database servers people inadvertently have opened to this not-so-private network.

Unfortunately, this isn't just DigitalOcean's problem. I ran a couple tests from one of my instances on Linode and was shocked to find their Private IP feature both assigns the address to the same interface as the public IP, and shares the subnet with many other customers. These are just the two providers I have used; I would hazard to guess this is a common practice among the smaller non-Openstack cloud providers.

To demonstrate this issue, I spun up a droplet in DigitalOcean's TOR-1 region. The private interface has the entire 10.137.0.0/16 network routed, and I decided to try doing a basic scan to enumerate the entire subnet.

Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-03 17:06 UTC

Nmap scan report for 10.137.0.4
Host is up (0.0037s latency).
MAC Address: F6:BF:AB:FF:ED:AA (Unknown)
Nmap scan report for 10.137.0.5
Host is up (0.0037s latency).
MAC Address: AE:FC:E8:AD:78:7F (Unknown)
[...]
Nmap scan report for 10.137.255.252
Host is up (0.0082s latency).
MAC Address: 1A:73:75:CA:44:1A (Unknown)
Nmap done: 65536 IP addresses (5588 hosts up) scanned in 229.23 seconds

I have exactly one other droplet in that region, and yet my brand new droplet can access over five thousand other customers' servers on a supposedly "private" network. This is not only misleading, but the messaging makes it sound like you can trust it; I don't even want to try and knock on ports to see how many open-to-the-world services are running.

I won't pretend to be all-knowing and condescending: nothing about running infrastructure is easy. However, having separate networks per customer would make dramatic improvements to the service's scalability and security, and at the very least having one VXLAN per customer seems like a painfully obvious good practice.

To verify this, I spun up a droplet in two completely separate DigitalOcean accounts (I added my personal CC to my work account to test this).

Right off-the-bat I was able to confirm that both droplets had private IP addresses in the same subnet range. Then, on one droplet I started nc -l 64321, and on the second I tried to nmap the private IP of my second droplet. I was successful.

Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-03 21:57 UTC
Nmap scan report for 10.132.146.7
Host is up (0.0056s latency).
PORT      STATE SERVICE
64321/tcp open  unknown
MAC Address: 6A:1E:6E:F2:92:59 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 0.19 seconds

And here's an obligatory screenshot. The left-side droplet is running under my personal account, the right-side droplet is under a completely different account (but the droplets are in the same region).

Luckily my further attempts to tcpdump the traffic between the two hosts were unsuccessful. But one could still do a fair amount of damage if someone entrusted a service like Redis or NFS to allow connections from the entire network of the private interface.

I take great issue with these features being labelled "Private Networking" as there is nothing private about them whatsoever. There's nothing stopping me from spinning up a new server in the same region as you and suddenly I am on the same "private" network as you.

By calling this feature "private", it implies privacy. This may mislead people into setting up unsecured services because they assume the network they are exposing them on is only for their consumption. As long as "hostile LAN" practices are followed, having a public private network should be fine, but I would strongly urge these cloud providers to implement at the very least customer isolation, if not full OpenStack-style SDN.

I hope this post serves as a warning to anyone running services on the "underdog" cloud providers: your private network is absolutely not private, and you may have an unforseen attack vector left wide open.