The Internet Needs A Red Pill

Internet routing is far too fragile

I consider the internet to be one of humankind’s greatest inventions. Computer networking began as a solution to the perceived problem that a single missile from the Soviet Union could take out the entire United States phone system and render the US unable to coordinate a response. Nobody thought about that before the Soviet Union launched Sputnik and suddenly the entire world was aware that those damn Ruskies were floating around above us doing god-knows-what devious deeds.

In the early ’60s, two things happened in rapid succession. I’ve read almost everything there is read about the origins of the internet and I’m still not sure what the true story is - feel free to search for a version you like. Regardless of the details, around this time, researchers proposed a “galactic network” of computers that could talk to each other. A few years later, another floated the idea of a “packet-switched” network. Packet switching was a necessary technology because it breaks messages into small packets before sending them across the network and the receiving system re-combines them into the original message. The critical part of packet switching is that it allows each little packet to travel to its destination by different routes if needed, thus ensuring that a single strike taking down a portion of the “galactic network” would not prevent packets from arriving at their destination via another route.

Although the capacity, size, and underlying technologies have improved over the years, the same basic premise has not: interconnected computers sending discreet small packets over the best routes. It is the “best routes” thing that I will be talking about in this article.

A primer on BGP

Despite popular phrases like “log in” and “surf”, we do not “go onto” the internet. Rather, we use the internet to transfer data into our computers. When you log into a website, you don’t “go to” that web site’s login page; you pull a copy of that login page into your computer, fill in the details, and then you send the whole thing back to the remote web server. Every bit of that page is loaded from some computer connected to the internet somewhere, and when you click the login button, all that stuff gets sent back to some other remote computer connected to the internet. But, how does that happen? How does your computer’s request for that login page find the correct destination computer in the mass of trillions of computers all connected to the internet? The short answer is that the remote computer has a unique IP address, as does your computer, so they can both be uniquely identified despite the overwhelming number of computers comprising the internet.

But that is not the whole answer. Just because my computer knows that has IP address doesn’t mean my computer knows where that computer is physically connected to the internet. My computer has absolutely no clue where that IP address, in fact. It just issues a request for the GMail login page to that IP address and then sits back to see what happens.

The internet takes care of figuring out where is plugged in so we do not have to worry about it. The movie The Matrix had a nice solution to this: the red pill. The blue pill would put Neo back to sleep in the virtual world the machines created and he would remain blissfully ignorant to the existence of the Matrix. The red pill, however, would pull him out of the Matrix by acting as a locator beacon for Neo’s physical body in the fields, allowing Morpheus’ crew to find him and physically unplug him. Sadly, we can’t ask GMail to take a red pill for us, so we have to rely on the intelligent routing capabilities of the internet; primarily, the Border Gateway Protocol (BPG).

The internet is comprised of millions of little systems named Autonomous Systems (AS), which is just a paper record of who owns and controls what IP ranges. Any organization that wishes to own IP addresses must first obtain their own AS from one of the Regional Internet Registries and then obtain IP addresses to associate with their AS. All major internet players all have their own AS and they tell the rest of the internet what IP address they control by announcing those IPs using BGP. So, when you type into your browser bar, the following process occurs:

  1. Your computer asks the Domain Name System (DNS) for the IP address of which is

  2. Your computer then splits your request into many different packets and sends them out that blue wire, or your wifi connection, onto the internet, with stated as the destination.

  3. Your packets go through many internet devices like switches and routers on the way to its destination. It is those devices that have knowledge of where IP addresses are physically plugged into the internet, much like Neo. They speak the Border Gateway Protocol to other devices and share information such as “I know where is and it will take 5 hops to get there”. Another switch may say “I know where and it will take 3 hops to get there”. In which case, most routing devices will choose to send the packet along the shorter route of fewer hops so it gets there faster.

  4. The internet is huge and ever-changing. No device can hold the entire IP allocation table of the whole internet. Therefore, a lot of devices do not know where is located. In those cases, the packet is forwarded “up” through larger and larger ASs until some device says “I know where is and ….”

  5. This process happens for every packet. Your login username and password can amount to hundreds of individual packets and each one of those could, potentially, take a different route to GMail. In practice, these routing devices cache BPG results for short periods, so subsequent packets should arrive quicker than the initial one and usually by the same route unless the routing information has changed, such as sudden damage along the way.

  6. Your computer is blissfully unaware of all this. Once it is done sending packets, it forgets about the whole affair and goes back to doing whatever computers do when they’re idle.

  7. Eventually, reply packets start coming back to your computer from the GMail server following the same routing process as before. Your computer reassembles the packets and eventually, you are rewarded by the GMail inbox loading in your browser. Hooray.

  8. Now you click an email to read it and the whole thing starts all over again.

BPG hijacking

BGP is an absolutely critical part of how the internet functions and when it breaks, it can create very large outages that sometimes can be very hard to fix. Strangely, despite its importance to the basic functioning of the internet, BGP has no security built into it. Remember those ASes that major internet players have? They can announce any IP address range they want and once those announcements are copied out to the routers on the internet, then some of the traffic destined for those IPs will be routed to that AS’s servers even if that AS does not legitimately own the IPs it is announcing. This is called BGP Hijacking and it’s a real thing.

Stay with me here: BPG Hijacking means I can configure my switch to announce Gmail’s IP range and then your GMail requests will start coming to me instead of Google.

This has huge security implications. If I can route all your banking, government, and medical information to my servers then not only am I bringing those legitimate services offline, but I may also find myself the recipient of lots of private, juicy information that I may be able to use for evil. This may sound like rare circumstance, but BGP Hijacking is more common than we’d like. Sometimes it happens by accident, but other times a nation-state actor like China routes all of Google’s traffic into China which doesn’t sound like an accident to anyone.

Preventing BPG hijacks

Recall that AS stands for Autonomous System. The keyword here being Autonomous. Nobody can tell me what IP ranges to announce from my AS. I can be as nefarious and incompetent as I like, and the internet will feel it as traffic goes haywire. The best protection we have for this at the moment is called Resource Private Key Infrastructure (RPKI). This is a system whereby I, as the legitimate owner of an IP address space, can sign my IP announcements with my private encryption key which proves that I am the legitimate owner announcing that IP range. If anyone else decides to announce my IP range, those announcements will not be signed because that person (hopefully) does not have my private key. That is a good start, but signing my announcements does nothing by itself.

Although we consider the internet to be a virtual world, it actually is very physical. Internet traffic travels through physical cables and ends up at physical servers. The companies that own those cables are called transit providers and they are a critical component of BPG security. In order for RPKI to work, every transit provider has to do two things:

  1. implement checks on BGP announcements to see if the AS announcing the range has signed those announcements, and

  2. confirm that announcements for that IP range should be signed.

This is where the BPG Hijack prevention mechanisms start to break down.

Not all AS owners set up RPKI which means transit providers cannot assume that every IP range announcement should be signed. Further, implementing RPKI costs transit providers something. They have to deploy extra software to manage RPKI and, honestly, there’s very little incentive for them to do so. Transit providers are content-neutral; to use a metaphor: they are just the highway - they have nothing to do with the cars driving on it. Consequently, many transit providers have not implemented RPKI which means it is not very effective, even if an AS owner has done their part and set up signed announcements (called ROA -Routing Origin Authorization records) for its IP ranges.

The good news is that we’re getting there. Like in any industry, there are big and small players. Most of the big transit providers which comprise a good chunk of the main internet routes, called the backbone, support RPKI. This has the effect of lessening the impact of BPG hijack attempts because those illegitimate announcements can’t spread too far before they hit a major transit provider supporting RPKI that will drop that announcement. However, regional problems are still possible.

But, we are still not there yet. If you want to get a sense of how fragmented the RPKI effort is, take a look at this site, Is BGP Safe Yet? Spoiler: no.

Scroll down a bit to the Status section and click the Show All link. You can quickly see that the major backbone providers (most of which you have probably never heard of) are marked safe because they are doing both things: a) checking for signed ROAs and b) dropping announcements that are not signed properly. But as you go down the list you will see a lot of “partial” listings which are transit provider only doing one of those things, and finally, you will see a lot of smaller players that you may start to recognize as consumer internet service providers, which are not implementing RPKI at all. That list is by no means exhaustive, but it gives you a good sense of how far away we are from safe internet.