Anycast routing simulator

The map shows six anycast sites (squares) and fourteen ISPs (dots), each coloured by the site its users reach. Click an ISP to see its AS path, how far its traffic travels, and every route its provider heard with the reason one won. In the Sites table, withdraw a site's announcement or prepend extra copies of the AS number to it; the panel below lists what changed in each BGP round until the network settles.

Dashed lines are ISPs that do not reach their nearest live site. Distances are straight-line, and the round-trip times are the minimum light needs in fibre; real paths are longer.

Anycast gives one IP address to many servers in different places. Each location announces the same prefix to the internet with BGP, the protocol networks use to tell each other which addresses they can reach. Every other network hears several routes to that prefix, keeps one, and so sends its traffic to one of the sites. To the user nothing is different: there is one address, and packets to it arrive somewhere that answers.

In ordinary unicast, an address belongs to one place, and every network's route leads there. RFC 4786 (2006) describes how operators run anycast services, and RFC 7094 (2014) discusses the trade-offs. The most visible users are DNS: every root server identity and most large public resolvers are anycast, and content delivery networks use it to reach the nearest copy of a website.

Nothing in BGP measures distance. When a network has several routes to a prefix it runs a decision process, set out in section 9.1.2 of RFC 4271, whose first and strongest step is local preference, a number the operator sets by policy. Most networks set it by who they learned the route from: a customer pays them, so customer routes come first; a peer exchanges traffic for free and comes next; a provider charges them and comes last. Only among routes of equal preference does the shortest AS path, the list of networks the announcement passed through, win. Further tie-breaks follow and end with the lowest router ID, an arbitrary number.

The same economics decide what is passed on. A network tells everyone about its customers' routes, because carrying that traffic earns money, but tells only its customers about routes learned from peers and providers. Gao and Rexford showed in 2001 that when every network follows these rules, BGP always settles on one stable answer. The simulator uses exactly this model, so the answer it shows does not depend on the order in which messages arrive; its test suite checks that against random orderings.

Here, Transit B is a provider of the Frankfurt and Ashburn sites, so it hears both directly as customer routes with the one-hop AS path 64500. It also hears longer customer routes and peer routes:

With all six sites up, 4 of the 14 ISPs in the model miss their nearest site:

• Toronto buys transit from Transit B and so goes to Frankfurt, 6,332 km away (at least 63 ms round trip), instead of Ashburn at 546 km (5 ms). A tie-break inside its provider decided this.
• Buenos Aires buys from Transit A, which has a customer route straight to Ashburn and only a longer one to São Paulo through the Latin American network. Its users cross 8,417 km instead of 1,682.
• Delhi buys from Transit C, a provider of the Singapore site. Transit C's route to Mumbai goes through India transit and is one AS longer, so Delhi goes to Singapore, 4,141 km away, rather than Mumbai at 1,143.
• Johannesburg has no site nearby at all; its provider sends it to Frankfurt.

These are the two causes Li and colleagues found when they measured real anycast deployments, including root DNS servers, in 2018: inter-domain policy that makes paths longer, as for Buenos Aires and Delhi, and arbitrary choices between equally good routes to different sites, as for Toronto. Most queries in their measurements travelled farther than necessary. Operators fight it by connecting each site to the same large providers and by peering directly with ISPs near each site, as the Ashburn and Frankfurt sites here do with ISPs in New York and Berlin.

Anycast's failover needs no health checker: when a site stops announcing the prefix, its neighbours withdraw the route, choose their next best, and tell their own neighbours, until every network has a route to some other site. Press Singapore fails: it takes 4 rounds, and Delhi, Singapore, Jakarta, Tokyo move to Mumbai. News of the withdrawal travels one hop per round, so for a moment some networks fall back to a route that still leads to Singapore: Southeast Asia transit switches to Transit C's copy of the Singapore route, which Transit C withdraws a round later.

On the real internet each hop adds processing time and timers, and Labovitz and colleagues measured routes taking minutes to settle after a withdrawal. During that time some traffic can still head for the failed site. Moving users also breaks anything that keeps state at the server: a TCP connection in progress cannot continue at a different site, which is why RFC 7094 discusses anycast for long-lived connections with care, and why short DNS exchanges over UDP suit it so well.

If only one site stays up, everyone reaches it: with only Ashburn announcing, 11 ISPs move there and BGP needs 5 rounds.

AS path prepending repeats the operator's AS number in one site's announcement, so the route looks longer and networks that compare path lengths choose another site. It only acts at the AS-path step, after local preference, and it affects every network that compares the route, so the effect is coarse and sometimes surprising.

• Prepending Singapore once moves no one: Transit C now hears three customer routes of the same length (Singapore, Mumbai and Sydney) and the router ID still favours Singapore.
• Prepending it twice moves Delhi to Mumbai and Tokyo to Mumbai. Delhi gains, but Tokyo ends up farther away than before.
• Prepending Frankfurt twice moves Toronto, London, Johannesburg to Ashburn. Transit B now prefers the shorter route through Ashburn for all its customers at once, including London, which is next door to Frankfurt.

Operators also use BGP communities, tags that ask a provider to lower its preference for a route or not to pass it to certain peers, and they announce a prefix only to selected neighbours from some sites. All of these change the inputs to every other network's decision process; none of them can choose the site for a particular user.

The alternative to anycast is to give each site its own address and let DNS choose: a GeoDNS server looks at where a query comes from and answers with the address of a nearby site. It can use any information, including measured latency and site load, which BGP cannot. Its weakness is that the query usually comes from the user's DNS resolver, not the user, so a user of a distant public resolver gets a site near the resolver. The EDNS Client Subnet option, RFC 7871, lets the resolver pass on part of the user's address to fix this. GeoDNS also fails over more slowly, since clients keep a cached answer until its TTL runs out.

Many CDNs combine the two: anycast for the DNS servers themselves and for some services, DNS-based steering where they want finer control.

The topology is invented. Three global transit networks peer with each other; five regional networks buy transit from them; fourteen ISPs buy from one of these; and the operator, AS64500, connects each site to one or two providers and, in two cities, peers with a local ISP. Other AS numbers come from the ranges RFC 5398 reserves for documentation. Each network picks routes by relationship, then AS-path length, then the lowest router ID, and follows the export rules above. The real decision process has more steps (origin type, MED, internal routing cost) and real networks are many routers each, so the model shows how the choice works, not which site any real ISP uses.