Breaking Down NA Server Instability, the East Coast Server Problem and Riot’s next step

The North American servers are currently facing their longest sustained period of instability in years. Presently, there’s a lot of static within the community about what is causing the problems, whose responsible, and a viable timeline for when those issues will be resolved. Now, the problems facing the North American server can be boiled down into two categories: packet-loss and high east coast ping. These are two entirely separate issues that have completely different causes, and yet keep getting confused with each other. Thus, it’s important to take a look at each issue individually as they are unlikely to be solved at the same time.

High Ping and the necessity of an East Coast server

East Coast players have struggled with higher ping since League of Legends was released. From 2010-12, the average ping on the east coast was in the 80 ms range. That’s what I experienced living in Upstate New York. Since then, that figure has crept up slowly over time. To provide a personalized opinion, my average ping was 94 four months ago. I now average between 100 and 110 ms with occasional spikes into the 140 range. Everybody knows why East Coast players have higher ping. The North American servers are located in California, almost 3000 miles away from the Northeast. Put another way, that’s roughly the same as the distance between Lisbon, Portugal and Moscow, Russia. When Riot moved the EU servers to Amsterdam, that put the entirety of the European continent within 1500 miles of both servers, half the distance between the location of the NA server and the east coast of the United States. At this range, the maximum ping a player would experience is around 60 ms. While the two servers would ideally be located on opposite sides of the continent, the above situation delivers a perfectly acceptable experience to everyone on either server.

If you look at the above map, the ideal location for the east coast server would actually be located in Chicago. Each of the 3 smaller rings represents a theoretical estimate of the area in which a player might have a ping of 50 MS or under. The black ring represents the current servers. As you can see, about half of the country is within that radius. The red ring represents placing the servers in New York City. You’ll notice that placing the servers in NYC ensures that the entirety of the continental United States is within that 50 MS zone. However, nearly half of that area of improved ping is ocean. Moving the east coast server over to Chicago, represented by the blue ring, provides a more improved experience for more people. For comparison sake, the outer black ring represents the zone where a player might experience 100 MS ping. Note that Hawaii, despite being thousands of miles off the west coast, is closer to the current NA server than the entire east coast.

Packet Loss and why your ISP is to blame

Packet loss is not a new experience for most gamers. It can happen when the server is under attack from an outside source (DDOS), and it can happen when your Internet Service Provider is having problems. The NA server has been experiencing server-wide packet loss for the better part of a month now. Below you’ll find a recent reddit post by Riot Tryndamere, President and CEO of Riot Games, addressing the problem.

Glad you guys are looking into this stuff. Long story short, the degradation is not on our end – and we are investing in a ton of infrastructure to address the issues and working more closely with upstream providers (hence why it’s not so simple as “omg east coast servers”).

Investing in infrastructure is a pretty easy concept to understand, and par for the course for a company with a constantly expanding player base. The more interesting portion is the bit about working closely with upstream providers. The concept of an internet super speedway is not new to anyone that has been following the ongoing battle for net neutrality. The short version is that ISP’s want to be able to charge you extra for access to high-traffic services like Netflix, Hulu and facebook. They argue that regular internet speeds would remain the same while the speeds available for these websites would be a marked improvement over those currently offered. Opponents argue that legalizing these super speedways would result in ISP’s making websites painfully slow unless the services ponied up the money to get those faster speeds, effectively holding the companies hostage. Naturally, these increased costs would then be passed down to the customer. These are tactics that are already being utilized at a corporate level. Verizon and Comcast have both had public disputes with Netflix over customers having their speed throttled, with the entire debate essentially boiling down to a he said/she said. Note that Comcast is also the company that’s trying to get the FCC to allow them to limit home internet data similarly to how your cell phone data is limited, and that redditers have correctly deduced that Verizon users are experiencing significantly worse packet loss than those of other ISPs. In short, these companies are the enemy. 

Matters are further complicated by the existence of network tiers. Tier 1 ISPs have complete and uninhibited access to the entire network. Tier 2 ISPs have access to a portion of the network themselves,and then pay tier 1 ISPs for access to the remainder. Tier 3 ISPs don’t own any of their own network and have to pay for all of their access. AT&T and Verizon are examples of Tier 1 ISPs, while Comcast is a Tier 2 ISP. Someone like Bright House, who negotiates all their deals through Time Warner Cable, would be an example of a Tier 3 ISP. Tier 3 ISP tend to have low internet speeds and spottier connections as pretty much anyone in Florida who uses Bright House can attest. The existence of these middlemen drive up the cost of internet for the consumers, and is one of the primary reasons why internet is much more expensive per megabit in the United States than it is in other countries.

In the end, everyone wants to get paid. The rapid growth of online video, casual gaming and the cell phone industry has forced companies to continuously expand their bandwidth in order to keep up with demand. ISPs are trying to find creative ways to pass along these costs to the company. People are typically more willing to pay more for a product they can’t live without. Services like Netflix have become an indefensible part of many households and the ISPs know such services are a great point of attack. Video Gaming is no different. As by far the most popular video game in the world, League of Legends will inevitably find themselves in the same boat. As a frame of reference, people play more hours of League of Legends in a month than watch hours of netflix, though both amount to over a billion hours a month. Should Net Neutrality ever come to an end, it will likely almost mean the end of free-to-play video games like League of Legends, Dota and Hearthstone. 

If you’re looking for a TL;DR, this is it. Riot will have to pay money to the upstream providers to ensure a stable connection. The Upstream Provider gets away with it because they argue that they can’t afford to maintain the integrity of the network due to the strain caused by the service, thus forcing Riot and other companies to pay them to do so or accept the fact that their customers will get an inferior experience. 

The Myth of Riot’s Revenue

One of the favorite arguments on reddit these days is that Riot has no excuse for the poor servers because of how much money they are making (On pace for $1 Billion in micro transactions this year). The problem is that the figures Riot has released are revenue, not gross or net profit. Riot’s costs are immense. The company is estimated to have a thousand employees. According to Glass Door, the average salary for a software engineer at Riot is $113,000. Some positions push the $200,000 level. Based on these figures, It’s safe to say that $100,000 is a conservative estimate of the average salary at Riot Games. That means that Riot Games has $100 million in salary obligations a year. Riot just finished building new European servers in Amsterdam, fixing EUW’s longstanding stability problems (problems that were much more severe and longer lasting than those currently facing North America). That project took a year to complete, and only just finished. Thus, it’s reasonable to expect a similar timeline for North America now that Riot has turned their focus in our direction. Riot is also in the middle of building a brand new headquarters to house their growing employee population. 

Something that often flies under the radar is the fact that Riot pays Garena and other companies to run servers outside the Americas and Europe. League of Legends largest player base by far is the Chinese Region, which is personally run by Riot’s parent company, Tencent. This likely means that a significant portion of the profits from League of Legends’s largest region don’t actually go to Riot.

Community Fervor

Despite the fact that North America’s problems aren’t nearly as bad, nor as long lasting as those that EUW faced, the outcry has been exponentially greater. It hasn’t helped that the Holiday Season has resulted in a more muted response from Riot than players are used to as most Rioters are currently on vacation. It’s not uncommon for a quarter of the hot page to be filled with a variety of posts either discussing the issue, or just complaining about it. Unfortunately, the petty nature of the majority of these posts have distracted from the real issue and prevented any serious dialogue on the topic. The fact of the matter is that it’s more important for Rioters to be able to spend time with their families on Christmas than it is for players to be able to play the game. It all comes down to priorities. While many would argue, or even demand that they, as paying customers, deserve to have the game run perfectly all the time, the world isn’t that cut and dry. It’s important to have some humanity.  

Outlook

First, Riot needs to make a statement addressing the problem as soon as the Holiday ends. It’s clear that the uproar has gotten so large that they can’t let it go without a serious, well thought response. After that, Server stability is likely to improve by default as we move away from the holiday season and the overall strain on ISPs across the country is lessened. It’s also the time of year that ISPs typically force the issue with the service companies that use their networks as they look to lock down revenue for the next fiscal year. It’s likely that Riot will have finalized their agreements with Upstream Providers within the next few months. After that, it all depends on how fast Riot is able to improve the infrastructure. They will also have to find a balance between spending money on stabilizing the current server, and building the new ones. Personally, I’d much rather deal with some instability now if it meant Riot could get a new server up and running faster. It’s likely that server stability will see an improvement in the short term, but fixing east coast ping is a long term issue. In short, I advise patience.