With only a couple of days left until the start of the League of Legends World Championships 2015, lolesports started to reveal their Top 20 Player list for this year starting off with CLG’s Doublelift and SKT’s Bang on places 20 and 19. One day later KT Rolster’s Piccaboo and CLG’s Aphromoo followed on places 18 and 17. Although the ranking only contains 4 names at the time this article is being written, it immediately sparked a fire in the LOL Community with fans, professional analysts and journalists expressing their primarily negative reactions on Reddit and Twitter. Questions and statements of legitimacy, reasoning and even bias were asked or made – some better or worse articulated and reasoned than others. It is not the intention of this article to discuss whether a specific player deserves the given spot; rather the aim is to critically analyze some of the major structural problems with Riots Top 20 Player Ranking in a constructive way.
A player ranking system in any kind of sports serves to determine which players are the best and why. Everyone who follows some kind of sports and has had an argument with friends about which player is the best knows that it is almost impossible to agree upon a player without specifying some sort of measurable criteria. Without measurable criteria the discussion goes on for hours without any progress. Opinion bias takes over and the result is not having a result. The same goes for a discussion about League of Legends players. And that is where the problem with this year’s Top 20 Player Ranking starts. By introducing a Top 20 Player list with a top to bottom ranking system without any obvious criteria and cross-referencing between the different ranks, the list lacks any sort of transparency and has therefore rightfully caused a stir among the community.
To go into more detail, there are three major issues with this ranking: (1) lack of transparency with regard to the criteria catalogue on which this ranking is based upon, (2) lack of objective statistical measurements which remove opinion bias, (3) lack of transparency with regard to who was involved in the decision-making process.
What criteria are used to determine a player’s position in the ranking?
Apparently last year’s criteria (importance to their team, individual skill, strength at their position, leadership, consistency and clutch play) were also applied this year. This piece of information is nowhere to be found on the ranking website, but was provided by Kelsey Moser on Twitter. Although Kelsey Moser is a well acknowledged analyst and is probably familiar to you if you follow the Chinese League of Legends scene, such a vital piece of information should be easily accessible to everyone who visits the website and not only to people who use Twitter and follow a specific person, who is not even affiliated with Riot. This kind of information policy is a recipe for disaster.
Further we need to take a closer look at the criteria itself. As stated above criteria, which allow for a truly objective analysis about which player is the best and why, need to be translatable into measurable variables or indices. Certainly, some of the criteria could be translated into measurable variables but based on the given information, we know that this is not what happened in this case. Rather the ranking is the result of a compromise based on the subjective opinions of the involved persons. This assumption is confirmed by Frank ‘Riot Mirhi’ Fields on Reddit: “This entire article series is a subjective exercise. These are opinions.”
Who was part of the decision making?
If the ranking was purely based upon a rating index which consisted of proven statistical measurements only the legitimacy and accuracy of the data would be significant. But since this is clearly not the case, the question who participated in the decision-making process becomes particularly important. According to Twitter well known and acknowledged analysts were not asked to contribute to opinion-forming. MonteCristo stated on Twitter, that: “Just like last year, I was not consulted about the Riot top 20 players at World series of articles (…)”.
Frank ‘Riot Mirhi’ Fields stated on Reddit that the ranking has been conducted by a group of people “(…) who provide analysis at Riot Games Esports”. The Top 20 player ranking committee for last year’s World Championship included journalists, casters and analysts – whether this also is the case this year remains unclear. Unfortunately there is no further information who exactly participated in the decision-making process. This is problematic since the ranking therefore lacks any form of context.
In the end the controversy around this ranking is homemade by Riot. If the purpose of the article series was to introduce certain players, showcase their strengths and weaknesses and show the variety of different playstyles why present it in form of a top to bottom ranking? A ranking system raises certain expectations in regard to data presentation. Unfortunately the article series cannot fulfil these expectations and is therefore rightfully criticized. This is unfortunate, as the article series provides a lot of valuable information – especially for more casual League of Legends viewers but the way lolesports decided to present it takes away a lot of credibility from the actual content.