‘A worse experience’: Activision declares secret CoD SBMM experiment a failure

Skill-based matchmaking has long been the bogeyman in the eyes of many Call of Duty fans, but a recent paper released by Activision revealed that they secretly ran an experiment on a portion of the player base and found that its absence made the experience worse for almost everyone involved.

Players of all stripes have long been unwitting guinea pigs of many changes when it comes to live-service games, but this is one of those rare cases when an actual experiment has been conducted by the publishers and a proper summation was released for everyone to check out, and the outcomes of Activision’s science session have been quite interesting indeed.

In early 2024, half of the North American MW3 players were queued with a modified version of the matchmaking algorithm, where the skill factor was tuned down for experimental purposes. Turns out, the much-maligned skill-based matchmaking system (SBMM for short) has an undeniably positive impact on the gameplay experience, as retention has significantly dropped in the portion of the player base without it during this experimental period, with the exception of the top ten percent of players, who happily kept running and gunning their way to victory, racking up the numbers along the way.

While the experiment didn’t completely ignore the skill element, the authors of the paper are convinced that things would have been even worse had they done so, writing, “It’s important to note that skill, as a factor in matchmaking, was decreased for this test, but not removed entirely from the matching algorithm. Based on our history of testing, completely removing skill from matchmaking would amplify the observed effect.” Further down, they drop the interesting fact that they have repeatedly run this test at various times over the past five years.

Beyond this experiment, the paper offers a fascinating insight into how CoD’s matchmaking algorithm works and how different factors are weighed in the process of finding you the perfect rage-inducing match. It is definitely worth a read if you are looking for a scientific reason to get annoyed at your teammates the next time you queue up for a MW3 match.