Image: Activision
Activision has released a paper observing the correlation between players’ skill differences and their influence on quit rates. The goal was most likely to justify its skill-based matchmaking (SBMM) algorithm, but whether or not the company succeeded is for you to decide.
Activision ran the Deprioritize Skill Test in Call of Duty: Modern Warfare 3 using its A/B test framework to “loosen the constraints on skill in matchmaking.” Basically, it matched people of wider skill groups than usual but didn’t remove the requirement entirely, although the studio believes that would “amplify the observed effects.”
Skill is calculated based on a player’s relative performance on a specific metric, including match total kills, kill/death ratio, and kills/deaths by enemy.
“After each match, we compute this performance metric for each player. All players in the match are then compared to one another, regardless of team. Based on these comparisons each player’s recorded skill value is then updated. … Note that the performance metric used only ever involves match performance; player progression or total time spent playing the game are not factored into skill.”
After a month, Activision divided the treatment group into 10 categories based on average skill to check how many people quit. It turned out the top 10% of players were happy to come back, but everyone below that didn’t return. The bottom 30% of participants suffered the most, their quit rate reaching 1.75%.
Image: Activision
“This effect may appear small, but this change was observable within the duration of the test. This will compound over time, just like interest, and will have a meaningful impact on our player population.”
Naturally, the Kill Per Minute (KPM) metric also increased for top players and decreased for everyone else.
Notably, Activision also tried to make skill differences stricter than it is in current matchmaking and noticed inverse trends, with up to 5% of the top players quitting the game.
Image: Activision
Activision concludes that people leaving due to unbalanced matchmaking will lead to worse conditions for everyone as once the top 10% players will become the top 30%, then the top 50%, and so on.
As mentioned above, the numbers are actually not that drastic. One X/Twitter user pointed out that the graph shows only about 0.5% average retention loss in a month and can be counted as “statistical insignificance” that doesn’t defend SBMM. While we could theoretically assume CoD would lose 6% of its playerbase, there is no straightforward evidence, plus there could also be other reasons behind the losses.
So when you find yourself in an unfair match next time, perhaps you’re just a part of an Activision test. Are you for or against SBMM? Would you climb ranks easier without it? Read the paper here and join our 80 Level Talent platform and our Telegram channel, follow us on Instagram, Twitter, LinkedIn, TikTok, and Reddit, where we share breakdowns, the latest news, awesome artworks, and more.
