After many weeks of Destiny 2 players requesting comment from Bungie on the matter, the developer has finally opened up about its plans for the next few seasons.
The degrading state of Destiny 2’s servers has been a hot-button issue in Season of the Deep. It’s an issue that has only gotten hotter as the game suffered from emergency downtimes and frequent cases of players receiving error codes.
In today’s This Week in Destiny blog, the Engineering team went in-depth about the issues currently facing the title and the steps they need to take to remedy them. By the launch of season 23, they’re hoping to target “deeper and broader” architectural improvements that will have a noticeable effect on the stability and rapid recovery capabilities of Destiny 2’s services.
“The services that power Destiny 2 are a critical part of every player’s experience, and we are committed to improving stability and reliability long-term,” the team said. “In preparation for Lightfall and for future large releases, we invested in a wide effort to update our internal services with the long-term goal of improving scalability, maintenance, and levels of support for our service infrastructure. However, we’ve identified several issues with those changes that we are working to correct.”
The team went into detail about Destiny 2’s key ‘Claims’ service—the system that routes all gameplay messages received from Destiny 2 to the server responsible for player data. Claims handles “a tremendous amount of volume,” and it’s because of this that it was the focus for a few major improvements ahead of Lightfall’s launch this year.
While those improvements did help with Destiny 2’s ability to handle scale, leading to a smooth launch for the expansion, other issues cropped up in the live game environment concerning the Claims service’s ability to recover after a disruption. These disruptions are one of the key causes behind a large subset of players receiving error codes such as Weasel and Baboon, especially recently. What’s worse, according to the team, “a rolling restart of our Claims service is not always enough to restore the service.”
Related: Bungie to compensate Destiny 2 fan artist whose work was ‘mistakenly’ referenced in cutscene
Instead, the team has had to perform full restarts of Destiny 2’s services to restore the Claims system. It’s an issue they are “rapidly working to correct,” but they also stress that addressing these issues “is not a process that can happen overnight.” It is at least an answer for why Destiny 2 has experienced so much emergency downtime lately, which has left the game offline for several hours at a time in the worst-case scenarios.
To assuage community concerns even more, Bungie lifted the veil on a lengthy roadmap for improving the game’s stability. This is divided up into different key sections—what’s ongoing, what’s slated for a mid-season update, and what’s to come in seasons 22 and 23.
The ongoing work is already underway and will continue throughout the timeline, which includes:
- Improvements to production and deployment processes to reduce the risk of disruptions for players while reducing maintenance and deployment downtime windows.
- Improving response procedures for incident recovery to bring Destiny 2 back online as quickly as possible when incidents occur.
Slated for an upcoming mid-season update are some other minor tweaks. These are mostly focused on diagnosis, in an effort to make the more significant improvements planned for future seasons more effective. They are also designed to minimize the risk of further degrading the already poor stability, something Bungie is keen to avoid.
- Targeted improvements to logging and alerting systems, allowing Bungie to diagnose issues more quickly with Claims and related systems.
Season 22 is where the majority of major improvements are scheduled to go live. Many of these improvements are focused around the ‘self-healing’ ability of Claims, and reducing the amount that Bungie should need to bring Destiny 2 offline when issues arise with the system.
- Adding functionality for services to detect Claims services that are in an unhealthy state and send their messages to healthy services instead.
- Six targeted fixes to Claims systems where issues have been identified that could impact Claims stability or recovery.
- Improvements to better evict old gameplay messages in their pipelines, which should help with faster recoveries and reduce the chance of a “death spiral” of slow messages causing more slow messages.
- Adding improved support for targeted “Chaos Testing” against Destiny 2 services, allowing for better simulatations of different failure modes for those services.
- Adding more logging for non-Claims portions of the messaging pipelines to detect other issues that could lead to connection problems for players.
After these sweeping updates go live, Bungie will then start to dictate the best course of action going forward into the launch of season 23. They want to make significant updates to the architecture surrounding Destiny 2’s services to make them more robust for the future, but what that will look like depends a lot on how season 22’s updates end up landing. With changes so structural in nature, Bungie warns that “additional instability” could be possible as some of these improvements roll out.
“As always, we will work to minimize this risk with deep testing and around-the-clock monitoring,” they quickly assured, “and will implement the most appropriate response needed to get everyone back online as quickly as possible when instability occurs.”
This does mean that a lot of the server improvements that players are hoping for are still a long way off in the timeline. However, it’s now clear that Bungie has a solid plan in place for Destiny 2’s foundational integrity. By the time The Final Shape rolls around, the game will ideally be in a state where players are no longer worrying about emergency downtime taking everything offline once or twice a week.
“Our goal is to give Destiny 2 players the best possible experience, and we look forward to seeing the results of the work we’ve been doing since Lightfall’s launch to improve stability,” the Engineering team concluded as they signed off. “Thank you for your patience and feedback as we work toward achieving the milestones above.”
Published: Jun 29, 2023 03:43 pm