For a layman, words like “tick rate” and “netcode” can be hard to understand—so Respawn brought in the experts. Lead engineer Samy Duc explained a series of issues and common scenarios that happen to players in Apex Legends, their causes, and what the developers are doing to fix them in a 5,000-word blog post today.
In addition, Duc touched on a highly debated topic: The reason why Apex runs at a 20 Hz tick rate—and why upgrading it may not have as significant an impact as some fans think.
What Respawn has done and upcoming upgrades
Respawn has already pushed a series of tools that will increase its capacity to deal with issues, according to Duc, and there are more on the way. The Performance Metrics menu introduced in season six, for instance, is more than just a handy tool for players to monitor their frame rate and ping.
“Those numbers help you—and us—understand what you’re experiencing while playing,” Duc said. “In other words, we can translate what you’re feeling into actionable and technical information.”
Coupled with the performance menu, Respawn “began tracking key performance metrics for players and servers” as part of a push to provide its team with “personalized and targeted means of investigation.”
The studio has built upon that strategy in the most recent seasons with automated data gathering and analyzing one week at a time. “This approach has been paying off already,” according to Duc. “Week after week, we can confidently say if the health of our server fleet is better or worse.” But there's still room for improvements.
Respawn has moved to real-time alerting in addition to the weekly report. This should help streamline the process and allow the devs to fix hardware problems and start investigating more quickly. The team will also add a unique Server ID (SID) that will make it even easier to spot the exact servers when players report an issue. Both changes should roll out during the next season, Legacy.
Server tick rate
Duc defined tick rate as “the number of simulations that the server runs per second.” The Apex community's complaints about the 20 Hz tick rate have been a fairly common sight since the game was launched.
Duc explained that upgrading to 60 Hz does have a benefit. But “it isn't massive,” according to him, and “it wouldn't do anything for issues that are tied to plain old lag (like getting shot while in cover), ISP-level issues, or bugs (like with hit reg and slow-mo servers).”
According to Respawn's calculations, a player with a 50-millisecond ping would experience a 75-millisecond delay (the full amount of the ping to receive the information, plus half of the delay to account for when the information gets to the server). This would result in a five-frame lag between players' input and its actual impact on the game when running Apex at 60 FPS.
Using the same model to calculate a 60 Hz netcode yields a three-frame delay—just two frames less than running at 20 Hz. “So for triple the bandwidth and CPU costs, you can save two frames worth of latency in the best-case scenario,” Duc said.
Increasing bandwidth is also a factor for Respawn. “Keeping bandwidth costs low for games is much more critical than, say, for video streaming,” Duc said. Streams can buffer and download speeds can oscillate without much impact, but for games, “skipping even a couple 50ms intervals can start to feel bad”—and can be even worse when they stack.
Duc touched on a series of common complaints by the community—from failed hit registration and slow-motion servers to packet loss and code:net. The developer discussed slow-motion servers and revealed that, despite a number of possible causes, “it’s sometimes linked to machines in the datacenter not performing as they should.” The upcoming improvements to detection, slated to roll out in Legacy, should help curb instances of that error.
Spiking latency can also be related to occurrences on Respawn's end. “Usually, the problem has something to do with increased performance demands introduced by new features season after season,” he said.
To monitor those requirements, the studio produced a tool called RPROF file, which “is basically a view of what the server is doing during every frame.” It should help optimize new features that get added to the game over time and generally reduce latency in the long run, according to Duc.
Packet loss is a tricky issue, according to Duc. Sometimes, the issue may not lie on the players' end or on the servers, but instead, in one of the nodes that connect the two. Detecting where the bumps lie requires tracing from both perspectives, a process that can be “extremely time consuming and slow to resolve.”
“Depending on our findings, we have to negotiate with different business partners all over the world," according to Duc. Reducing bandwidth and implementing automation can potentially help curb the issue.
Getting killed behind cover is not uncommon in Apex and its occurrence has to do with lag compensation. Servers have to redistribute the information between players, and sometimes, two players will have significantly different ping. “To solve this, our servers have to constantly look at not only what’s happening for you and your opponent at that moment, but also what was happening from both your perspectives at the time both of you input your actions,” Duc said.
Apex has a built-in tradeoff that mitigates the advantage that low-ping players have on a skirmish. “The upside is that you can play Apex Legends and play relatively well even if you have higher than average latency,” according to Duc. “We believe we should reduce “nonsense” at every opportunity, but when we have to deal with less-than-ideal experiences, we want to do so in a way that’s equal and fair to all players.”
Lastly, Duc touched on the infamous code:net issue and its variants. The occurrence "is a generic error message that the game displays whenever your game timed out from the server," according to Duc, and can be caused by problems on either end of the connection.
"In fact, we’ve found that some of the most serious code:net bugs (and related bugs like code:leaf, etc.) might have more to do with Respawn's services supporting the game that may need to be investigated," the developer said.
Although Respawn has preemptively implemented measures to mitigate the error, if you're experiencing code:net issues, contacting EA Help is a good way to let Respawn know about the occurrence and to have it fixed. "We’ve had some success in recent weeks of addressing some of these, but we know we have more to do," Duc said. "Report issues to us and we’ll do our best to resolve them ASAP."