Positions:

- Retired Staff

Join Date: 06 October 2013

Posts: 17972

Posted on 23 November 2015 - 08:02 PM

In this special developer log I will be focusing on discussing what makes the ArenaPvP servers so fragile, why they have had lag issues in the past, and what we have done to fix the issues one by one in the past couple of weeks. Please note there are a lot more optimizations I have made in the past two weeks that I am not going over in this thread for intellectual reasons.

The Tab List

Let's start off with this one. The tab list is an amazing feature for the server. It allows our users to quickly access a lot of information about their ratings, the staff online, the number of players online, etc. The tab list is by far one of our favorite features. Originally designed by @SmellyPenguin over a year ago. This plugin has gone through numerous revisions. The latest set of revisions making it the more powerful, easier to add features to, and way less laggy.

The problem

1.8. That's literally the easiest way to discuss the lag issues. It is because of the decisions that Mojang made with the tab list in 1.8. The way the tab list is organized by the 1.8 client is by team name (scoreboards), player names, and then uuid's. In 1.7 the tab list was simply organized by the order of the tab slots sent by the server. This was a lot easier to work with since the server had full control of the tab slots. Now that the client decides how to render them we have to try and solve this problem.

The original solution

The original solution (and still partially is the solution) is to force all of the players onto a team together (even if they don't know it) and make that team priority lower than the team priority of the 80 tab packets we send the client (the ratings, staff, etc). The original way we did this was by making a new scoreboard for every player who logs into the server. Then after making that new scoreboard all 750 players online need to be added to the scoreboard. Then all 750 of the players need to have the new player who just logged in added to theirs. Once all of these team additions were done the tab list would look proper on 1.8. Also every time a party was created or left, an event was joined or left, the scoreboards were reset and all 750 players had to be alerted of this.

This is fucking slow and terrible, but it was our first attempt at making the 1.8 tab list work.

The new solution

Threads. Lots of threads. Instead of creating and destroying the scoreboard constantly every time someone joins and leaves a party, joins or leaves an event etc we just keep one scoreboard per player. Scoreboards are no longer (on our side at least) shared between a party. The user doesn't see any differences with the new tab list plugin. There is a huge performance increase on the server though (and as a result less lag). We have a bunch of threads that send this information to the clients now instead of the main server thread trying to process 750 team additions etc. This new solution is now live since yesterday.

BuildUHC/SkyWars

These ladders are some of the more fun and creative ladders we now have on Badlion's ArenaPvP server. These ladders were also the first attempt at making an ArenaPvP server with changing content (blocks that can be modified). Up until now we had specifically made it so that all arenas were static (minus fire from MCSG) and it was a lot easier to manage.

The problem

When we are resetting the BuildUHC or SkyWars maps sometimes thousands of block updates need to be done. It really depends how much was changed (e.g. lava being put down, or the islands in a SkyWars Arena disintegrating). All of these block updates trigger in Minecraft what is known as lighting updates. These lighting updates are really slow when you are doing thousands of them.

The Solution

https://github.com/desht/dhutils/blob/master/Lib/src/main/java/me/desht/dhutils/block/CraftMassBlockUpdate.java

This nice piece of Java code bypasses these lighting updates that Minecraft natively tries to do. It then gives us the option to do lighting updates in one go instead of trying to do a bunch of little ones that are not needed. By using this class instead of using the regular block.setType() method we have been able to save a lot of processing power on the servers, thus making it faster.

Conclusion

I'm sorry that I can't share much more about what I have done recently to speed up the server. I can just give some general information of things I have done over the past couple of years in total of my work:

1. Made over 140 patches to Spigot (1.7 jar). A lot of these patches are optimizations, and some are feature additions/tweaks that we need for our plugins.
2. Rewritten almost every public library/plugin that we ever used to make it faster and for our own internal usage (permissions, chat, worldguard, worldedit (work in progress), etc, etc).
3. Made the server go from a max of 60-100 players to 750 with almost perfect TPS (up to about 19.97 or 19.98 up from 19.92 a few weeks ago [still trying to get to that 20.0]).

One of the biggest problems and challenges with the fact that we have 750 players is just handling things with all of the players in the world. Considering that Minecraft naturally is mostly single threaded, it has taken a lot of tweaks and work to make it so that we are able to support the amount of players that we are. The problems are always very challenging and we keep trying to find ways to increase performance and allow for more players on a single server (we are seeing if we can increase the # of players in season 11 without hurting performance/gameplay).

That is it for now.

~MasterGberry

PM Link