Last edited on 18 February 2016 - 08:21 PM by MasterGberry

Ok so I decided to make another special developer log for the monstrosity that has been the lag on ArenaPvP now this past season. This issue has been known for a couple of months now, but we had been so busy that we never had time to investigate it. About a month ago we started digging into it since we had more time and it was becoming a serious issue (even more so with 1k player limits). Unfortunately we wasted countless hours looking around for this stupid small bug that was causing terrible 10 second lag spikes on the server. Going to break down the post into the problem and the cause.

The problem

The actual problem we eventually figured out from timings and such was chunk updates. Our Arena servers have to reset the arenas for certain kits like BuildUHC, Spleef, and SkyWars. This was not an original feature built into PotionPvP 3.0 when the original plugin was constructed back in late 2014. We never imagined that we would do something of the sort. To be honest the way most of it is handled is not terrible, but it could be better. The issue was that something in our arena resetter was screwing up and sending way too many chunk updates (which get sent to all clients in the render distance of any of these chunks). Now imagine what happens if the spawn is included in these chunk updates? Now the majority of the server is being sent a lot of chunk updates. To clarify the number of chunk updates would sometimes be as high as 100,000. These were the worst of the spikes needless to say and would lag for even longer than 10 seconds.

The debugging

Ok so the simple solution to us was "spit out a shit ton of log information until we figure it out." Sadly this didn't really help in the end. We actually found some other bugs which we have patched and you will find in the next dev log while searching for this lag source bug (which isn't a bad thing). We then turned to the community in our announcement thread trying to find out if someone could reproduce these terrible lag spike issues. We did have some good bug reports, but in the end none of them were "the issue" that we were searching for. Regardless we want to thank everyone for trying to help us.

People might not want to hear it, but it's very hard for us to test everything. There are a lot of features and unit tests are not very plausible in Minecraft. As a result sometimes throwing things on production and watching a server catch on fire and fill up with exceptions in the logs is actually the fastest way for us to test.

The actual cause

So at this point I have probably re-read about half of of the ArenaPvP plugin (which is around 30k lines + all of the other plugins/libraries it uses). Just been chasing debugging log leads and thinking of any crazy scenarios I could that would cause issues. In the end this morning when I woke up after going to sleep frustrated at figuring out nothing yesterday, an idea dawned on me. "The way we handle water/lava buckets is kinda shitty. I should poke around there." So I did. I finally found the issue causing all of our headaches.

How to reproduce bug:

Step 1: Play a UHC Meetup/BuildUHC FFA
Step 2: Place water/lava in the meetup/ffa
Step 3: Die in the meetup/ffa
Step 4: Leave the meetup/ffa
Step 5: Queue up for 1v1 or party ladder or another party ffa
Step 6: Play and wait for shit to go bad

The main issue is step #2. The problem is if you placed a water/lava bucket down and died and then later in the match someone else put a water/lava bucket nearby and it created cobblestone/obsidian. You are now playing in a new match. We track what you are doing in that match. The way we were tracking these cobblestone/obsidian blocks was based on the player instead of the arena. This was a terrible design decision. Our arena update keeps track of the min and max coords so that it knows which chunks to update. As a result you might be at -5000, 1000 and the last arena you were in was -1000, -4000. Guess what? That one block of cobblestone/obsidian that just got formed because of your water/lava bucket from the previous match just got attached to your current match. The min/max of your current game just went up by a LOT. Now we are sending 30k-100k chunk updates because the min/max's are so screwed up (even though only a few chunks were modified).

Conclusion

Testing can be a bitch. Finding these types of weird edge cases is difficult and hard. It takes us a lot of time to find these things sometimes, and we are shooting in the dark a lot of the time (like we were with this one). We had some good debugging ideas (in the logs) but in the end they really didn't help us to find the issue since it was a memory management issue and was a rare edge case that we could not reproduce in our own testings.

This bug has actually been there since BuildUHC became a thing. The number of lag spikes went up this season because we added BuildUHC FFA and Spleef. Also in addition the arenas are situated differently this season and the FFA/Event and regular 1v1 arenas are much further apart.

Be patient with us, we are always working to make things better.

Posted on 18 February 2016 - 08:23 PM

first

ty for your patience through this and fixing it

Posted on 18 February 2016 - 08:37 PM

2nd

Posted on 18 February 2016 - 09:08 PM

tyty

Posted on 18 February 2016 - 09:47 PM

4th

Posted on 18 February 2016 - 10:08 PM

5th

Posted on 18 February 2016 - 10:50 PM

Cool

Posted on 18 February 2016 - 10:55 PM

Always amazed by both how hard you work and how thoughtless and impatient people can be when criticizing Badlion for things like this.

Glad to hear you figured it out!

Posted on 18 February 2016 - 11:02 PM

Hype!!! I got a D+ so it was all worth! xD
In all seriousness I'm excited to see less posts like fix lag!!!!!11111!!!!11

Posted on 18 February 2016 - 11:16 PM

Wow… figuring that out is pretty insane, good job!

Posted on 19 February 2016 - 06:45 AM

Good work! This is why I play on Badlion.

Last edited on 19 February 2016 - 08:14 AM by Hivlik

Edit: I'm a dumbo

Last edited on 19 February 2016 - 08:14 AM by Hivlik

Big dumbo

Posted on 19 February 2016 - 08:48 AM

What did you snip?

Posted on 20 February 2016 - 02:36 AM

So are lava buckets going to be fixed in terms of it not spawning when you place it?

Posted on 20 February 2016 - 08:39 AM

Good luck :)

Last edited on 20 February 2016 - 01:14 PM by ThatOneCombo

-

Posted on 20 February 2016 - 09:11 AM

what no

Posted on 20 February 2016 - 09:47 AM

thats not lava

Posted on 20 February 2016 - 10:01 AM

Some servers have it where lava and water doesn't spread.