Posted on 31 March 2015 - 05:10 PM

Here at Badlion we believe in being very open with the players. I am writing this post here today to explain the recent events that have been happening on Badlion for the past month or two. We have grown a lot and hit a bunch of different issues with our growth along the way. I had received a few messages recently about donators feeling that they were wasting their money on Badlion, and this is literally the last thing I ever want to see. The donators are what makes this network run, it's the reason we do not have advertisements on the website or do other means of making money to pay the bills here. The freemium model is how Badlion survives.

We started having network latency issues back in late January with our old data center OVH. In mid/late February these became really bad and more common. Basically they were adding a lot of new clients and not increasing the traffic throughput. About 2-3 weeks ago I ran a MTR (a ping test that checks every internet node between my router and our server) and pretty much about 10% of the packets were being dropped. Some players were being affected earlier than this…but no one wanted to sit down and help me to debug the situation. Rather people wanted to bitch about a very complex system which I am just shooting in the dark when trying to debug. Last week with them (March 16th - March 23rd) was absolutely terrible. Everyone was having issues, ping was randomly spiking, just terrible. It was at this time that we decided we HAD to move to a new data center.

We moved our servers last Tuesday to a new data center and for most people things were pretty solid minus the long downtime. Last Friday/Saturday we got hit by a series of DDOS attacks and our data center could not handle them like the claimed.



Every single blue spike above is a DDOS attack on one of our bungees. There are 9 more pictures that I'm not going to post because they all look the same.

This is the main reason we had moved to OVH in the first place, they had a really solid anti-DDOS solution in place. Sadly Badlion has been the victim of DDOS attacks for a very long time. Just to prove a point, here is a screenshot of one of the attacks from yesterday that someone attempted on our network (I blanked out the server id and ip address):



This past Sunday we pointed the NA IP's to our EU location so that people could at least connect and play while we were trying to sort the DDoS issues, and Sunday we ordered Staminus DDOS mitigation to help with these issues and spent 8 hours configuring them (which is why there was super high latency over the weekend).

There was an incident around 7 PM EST also yesterday where a bunch of people randomly timed out from the servers. I had not yet fully configured the new anti-DDOS solution with Staminus, and as a result people dropped connection when one of our UHC's was about to start because they thought it was an attack. I have spoken with them over the phone and they assure me now that it is configured for Minecraft this should not happen. If it does continue to happen they will help us analyze and fix it.

The last of the chaos was last night when everyone dropped off the network at 11:10 EST during one of the UHC's. After all of this garbage I've been dealing with I was furious. At first I thought Staminus fucked up, but it turns out some internet provider in the middle screwed up. 12 minutes later they were re-routing traffic between the protection and our servers and we were back online. Sadly with the nature of the internet these things do happen…just really shitty timing

I think it's safe to say that we have pretty much spent every free minute of our time the past week trying to stabilize the new location/DDOS's etc so that players can enjoy the network. Something all of the regulars here know but none of the non-regulars know is that Archy and myself are hardcore gamers. Archy almost went pro at one time and I never strived hard enough for it. In every game we pick up we are in the top 1% of players, because we work to get there. We absolutely hate lag/bugs even more than you guys do, trust us, we do. It's why neither of us got much sleep the past week with all of these problems. Last night was the first night I actually got to sleep for the majority of the night.

I challenge everyone to try it for the next week or two while we continue to fine tune the new systems and see how it performs now that we are using a premium set of networks (and yes the donator money was used to fund this new addition). Archy and i both want everyone here to have a good experience, and it really upsets us when we see people non-stop bitching about things that we cannot completely control. The things like "laglion" are offensive and it's why we auto-mute for them. It's childish and does nothing to help the situation.

~MasterGberry

TL;DR Network should be more stable now. Fixed a lot of issues. I recommend reading last 2 paragraphs at the very least.

Posted on 31 March 2015 - 05:22 PM

We appreciate (well, at least some of us >.>) all the work you guys put at fixing these issues and hopefully these problems stop arising soon.

Posted on 31 March 2015 - 05:25 PM

Thanks for the hard work, really appreciated :)

Posted on 31 March 2015 - 05:29 PM

Thanks for all the time and effort you guys have put in recently - Hopefully it was worth it.

Posted on 31 March 2015 - 05:31 PM

Hopefully all these problems can finally rest and everything can go back to being the "lagless" experience it's supposed to be without any more complications.

Many of us may not express it directly, but I'm sure we all appreciate your hard work trying to deliver the best experience possible. It's such a shame things have been as complicated as they have been.

So, about those NA West servers…
Kidding. ;D

Posted on 31 March 2015 - 05:33 PM

As per usual: outstanding. Good work.

Posted on 31 March 2015 - 05:43 PM

The sad thing is that behind every one of these ddos attacks is a kid literally spending money to make others miserable.

Posted on 31 March 2015 - 05:47 PM

LAGLION AMIRITE HARDEEHARHAR

in all seriousness, i (and many others) strongly appreciate all the effort you guys are going to to make this server a better place

Posted on 31 March 2015 - 05:51 PM

Totally agree with you and thanks @MasterGberry @Archybot for all your amazing time that you have dedicated to this server, and id like to say that we ALL appreciate it, some more then others <3

#TYMasterBot

Posted on 31 March 2015 - 06:08 PM

@stheegala


we're not alone…

Posted on 31 March 2015 - 06:53 PM

Thanks for all the hard work you guys are putting into this ;)

Cheers.

Posted on 31 March 2015 - 06:55 PM

dont worry badlion u still be my fav server

Posted on 31 March 2015 - 06:55 PM

Thank you so much for this awesome server!

Posted on 31 March 2015 - 07:01 PM

Hopefully it will be fixed, I hope.

Posted on 31 March 2015 - 07:02 PM

Thank you to everyone putting time out of your day to fix these issues. I really appreciate and everyone on the Badlion Network should too. We need everyone to read this so we stop hearing, "Laglion" or "This server sucks stop lagging!" Because for me to hear these statements annoy me greatly. If anyone is reading this and has said "Laglion" or something along those lines, please see that the owners of the server are trying their hardest to make this an amazing server. (It already is ;p)

Posted on 31 March 2015 - 07:43 PM

Thanks for making all the problems clear to us and dedicating so much of your time to fixing these annoying lag problems. It is much appreciated by me and other competitive players alike.

#TYBADLION :D

Posted on 31 March 2015 - 08:17 PM

Thank you, you guys have done a great job

Posted on 31 March 2015 - 08:17 PM

^^^^^^ amen G33ke

@ArthiumG lol tru
Badlion is love

Posted on 31 March 2015 - 08:41 PM

Never understood how you guys can consistently pull through all the shit that this Minecraft community throws at you.
What a lot of the community doesn't understand is that nobody here, especially not Archy or Gberry, is getting rich off of Badlion. They never have. Every dollar and cent that has been handed over to support this server has done just that - support this server. And in the time I've been here, I've seen both of the owners work like slaves for something just because they are dedicated to this community. A game that neither of them really play (at least, not anymore) with a community that sits back and takes for granted all of the sweat and tears put into this network? I'd be out of there as fast as my proverbial feet can take me. But these guys are some of the most dedicated owners I've ever known. And, at the same time, they would take the time to help a depressed kid that they hardly know just by having a conversation with him over Skype. It's something I have never understood, but hey - I'm sure as hell not complaining.

@Archybot and @MasterGberry, thank you for all you've done for us, and sorry for all we've put you through. You guys are the best.

Posted on 31 March 2015 - 08:55 PM

o/