Over the years, I have learned that often, the unexpected will happen. Of course, we make plans, think out different issues or problems we think may arise, work out solutions to mitigate those potential issues, but we know that things within and even beyond our control will happen and cause problems for us and our clients.
unk-unk n. especially in engineering, something, such as a problem, that has not been and could not have been imagined or anticipated; an unknown unknown. – waywordradio.org
The question isn’t will problems arise, but how will I / you / we respond to the situation?
Last week we ran into just such a issue. Our DNS service provider, Digital Ocean, was being hit by a large and focused DDoS attack. They had set up CloudFlare to help protect them from DDoS attacks. However, this particular attack was able to bypass the protection setup and bring down their service for a couple of hours. This caused our clients websites to not be accessible.
After determining that the issue was outside of our control, and that our service provider was working on the issue, we could have left it at that. Instead, our engineering team started brainstorming ways we could get our clients up and running again. We were able to come up with a solution and get all of our clients switched over to it before our service provide was able to mitigate the attack.
We also didn’t stop there. We have since implemented a strategy that allows us to have a live backup DNS service running just in case another successful attack is launched against Digital Ocean’s DNS. This is the type of service you can expect from Xennsoft. With unexpected problems, come unexpected solutions.
