Yesterday, August 23rd was a difficult day for Dovetail, but even more so for our valued hosting clients. We discovered early in the day that our primary Internet router was not responding to network traffic. This resulted in our web traffic failing to be served to the Internet, causing a wide-spread outage.

Ultimately, we were able to determine that the router was overloaded by traffic caused by a single, compromised host. Once that host was disabled we were able to restore traffic to all unaffected hosts starting at 6:15 pm. It is important to note that at no time was the affected host able to propagate itself within our network. This is due to the advanced segmentation of our network designed to mitigate against such a risk.

Compounding the issues, the nature of this incident presented to us many symptoms simultaneously. It was only by the systematic troubleshooting of each symptom (and implementing each solution in turn) that we were able to discover the true root of the problem and restore service.

We very much appreciate our customers, and we do not take for granted the trust that we had earned over the years. We also understand the seriousness of this situation.

We did not have access to some of the communication tools we have come to rely on during service interruptions, and have found that there is room for improvement. To that end we are reviewing better ways to communicate to our customers in the future, and will begin implementing them immediately. 

We are taking this incident very seriously, and hope that we will enjoy your trust, not through any statements or promises, but through our actions and continued quality of service.

Sincerely,

Michael Villa
CEO and Founder