Yesterday, August 23rd was a difficult day for Dovetail, but even more so for our valued hosting clients. We discovered early in the day that our primary Internet router was not responding to network traffic. This resulted in our web traffic failing to be served to the Internet, causing a wide-spread outage.

Ultimately, we were able to determine that the router was overloaded by traffic caused by a single, compromised host. Once that host was disabled we were able to restore traffic to all unaffected hosts starting at 6:15 pm. It is important to note that at no time was the affected host able to propagate itself within our network. This is due to the advanced segmentation of our network designed to mitigate against such a risk.

Compounding the issues, the nature of this incident presented to us many symptoms simultaneously. It was only by the systematic troubleshooting of each symptom (and implementing each solution in turn) that we were able to discover the true root of the problem and restore service.

We very much appreciate our customers, and we do not take for granted the trust that we had earned over the years. We also understand the seriousness of this situation.

We did not have access to some of the communication tools we have come to rely on during service interruptions, and have found that there is room for improvement. To that end we are reviewing better ways to communicate to our customers in the future, and will begin implementing them immediately. 

We are taking this incident very seriously, and hope that we will enjoy your trust, not through any statements or promises, but through our actions and continued quality of service.

Sincerely,

Michael Villa
CEO and Founder

RESOLUTION: 11:00am - Our upstream provider has indicated that the temporary repair was not necessary and that the routing issue which was affecting not only Dovetail but Charter Communications, Comcast and other large ISPs with peering locations in Boston has resolved itself. Thank you for your patience with this matter.

UPDATE: 9:40am - Our upstream provider has disabled their COGENT link as a temporary repair. Web sites hosted with Dovetail which were unavailable should begin to be visible again as this routing change is distributed through out the Internet. Once a permanent fix in in place, we will provide further information.

UPDATE: 9:15am - A routing issue has been discovered with COGENT which may be impacting Internet service through out the area. We are continuing to work with out upstream providers to resolve this issue.

INITIAL REPORT: 8:30am Several customers are reporting possible upstream Internet issues which may be impacting web hosting for some locations.

We are currently aware of the issue. It appears to be upstream from Dovetail's main hosting facility and we are working with our upstream providers to resolve the issue.

Thank you for your patience. We will provide further information as it become available.

• 02/04/2009 08:30pm - Full service was restored to 100% capacity after Dovetail's ISP completed repairs on the broken fiber line. Subscribers to Dovetail's AVOID Emergency Page Service have switched sites to a normal state.
• 02/04/2009 05:00pm - Dovetail's ISP has located the issue with the fiber optic cable - a kink due to a downed limb from last night's snowstorm. Current estimates puts the repair at 2 hours or 6+ based on the severity of the kink. ISP has promised to update Dovetail within the hour with more solid estimates.
• 02/04/2009 03:24pm - ISP still attempting to successfully splice Dovetail's fiber optic cable. In an attempt to help stabalize web hosting service and provide more pertinant information, Dovetail has forced A.V.O.I.D. to display the maintenance page for all subscribers.
• 02/04/2009 11:45am - ISP attempting to migrate us to a spare pair of fiber optic cables. ISP discovered a break in their fiber splice. If the migration to the spare pair does not work, it should only be a few hours to resplice the broken fiber cable.
• 02/04/2009 9:45am - Spoke with upstream ISP again. Issue appears to be in the fiber optic cable between their facility in downtown Shrewsbury and our hosting facility.
• 02/04/2009 9:00am - Technician determined issue is not ISP equipment in our hosting facility.
• 02/04/2009 8:00am - Spoke with upstream ISP. Technician has been dispatched.
• 02/04/2009 6:00am - Discovered that primary Internet line is down. Logged call with ISP. Discovered that Dovetail VoIP telephone service also affected

Dovetail's hosting services were seemlessly shifted to use our other Internet lines when the issue was detected.  As such, websites may appear to load slower due to increased traffic.

This outage is also impacting Dovetail's Voice over IP (VoIP) office telephones as well.  Though voice mail is still being delivered and responded to via email.

Our engineers are working feverishly to help resolve the issue, for which information will be posted on the network status page at http://www.dovetailinternet.com/default.asp?LINKNAME=NETWORK_STATUS  as it is made available.

Thanks for your patience, and we do apologize for this and any inconvenience you or your users may encounter.

September 19 , 2008 - Dovetail experienced an email server outage which effected some of our clients who use the IMAP protocol for sending and receiving email. The IMAP service was temporarily unavailable on incoming.dovetailinternet.com. While this issue was quickly resolved, we recommend that our users alter their email applications using one of the following server configurations:

All outbound mail queues have been cleared and mail delivery is back to normal for all users.

Thank you for your patience and understanding while we fixed this issue.

Currently some users are experiencing issues while attempting to access mail. Most users accessing mail via webmail may experience slowness in navigating the interface, and may get 500 Internal Errors. About 40% of users may also experience connectivity challenges while accessing mailboxes via a POP or IMAP connection and sending mail. 

Our engineers are working feverishly to help resolve the issue, for which information will be posted on the network status page at http://www.dovetailinternet.com/default.asp?LINKNAME=NETWORK_STATUS  as it is made available.

Thanks for your patience, and we do apologize for this and any inconvenience you or your users may encounter.

Updated: 03:51 pm

As of 2:39pm, this issue was resolved.   We appreciate your understanding.

Updated: 12:16 pm

Currently, connections for many users to the SMTP Outbound mail servers are not resolving correctly, resulting in connection errors and an inability for those affected users to send mail via third party mail clients.

Updated: 12:04 pm

Currently users may experience access challenges while checking email via a webmail interface. 

Technicians are working to resolve the problem. Please note this interuption is only affecting some domains and is limited to the Web based email portal -- Direct connections via Outlook are NOT affected.  All other services are operating normally for subscribers.

We will update this post as events unfold. Network status can also be reviewed here

- The Dovetail Customer Service Team

Updated: 11:47 am

All services are running normally.

Updated: 9:11 am

One of Dovetail's web servers is currently experienceing downtime as the result of a hardware issue.

Technicians are on-site working the problem. Please note this interuption is NOT affecting all sites, and AVOID Emergency Web hosting is operating normally for subscribers.

We will update this post as events unfold. Network status can also be reviewed here

- The Dovetail Customer Service Team

From time to time someone asks what's going on this quarter with the hosting environment.  It is a great question and we are so busy we don't always stop to announce what we're doing!  So I thought I'd take a minute to post this list of strategic projects already underway...

Current Dovetail Internet Technologies Hosting Initiatives:

• Extending secure off-site storage practices.
  • New arrangements for data archiving and storage will enable us to extend the benefits of off-site storage.
    
• Increasing the capabilities of the core network infrastructure to gigabit bandwidth capacity.
  • Replacement of the core network with higher capacity switches will enhance performance and increase internal traffic speed.
    
• Continued improvements to resource management tool sets.
  • Many tasks performed manually by our staff will be automated such as user reports, domain reports, disk usage, etc.
    
• Increasing backup capacity.  
  • Having complete the implementation of a gigabit SAN we will be adding more backup drives with higher capacity, updating the backup software and replacing the backup servers.
    

Have a wonderful day!

--Mark