Yesterday, August 23rd was a difficult day for Dovetail, but even more so for our valued hosting clients. We discovered early in the day that our primary Internet router was not responding to network traffic. This resulted in our web traffic failing to be served to the Internet, causing a wide-spread outage.

Ultimately, we were able to determine that the router was overloaded by traffic caused by a single, compromised host. Once that host was disabled we were able to restore traffic to all unaffected hosts starting at 6:15 pm. It is important to note that at no time was the affected host able to propagate itself within our network. This is due to the advanced segmentation of our network designed to mitigate against such a risk.

Compounding the issues, the nature of this incident presented to us many symptoms simultaneously. It was only by the systematic troubleshooting of each symptom (and implementing each solution in turn) that we were able to discover the true root of the problem and restore service.

We very much appreciate our customers, and we do not take for granted the trust that we had earned over the years. We also understand the seriousness of this situation.

We did not have access to some of the communication tools we have come to rely on during service interruptions, and have found that there is room for improvement. To that end we are reviewing better ways to communicate to our customers in the future, and will begin implementing them immediately. 

We are taking this incident very seriously, and hope that we will enjoy your trust, not through any statements or promises, but through our actions and continued quality of service.

Sincerely,

Michael Villa
CEO and Founder

RESOLUTION: 11:00am - Our upstream provider has indicated that the temporary repair was not necessary and that the routing issue which was affecting not only Dovetail but Charter Communications, Comcast and other large ISPs with peering locations in Boston has resolved itself. Thank you for your patience with this matter.

UPDATE: 9:40am - Our upstream provider has disabled their COGENT link as a temporary repair. Web sites hosted with Dovetail which were unavailable should begin to be visible again as this routing change is distributed through out the Internet. Once a permanent fix in in place, we will provide further information.

UPDATE: 9:15am - A routing issue has been discovered with COGENT which may be impacting Internet service through out the area. We are continuing to work with out upstream providers to resolve this issue.

INITIAL REPORT: 8:30am Several customers are reporting possible upstream Internet issues which may be impacting web hosting for some locations.

We are currently aware of the issue. It appears to be upstream from Dovetail's main hosting facility and we are working with our upstream providers to resolve the issue.

Thank you for your patience. We will provide further information as it become available.

Last Friday we jumped in to the fray and purchased several iPads from Apple. (We’d actually like to think that we purchased the one-millionth iPad, but probably not).

We handed out the devices and charged the team to do two things:

1. Integrate them into the day to day tasks here at Dovetail -- to find new ways to solve old problems -- to be more productive and to explore.

2. To innovate -- to use the experience to think about the web differently than we do today -- to discover ways to build upon the iPad as a framework for how our web will evolve and how we can revolutionize the work we do for our clients.

So with that, some bleary-eyed folks walked in this morning, and while we as we met around the conference table with a cup of coffee we were all intrigued about sharing and hearing the experiences of the last 48 hours as unapologetic Apple junkies.

I’m not sure what this experiment will yield, but I do know that such an exercise is a worthy investment for Dovetail to find out.

Time will tell and we shall see, and I’ll be posting throughout right here, so check back.

Do you have an iPad? Do you want one? Are you trying to figure out why anyone would? and Share your thoughts.

-- Mike

We continue to get more requests about the Regulation E opt-in process on our banking customer websites. To get up to speed, you can read my first post last week on recommendations and the second one this week “in English.”

Since then, we’ve prepared a sample page on the Dovetail site that provides an example of the combined technologies for a possible form. To check it out, just follow the link to the Regulation E Overdraft Opt-in Form Sample.

-- Mike

I’ve received quite a bit of feedback on my post from last week on recommendations for implementing a Regulation E opt-in form on your banking website. My favorite though was a request to see if I could re-write it in “English.”

I am certainly guilty of sometimes talking a bit too technically on such things, so with a bit of a mea culpa, let’s see if I can explain how the recommended scenario would actually work. And if I feel I have to get technical I’ll footnote it and put it in a “Techie Note” at the end of the post, so feel free to glaze over those if need be.

1. Create an online version of your opt-in form as a new web page.
    
2. Include all the appropriate verbiage recommended along with your overdraft fee disclosures as required along with fields where the visitor can enter their name, account number, date and opt-in or opt-out selection.
    
3. Once a user completes the form, they will click a button to send the request. The contents of the request as entered by the user needs to then be sent securely to the bank. The best way to do this is to send all submissions as a secure email to a designee at the bank or credit union. [1] 
   
   NEVER SEND THE FORM INFORMATION VIA UNENCRYPTED EMAIL. 
    
4. To maximize effectiveness of the online form, and minimize customer support needs, you will want to make the form easy to find and freely accessible to anyone. This will mean that you will need to properly authenticate all requests to validate them as true. [2]
    
5. Next, you will want to work with your web host provider to set the new form to only operate when visited by users with an HTTPS address. This will ensure that any contents submitted are encrypted at the same high standards as your online banking applications. [3] 
    
6. Now, add the page into your website’s navigation so that it can be found in your site’s menus easily. You should also provide the link to your customers in any notifications that are sent to them. [4]

Once you’ve made your form live, you will start to receive the opt-in (or out) requests securely to your bank or credit union. Hopefully this post is a bit easier to follow, but please comment, let me know what you think or if you have any further questions. Thanks.

-- Mike

Techie Notes:

Techie Note 2: Since there are programs (known as “Spam-Bots”) that search the Internet for forms and then submit bogus content, an anti-spam-bot technique known as CAPTCHA can thwart these entries by presenting an image of text that the user has to type in to confirm that the form is legitimate. Below is a picture of what a CAPTCHA form might look like.

[top]

Techie Note 3: HTTPS is driven by what is called an SSL Certificate. A certificate is a digital key that serves two roles, first it handles the encryption between the user and the site server, but second an SSL certificate also verifies the identity of the website owner (i.e. your bank or credit union) and helps to ensure that users are comfortable submitting their information.

Additionally, a specific type of SSL Certificate, known as an Extended Validation (or “EV” for short) has the added benefit of displaying green over your address bar in the browser (an example is seen below) and goes through more rigorous validation than a normal certificate.

  [top]

Techie Note 4: a friendly page address can go a long way towards making the form easy for customers to find. Perhaps an address like http://www.yourdomain.com/overdraft-opt-in might work well.  [top]

On July 1, 2010 the Regulation E rules established by the Board of Governors of the Federal Reserve take effect. By then all financial institutions must have in place controls that opt-out all customers from overdraft protection for ATM and one-time debit card transactions if they have not affirmatively consented, or opted-in for the service.

Over the past couple of weeks, we have received a number of inquiries from our bank and credit union customers looking for guidance on how to create online forms that can meet the opt-in and opt-out requirements of Regulation E.

Based on our research on the topic and various conversations with customers here is what we understand.

1. Before the compliance deadline, all customers must be set to an opt-out status for specified overdraft services.
2. Notification must be sent to customers instructing them that to continue the specified protection, they must opt-in.
3. The institution must provide a full explanation of the overdraft protection including all fee disclosures.
4. Customers can then indicate their consent for the protection or continue to opt-out.
5. Assuming that the first two requirements are met by the bank or credit union, they may direct customers to an online consent form in order to opt-in.

Suggested wording of the form is provided by the Federal Reserve and should include the ability to select the opt-in or opt-out status, the customer’s name, the date, and the customer’s account number. In order to request this information online, special attention must be paid to security.

The following recommendations offer a guideline for your online form.

1. Ensure that the form is only available through SSL encryption.
2. In order to help the customer verify the validity of the form, an Extended Validation SSL certificate (such as a VeriSign Secure Site with EV) is also recommended.
3. Use secure email to deliver the form contents. Sign and encrypt the email using a personal certificate associated with the recipient email address (such as a VeriSign Digital IDs for Secure Email).
4. Include a form validation such as the free reCAPTCHA anti-bot service to reduce the number of false form submissions.

Financial institutions that are users of the novo for Banking web content management system can meet all of these recommendations. If you’re not currently using novo for Banking, there are implementation steps that can be applied to your site as well.

If the above recommendations cannot be met for whatever reason, then we would further recommend that the website provide only a PDF version of the form which customers can print, complete, and deliver it to a branch personally.

For assistance, contact any of us here at Dovetail and we’ll be happy to help ensure you’re ready for Regulation E.

-- Mike

For anyone in banking, regulations are a constant, and rightfully so. However that doesn’t make it any easier for those who are responsible for ensuring compliance.

As you know, changes to “Regulation Z” (which became effective yesterday, October 1st) are impacting the way financial institutions have to post certain product and rate information.

As the go to web partner for over 40 banks and credit unions, we’ve seen a fair increase in support calls for users of our RateDisplay rate web publishing system. Specifically users are needing to alter the way certain tiered rate products are listed on their tables.

The great news is that we’ve got you covered. With the latest release of RateDisplay, version 3.11, users can now create a custom rate field. By using that field and making a few adjustments to the individual rate products, you can present these rates in a Reg-Z compliant manner.

If you are already on version 3.11, contact our Customer Service team to find out how you can rework your rate tables.

Over the coming week we will be contacting our clients not currently on the latest version, to discuss what your upgrade path is and what needs to happen. You may also contact us at anytime if you'd like to get the process started sooner.

-- Mike

follow me on Twitter: @mikevilla

On Tuesday, the National Credit Union Administration (NCUA) issued a Fraud Alert indicating that fraudulent letters were being circulated to credit unions along with two compact discs labeled as training materials which the letter instructs recipients to review.

The release goes on to warn:

“DOING SO COULD RESULT IN A POSSIBLE SECURITY BREACH TO YOUR COMPUTER SYSTEM, OR HAVE OTHER ADVERSE CONSEQUENCES.”

And further instructs that “Should you receive this package or a similar package DO NOT run the CDs. You should contact your NCUA Regional Office or the NCUA Fraud Hotline at 1-800-827-9650.”

You can view the original alert here and view the bogus letter here.

-- Mike

The other day, a White House special committee released information predicting the potential far-reaching impact of the H1N1 Swine Flu and urged businesses to prepare for a potential pandemic.

The next day I received a call from a customer wanting to understand what the impact would be on his company’s website in a pandemic. I have to admit that I was a little thrown off by the question. It wasn’t something I expected to be asked about and wasn’t immediately prepared to respond either.

Upon further reflection after the call, though, I had the opportunity to review our standard Disaster Recovery Plan and service offerings and was able to bring the unique question into standard operating elements. That is probably best advice I can give.

If you’re responsible for such planning in your organization check see what happens according to your current planning. See how they relate to the potential effects of an outbreak.

There are plenty of resources online from one extreme to another. The article linked above offers some good starters and the World Health Organization has a complete Pandemic Preparedness guide available for download.

What are your pandemic plans? Are you thinking about it? What are your thoughts? Comment here and let me know!

-- Mike