Nothing like learning that Sector Link (a web host in Michigan) is apparently having problems with its DNS or servers. We’ve got many, many websites completely down as a result. Best guess I have is that the problem began right before 10 AM EST. Hard to be sure. Even their own website is down. Phone calls to their toll free number (1.877.634.HOST (4678)) result in a busy signal (after it rings once or twice) as do calls to their direct number (1.810.213.4640). I’ve been tempted to try their fax number, but I wouldn’t be surprised one bit if the same problem was there. Ftp’ing any site hosted with SectorLink results in no connection.
Got to love it. Makes me wonder if it’s time to change. I’ve been relatively pleased with SectorLink for a couple years now (former customer of Ipower), but due to situations like these (especially on Cyber Monday with clients losing potentially hundreds of thousands of dollars with a “down moment” like this), I’ve been gradually shifting to HostGator. No web host is perfect, but this kind of delay (almost 3 hours as of this post) on such a crucial ecommerce day is painful. But the guys at SectorLink stand out from the crowd – that’s the whole reason why I went with them to begin with – their customer service actually answered the phones & responded to emails without waiting several days like my former host. No, I’ll wait this one out. HostGator has some benefits (larger), but SectorLink still has the small host customer concern.
Here’s a helpful thread in a forum about the current situation at SectorLink.
Recent post in the above forum: “What we were able to find out was that a main pipe broke and flooded the electrical circuits & backup generator. They have backup-backup generator that an electrician is installing now. No server hardware was affected. Should be back online shortly. We apologize for the inconvenience.”
From the same forum, a message from someone who popped by the Sector Link facilities: “The water main break is a big mess. I don’t usually see them get out the big backhoe’s, and have consumers on site for a normal water main break. I was just downtown where they’re at. When I was there (25 minutes ago) and they were just waiting on the electrician. No power though means no phones which is why they’re down. They did not give me an ETA when I was there.”
Another thread from the same forum (posts should be made in the first forum link, though)
This website review page is full of some heated comments about today’s downage. One interesting comment from that page sheds additional light on the situation at hand: “Just say this on the noon news in Flint – there’s a power outage and water main break in downtown Flint. Must be the cause of the failure. We just received half a foot of wet snow overnight. They will probably return when the power comes back, unless the server room is flooded. Let’s hope not!!!”
Here’s a blog post with some comments about SectorLink’s outage.
As of 2:45 PM EST email is starting to come through. Some websites (not sure if all) are up too. Yes!!!
Latest message posted at WebHostingTalk.com: “Mobile backup generator is in place, the power has been connected. All ETS Host servers should be back online now. As for other Sectorlink customers, they are powering them on server-by-server.”
And the official news release from SectorLink itself:
“Earlier today a water main broke under the street just north of the Sectorlink Flint data center, approximately 40 yards from the facility. Local power was lost due to a flood of water that penetrated the underground power room vault(s).
Backup generation went in to affect as soon as power was lost. Shortly thereafter we were advised to cut all power to our facility, due to the possibly of gas distribution shut off. At the time Sectorlink was utilizing natural gas powered generators. Battery backups or generators being cut off unexpectedly is extremely detrimental to equipment, causing surges that can damage components.
The road ended up collapsing and heavy equipment had to be brought in to remove a large portion of the street, which is why this drastic measure was advised. At that point we moved to our secondary emergency plan, which is mobile generation, and a unit was delivered within the hour. An electrician accompanied the generator to hard wire the unit in to the facility.
Sectorlink will be switching back over to grid power and internal backup generator infrastructure at 11:30 p.m. EST. There will be a 15 minute outage while this occurs.
We do thank all of our customers for their patience and understanding. We have been working very hard to make sure that the downtime due to this issue is as limited as possible.”
Note: Friday, February 13, 2009 – Sector Link is down again. The websites are loading extremely slowly while some heavy graphics are not loading at all. SectorLink’s toll free number is throwing people into an endless loop (they apparently use VOIP phones so when their servers are down good luck reaching them). We’ve got clients who aren’t too happy, and we’re putting shoes to our words now. We’re gradually moving our client sites elsewhere. Sorry, SectorLink. Downtime becomes expensive for us too in terms of client retention.
Update Friday, Feb 13, 2009: Just got an email from SectorLink support – We are aware of the problem, it is currently DDOS attack on part of our network. We are working on the problem and it should be resolved shortly.
10:13 AM Sites seem to be back up. Haven’t tested all the servers I have sites on, but things are looking up.