Blog Post

By Richard A. Johnson | March 14th, 2011
Tweet

Site is back up

By Richard A. Johnson on Published: March 14, 2011 at 5:40 pm

Wow, what a crazy day. Yesterday there was amazing discussion concerning the new laptop research I am doing since my main development laptop was stolen recently. I woke up this morning to absolutely 0 emails pertaining to comments from people, and right away I knew something was wrong. Sure enough, my web host, Dreamhost, had a switch puke more than 14 hours ago. It is now around 23:23 UTC and my site and services are back up.

OK, so a switch puked, it happens, but 14 hours of downtime? That doesn’t happen, not even in my house! Oh, the excuse that Dreamhost provided was way beyond pathetic too.

As we posted above, one of our main switches in one of our datacenters locked up and needed to be rebooted. When it came up, it was missing its configuration files. This caused all kinds of routing issues on our backend and resulted in many vital servers to lose their connections. Unfortunately, there was no way to recover the configs, so they had to be rebuilt from scratch. Most of the day was spent reconfiguring the switch, so we didn’t have any new updates other than what was already posted above.

What is wrong with this picture? I used to manage one of the largest, private, data centers in the US about 10 years ago. Sure, we had switches puke, routers puke, servers puke, the OC-48 puke, you name it. Our longest recorded downtown? Less than 1 hour! 99.9% of the time when a piece of hardware puked, we had redundancy, so you, as a client, didn’t even realize we just suffered a major hardware glitch. Oh, and our configs, mostly Cisco configs of course, we stored them all in CVS (I think we used CVS back then, can’t remember what it was to be honest). SunStore! That is what we had all of our backup stuff, I remember that big ass robot now :) If I had to reload a config, I didn’t have to physically get up and do anything, all of the techs could do the work right from their desk.

Dreamhost, I love you still, you have been great to me, but your PR people, managers, and techs seriously need a review. I don’t get how this one config was never backed up, it is hard for me to fathom that nobody cared to save this config other than the switch it was a part of. I know some of your techs, they are pretty damn smart, and that is why I bring up PR. Either you didn’t pay a bill and lost connectivity to the LA data center and your PR people drew up the worse excuse ever, or you guys dropped the ball to one of the biggest IT blunders known to mankind. BACKUP! BACKUP! BACKUP!

Now back to my laptop research. It is still going. I am now looking into cheaper, much cheaper, options, because as I am sure most of you are already aware, money is insanely tight right now. That’s part of the reason I have added the donate/contribute button over in the right sidebar. So far I have had a few generous people help out. Two I have known for a while as long time contributors to Ubuntu, and one was a new person just yesterday. I am forever in your debt, and I promise one of these days I will repay you somehow or someway.

OK, Dell Outlet is freakin’ sexy. I found a few really good deals, but wasn’t aware really that the processors were 1st generation i7 or i5 and not the 2nd generation ones, and some people stated I shouldn’t go with the 1st gen when the 2nd gen ones were available for just a few dollars more. With my money and the donations I have received, I have about $400 to spend, and I know this isn’t enough for a development machine. So I will continue to try to gather some more money and save a little so I can get a development laptop that will help me out and not hinder anything. Better safe than sorry I guess. So, if you still have recommendations or amazing deals that can’t be passed up, let me know!

NOTE: Hey Harald, Oneiric Feature Freeze is July 21st :p See, I know the dates man, so get Phonon rocking, I have some tricks up my sleeve, plus I have to hunker down and help Dave with a ton of doc work and packaging work related to that, oh and translations, my favorite thing in the world! Here I come!

This entry was posted in Personal and tagged . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.
  • http://twitter.com/tonyyarusso Tony Yarusso

    One of the sorts of reasons I ditched DreamHost… They have a history of thoroughly unprofessional responses to things going wrong, along with things going wrong far more than they should. For instance, when I was with them they had a tech working remotely load a config into the firewall that locked EVERYBODY out of everything for the entire datacentre, including the tech, so we all had to wait while he made the two-hour drive over to the datacentre to fix it. Meanwhile, there was a multi-month issue with a fileserver being overloaded and failing, making things all sorts of wonky with completely inadequate diagnostic work such that it didn’t get actually taken care of until things basically went down in flames one day. The real kicker that made me close my account though was back on the sales/PR side. There was a “one week FREE trial” being advertised all over the dashboard for their DreamHost PS service, and I went ahead and clicked it. When the trial expired, I decided it wasn’t worth it, and filed my ticket to be switched back to my regular plan. Thus began a long, long, long exchange where they INSISTED that my sites were using too many resources to be allowed on their shared hosting. This was a baldfaced lie. So, my options became to pay $120/month to stay on the PS service with the artificially inflated resource usage they were claiming I had, or close my account. Obviously I did that latter. Interestingly enough, after I had moved everything off, with not a single site, file, script, or process still on DreamHost, they still claimed I was using too many resources to be switched back to shared hosting. Ultimately I got a partial refund of my money (my account was prepaid for another year and a half), a YEAR later, after jumping through many, many layers until someone in management saw my full support thread and became horrified. Of course, by that time I’d been a happy Linode customer for many months, which is where I still am now.

  • Akshat Jain

    They made the Project Bossanova site lose some precious traffic :(

  • http://maco.myopenid.com/ maco

    The deciding factor in my getting a Rackspace account was that it’d only cost me $1-2 more per month than Dreamhost while allowing me full control over how the system ran and what was installed, and none of the risks of someone else on the shared system getting owned and that making its way to my directory.

  • http://www.nixternal.com Rich Johnson

    If my site was making some money that would allow me to afford bumping up to something like Linode I would definitely do it, but there is no way it is worth for me at this time, unless of course I am totally missing something.

  • http://www.nixternal.com Rich Johnson

    I have looked at Rackspace and I can’t find any hosting plan that would be $1-$2 a month more than my DH account. Plus right now, I am just paying for my domain name and the hosting is free, so I won’t bitch to much about the latest issue, just as long as they do a bit better. I have had really good history with Dreamhost. I spent 4 years on the Mt. Dew server, which hardly ever went down, and if it did, it was for a late night reboot after an upgrade, so maybe a couple of minutes at most.

  • http://www.nixternal.com Rich Johnson

    Yeah, totally sucks. I will be interested to see how they repay their customers after this screw up. They owe everybody something, and not saying they owe them money, but they at least owe everyone improvements in their infrastructure. I would definitely can whoever dropped the ball too, no ifs, ands, or buts!

blog comments powered by Disqus

  • Subscribe to nixternal.com

     Subscribe in a reader

    Or, subscribe via email:
    Enter your email address:

  • Archives

semidetached
semidetached
semidetached
semidetached