Blog Post

libc6: nightmare or just a bad dream?

night · mare [nayht-mare]

  1. a terrifying dream in which the dreamer experiences feelings of helplessness, extreme anxiety, sorrow, etc.
  2. a condition, thought, or experience suggestive of a nightmare: the nightmare of his years in prison.
  3. (formerly) a monster or evil spirit believed to oppress persons during sleep.

That is the definition of nightmare according to Dictionary.com.

How many people had those feelings expressed in the definition? Well, from looking at the bug report, forum thread, mailing lists, and IRC, I think some may agree it was a nightmare. I am here to tell you that it was nothing more than a bad dream however.

A bad dream? Rich, you have got to be out of your mind! True, but I am always out of my mind, nothing new here.

Lets think about it for one minute. This libc6 issue didn’t occur in Dapper, Edgy, Feisty, or Gutsy, it occurred in Hardy. Hardy is currently in, what we like to refer to as, a development cycle or unstable release. Did it render our systems useless? Of course it did, but guess what? We have these great things called Live CDs, not only so you can test out Ubuntu or Kubuntu, but so you can also fix breakage that comes your way. Think about other operating systems where you don’t have this option. Sure, they may have a rollback option (which I am sure we will see eventually in Linux), or you may be one of the lucky ones where selecting “repair” from the installation CD fixes your problem, but most of the time with those other systems, you have to do some R&R, and I don’t mean rest and relaxation, I mean reformat and reinstall. Thank goodness for those Live CDs though, as I have seen more people fix their Windows machine with a Live CD more so than they have with the Windows CD or DVD that they paid hundreds for.

But still, MY SYSTEM DIDN’T WORK AFTER THIS UPGRADE! THAT’S RIDICULOUS! No it isn’t, you are using a distribution release that is meant for developers and for those who are not faint of heart.

OK FINE! BUT WHY WASN’T THERE TESTING DONE ON THIS!?! There was, it broke your system didn’t it? You tested it, now the developers know it didn’t work. Don’t say we didn’t warn you.

Pre-releases of Hardy are *not* encouraged for anyone needing a stable system or anyone who is not comfortable running into occasional, even frequent breakage. They are, however, recommended for Ubuntu developers and those who want to help in testing, reporting, and fixing bugs.

Hey, don’t get me wrong, I was just as pissed off as many of you were, and actually on a couple of occasions spoke before thinking when this happened last night. We are human, we make mistakes, and we learn from our mistakes. With this being the only major issue I have seen come across during the Hardy development cycle, I have to say, we have learned a lot over the past few years if only one nightmare, or really a bad dream, occurred during a development cycle.

So remember, before you go off saying “I thought Ubuntu had a QA process”, or “Why wasn’t this tested?” Remember, in a development cycle, the people using and working with the OS are the QA process and they are the same people testing it.

The FIX took me no more than a few minutes in order for me to get my system(s) back up and running, probably less time than it took for me to bitch on IRC last night, or more time than it took some of you to bitch in an email, a bug report, or a forums post. I was always told that cooler heads prevail, and I seen that first hand. The people in the forums last night with the cooler heads, they had a quick howto on fixing the problem in a matter of minutes, they didn’t flip out claiming their company went out of business or something dramatic because an unstable version of their OS broke.

So with that said, this was nothing more than a bad dream ๐Ÿ™‚

This entry was posted in Development and tagged . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.
  • jw

    I don’t get the spazing over this. Hardy is in what… Alpha? We should expect breakages.

  • There is a problem with this and that is that changes are made and then they are sent out in the wild with nobody actually testing the real deal package on a real system. I don’t think minimizing the issue because this is a development status release holds much weight because when the product is out the door the rules change and emphasize no changes being better than changes that fix things.

    This a problem that would have never been out in the wild if one person would of installed the real package before it was released. Without that real testing there’s no Q&A and the common line with this problem seems to be “it’s a development release so expect breakage” and it’s been repeated over the years and for many different distros.

    There is a simple solution for this issue that came up, require that the actual released package for all updates be tested on a live system to ensure that it installs and doesn’t render a system useless. This is a process that can’t take more than a few minutes for any package and I don’t think any one developer has so much to maintain that this task would be a burden on them. This avoids fixing things with a livecd because that’s similar to going out and not caring if you get a flat tire because you have a spare in the trunk. ๐Ÿ˜‰

    Testing should focus on functionality and user experience not if they have to worry about broken packages that are beyond their control or ability even. Not everyone can code a program or even cares to understand it and the bar shouldn’t be set that high to be able to test a pre-release in an attempt to provide useful feedback to improve the product for the users, the common users even.

  • Hi Richard,

    I work in the QA department of a software company, and let me tell you that the developer *should* test the product before give it to QA, why? because here, that’s is a irresponsibility that means our company will loose money in development time, while the developer should just test that once, what the Ubuntu glibc6 maintainer did is like give a program to QA without even run it!… I’m wondering… who is the maintainer of glibc6?, did he test the thing?, was it broken for him?, if it was why did he release it?.

    And some other people pointed it very well, why does nobody pre-tested(developer testing) it if it is a very important package?, was it just a “let see how it goes!” if it is why you don’t do that with Pidgin 2.4 for example? or Banshee 0.99-pre? or GCC 4.3?

    I think you’re trying to cover the sun with a finger, this is a big problem and Ubuntu community must not allow this type of development practices.

  • MrDudeMan

    I’m not here to argue whether this is appropriate for an alpha release or not.

    But what I would like to say is, even though stuff like this happens, we still need a larger group doing the testing. Just because stuff like this happens occasionally, doesn’t mean we should try and scare people away from alphas (of course, not on production machines) if they want. We could use the extra eyes.

    There were workarounds posted. They took a little while to set up. Anyone running alpha releases should accept breakage now and again, that’s inevitable. But the vast support network that is the Ubuntu Community makes this less harsh. A good number of people, if they hopped on a liveCD and did some research, can fix their systems.

    I guess what I’m trying to say is: Don’t be afraid of alpha releases if you’ve got a good head on your sholders (and you stay smart when faced with bugs), and are willing and able to be guinea pigs of sorts, but guinea pigs with a ton of other guinea pigs who are able to help you out.

  • The fix only cost me about 5 minutes of time, so no big deal. I upgraded to Hardy knowing to expect things like this to happen occasionally and specifically to help find and identify these types of problems so they can be fixed before the final release. If the bug had nuked my home partition, then I might be a little bit perturbed…

  • The developer did test and it worked fine, however the default value for the LDFLAGS were changed in the toolchain and wasn’t picked up until prior to the buildd’s building libc6. It was these changes that caused the issue. It’s is in the past now, we have learned from this and will move on. There isn’t one software company, no matter their QA process that doesn’t break stuff after it goes through QA, impossible to catch every issue, even the major ones at times.

    Think Microsoft updates that took out Small Business Servers all around the world, or OS X updates that totally rendered 64bit apps useless, or the NVidia nForce updates that took out networking support causing one to download the fix on a another computer and burn it to cd or a floppy drive. Even the largest technology companies in the world make mistakes and guess what? They learn from it, some better than others.

    As for it being a development release and possibly breaking things, live with it. Whether it went through a proper QA chain or not, it got fixed and fixed quickly. There are many bugs reported on launchpad that I can’t confirm as it doesn’t mess with my system, so it is difficult to find all of the bugs as every system is different, even in the software company environment. What may have worked through QA, once it gets to the wild, may cause more damage than good.

  • Pretty good explanation, that what I wanted to hear, thanks. And this problem only makes Ubuntu community better!, so we are cool.

    I like the way you said it, it was a problem and it was a mistake.

    And also I wanna point out, that it wasn’t a 5 mins fix for me, because vista took forever to boot!!!, gosh!, so I can login to IRC to get the workaround.

  • @Igor: hahaha, Vista took forever. I think I just wet myself laughing. I was lucky, I have Foresight on my desktop in super-duper-dualx4-boot mode with a bunch of different distros. However, my current Foresight install has a bunged SSL, so I couldn’t get to any https sites with it.

    Exactly right too Igor and this problem only made the community better. Just like the old saying:

    What doesn’t kill you, only makes you stronger.”

  • This is right from UBUNTU.COM/testing/hardy “Note: This is still an alpha release. Do not install it on production machines. The final stable version will be released in April 2008.” First of all the /testing folder should give you a clue, DON’T RELY ON THIS alpha version. Only for testing so if you want stability USE GUTSY or wait for the HARDY final release.

  • This is a good attention getter, I like your blog. If it wasn’t for my 3 different GNU/LINUX boxes I’d have no way of being constantly connected. I have a some sort of breakage at least once a week because I test alot of ALPHA releases. My Linux Mint 4.0 boxes based on Gutsy have been very stable, though I did crash them once by maxing out BOINC world community grid.

  • Tom

    Well ..
    i think bugs like this (for base packages) can be easily avoided. Just have virtualbox machine that has to eat the update and do a whole reboot.

    General rule:

    Only send the update if you see X11/login on the vbox ๐Ÿ˜‰

    Really a bit lame ..

  • Damn… thought I dodged a bullet on this problem yesterday. Turns out my wife ran the updates at just the wrong time and got bitten by it. It’s my fault really for moving her machine up to hardy before release…

  • @Tom: all of my software I do similar, however I don’t use virtualbox. I use my own production laptop. Then I do a test build and install on my amd64 box. I feel safer knowing that the package I have uploaded does the following:

    1) Installs w/o any errors
    2) Uninstalls w/o any errors
    3) If system update, then reboot and login then test otherwise test and make sure it isn’t a regression from the previous release. That is why I tend to work on software/pakages that I use. That way there it is easier for me to notice regression.

    A lot of the developers already do this as well. This one supposedly built and test fine locally, but once it hit the buildd’s, it produced a totally different outcome.

    @Michael R. Head: You got it all fixed up already? Fairly easy to go through and fix. A wise man once said, never piss off your wife by installing software the could break things ๐Ÿ™‚

  • Tom

    Oh sorry,

    i meant the actual update developer at Canonical. He should have done the virtualbox thing. Packages for LTS ( even an alpha ) shouldnt have these obvious and devastating effects. I will add my idea to the lauchpad bug ..

  • @TOM: You are so wrong, That’s what you are supposed to expect in alpha software. Alpha is for testing so if things break they get tested by alpha testers (any one using the alpha version) if the test breaks something then it’s reported and fixed. This may have been a lame oversight for an alpha version but it’s not at release candidate stage for a reason.

  • jim

    I don’t have a clue what happens with the devs when they release these things or what testing they do; however, if you are running hardy right now you ARE a tester too. Thanks for your help. Please report any bugs you encounter so they can be fixed.

  • Tom

    @CLINT:

    So you think alpha software is about running into breakage on purpose. Interesting. I think if bugs like this can be found by an automated system they shouldnt reach alpha testers. Most alpha testers want to help but it is not like most of them like having their system broken on purpose. You prolly do, but i dont.
    I think running packages through a quick automated test before you send them to lots of people lets you concentrate on more important stuff cause you do not spend time on the easily fixable problems.

  • Subscribe to nixternal.com

     Subscribe in a reader

    Or, subscribe via email:
    Enter your email address:

  • Archives


semidetached
semidetached
semidetached
semidetached
%d bloggers like this: