See “‘Side By Side Assemblies’ Bring DLL Hell 2.0” at Slashdot.

So I was tickled to see Jeff Atwood’s blog post on 5/13, “Why Do Computers Suck at Math?” in which he mentions several infamous glitches (mostly fixed now) in the Google and MS Windows calculators, MS Excel 2007, and the ill-fated Ariane 5 rocket. He also gives a summary of floating point and recommends David Goldberg’s “What Every Computer Scientist Should Know About Floating-Point Arithmetic“. You may want to see Wikipedia’s “floating point” entry for a less technical overview.

As usual, the many comments expand and correct his main article. I encourage you to at least skim through them. However, there were a couple things that I did find more amusing than edifying.

First, there was a comment that said floating point math was developed for the x87 coprocessors to compensate for low power and scarce memory. Ahem. That’s right up there with Al Gore inventing the Internet and Bill Gates inventing UNIX. Digging through my own junk closet I found a 1975 printing of “Introduction to Programming” which includes assembly level programming of the PDP-8e’s floating point unit. A little more digging turned up a 1970 copy of Schaum’s Outline Series “Introduction to Computer Science” which not only discusses floating point, but also has you work out fp logic circuits (yes, really: and, or, and not gates)! Wheee, I don’t remember doing those exercises. Oh, to close the circle from the comments, there’s BCD — my 1981 MC6809 Microprocessor Programming Manual shows the MC6809 had instructions for BCD. These were mini- and micro-processors; the actual history of these concepts is much older of course.

More entertaining, however, is the lengthly discussion of whether “0.999… == 1″. Really. Now, to be right up front about it, this is true. 0.999... is 1. None of this “approaching” or “almost but not quite” stuff. Those are two ways of representing a single ideal number: 1. Apparently this is a big deal, kinda like the thermal properties of blankets I also mentioned in the April post. For the short answer, see “Why does 0.9999… = 1 ?“; for a longer discussion, see “0.999…“.

I guess I have the sort of twisted mind in which all of this makes perfect sense. Everything is just layer upon layer of convention, filtering, abstraction, and indirection.

I just recalled a discussion of how computers “really” store numbers that came up while going over converting among decimal, binary, octal, and hex representations. Along the lines of how can “it” tell whether it’s storing a hex or octal number, or maybe even a character, or something else. Hmm, kinda like the “how does a thermos ‘know’ whether to keep stuff hot or cold” — it’s the wrong question; you’re using the wrong filter.

I once had someone dabbling with programming ask me what arrays were good for, since they didn’t understand them at all; I wasn’t sharp enough to give them a good answer. But for “real” programming, it seemed that the big sythe-level filter was pointers, or more generally, indirection. It either clicked or they were lost. Nightmare: an array of pointers to functions returning pointers to structures containing pointers to ….

But I love that sorta stuff; you shouldn’t be surprised that Gödel, Escher, Bach: An Eternal Golden Braid is one of my favorite books.

As a friend of mine says “Well, that explains a lot.”

[T]he manufacturer maintained that the system was perfect, and that revealing the source code would be damaging to its business. They were right about the second part, of course, because it turned out that the code was terrible.
[...]
Basically, the system was designed to return some sort of result regardless.
–Bruce Schneier, Software Problems with a Breath Alcohol Detector

Schneier goes on to say that ‘”You can’t look at our code because we don’t want you to” simply isn’t good enough.’ in the realm of evidence and other legal situations.

Don’t forget the debacle with electronic voting systems, either.

You would think that the primary goal of a programmer would be correctness. After all, whenever I use something, whether software based or not, I expect correct functioning, and if I don’t get that, then I consider the product or procedure broken … but then I’m just a naive consumer. An ideal product would first have to be “correct”, but also relatively accurate (for instance, “a few meters” is good enough for GPS), and performant (a speedometer should display your current speed, not what it was five minutes ago).

However, programmers don’t work in an ideal world, and are rarely in charge. Usually the real priority is the bottom line, and every thing else follows. You probably have seen the meme

1. something
2. ???
3. Profit!

Let me put that in priority order and expand it:

1. Profit!
2. release
3. pretty
4. fast
5. precise
6. correct

Years ago in Windows Developer ^[1] someone from a major software house (heh) explicitly stated that the “user experience” was the most important thing, and that correct functioning should be sacrificed for the sake of a better UX. WTF? I obviously have never been in the target demographic for those products. I think this was the same issue where they kept referring to a current software development product, still in active development, as “legacy code”, pushing instead a still-in-beta product as the only way to go, leaving those dinosaurs (who converted to the legacy system just a year or so ago) behind in the primordial dust. But I’m not cynical.

In the precise category (precision, accuracy, and resolution are unrelated), I recall a 10KVA UPS system I ran across. I thought it was pretty cool, if overkill, that it showed the current flow to a couple or three decimal places … until I realized that it only had so many discrete values that it displayed. The “precision” was meaningless; there were only about three different “.nnn” values ever displayed. From what I remember, the current values were probably a geometric sequence — sort of like the 10% resistor preferred value scale (10 12 15 18 22 27 33 39 47 56 68 82) — with the decimals thrown in just for appearance’s sake.

image: Lady justice standing.png, Wikimedia Commons

1. I kept this particular issue for a long time, due to this article, but I can’t find it now; I’ll update this post if I do

The Parable of the Broken Window describes

a shopkeeper whose window is broken by a little boy. Everyone sympathizes with the man whose window was broken, but pretty soon they start to suggest that the broken window makes work for the glazier, who will then buy bread, benefiting the baker, who will then buy shoes, benefiting the cobbler, etc. Finally, the onlookers conclude that the little boy was not guilty of vandalism; instead he was a public benefactor, creating economic benefits for everyone in town.

The Wikipedia article quotes the original parable by Frédéric Bastiat, refers to the law of unintended consequences, then goes on to discuss the parable’s fallacy, interpretations, and applications.

Something current economists and pundits should keep in mind, don’t ya think?

However, I didn’t discover the parable until after I’d already started working on this post, triggered by reading Bruce Schneier’s “The “Broken Windows” Theory of Crimefighting, which links to a Boston Globe article by Carolyn Y. Johnson: Breakthrough on ‘broken windows’:

The results, just now circulating [the study was in 2005] in law enforcement circles, are striking: A 20 percent plunge in calls to police from the parts of town that received extra attention. It is seen as strong scientific evidence that the long-debated “broken windows” theory really works – that disorderly conditions breed bad behavior, and that fixing them can help prevent crime.

Long-debated since the idea was introduced in The Atlantic’s March 1982 “Broken Windows” by George L. Kelling and James Q. Wilson, which led to the 1996 book Fixing Broken Windows: Restoring Order and Reducing Crime in Our Communities by George L. Kelling and Catherine Coles.

But, again, this post really wasn’t meant to be about crime or urban renewal. Whenever I hear “broken windows”, I’m always reminded of one of favorite short books on programming: The Pragmatic Programmer: From Journeyman to Master, by Andrew Hunt and David Thomas. In this extract they say:

Don’t leave “broken windows’’ (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered. If there is insufficient time to fix it properly, then board it up. Perhaps you can comment out the offending code, or display a “Not Implemented” message, or substitute dummy data instead. Take some action to prevent further damage and to show that you’re on top of the situation.

We’ve seen clean, functional systems deteriorate pretty quickly once windows start breaking. There are other factors that can contribute to software rot, and we’ll touch on some of them elsewhere, but neglect accelerates the rot faster than any other factor.

You may be thinking that no one has the time to go around cleaning up all the broken glass of a project. If you continue to think like that, then you’d better plan on getting a dumpster, or moving to another neighborhood. Don’t let entropy win.

So. I’d intended to quickly jump from a passing reference to Schneier’s post into software development, but got sidetracked and now I’m out of time (or attention span, anyway).

I started out saying that this wasn’t about Microsoft Windows, but I just can’t help myself … perhaps if Microsoft had paid more attention to broken windows, then Windows wouldn’t be so broken.

Sometimes, when I’m looking at the code of a complete stranger, I get that same, weird feeling I get when I’m creeping through my neighbour’s house. Picking up their stuff, looking through their fridge.

That’s from item number 5 (Read the code from an open source project) in Leon Bambrick’s 8 ways to be a better programmer in 6 minutes.. And, if you’re saying “WTF!?” after reading the list, do take a look at his “About” page.

Atwood’s post was inspired by Bill Higgins’ older post ‘the Uncanny Valley of user interface design‘ that says “[W]e must ensure that we design our applications to remain consistent with the environment in which our software runs. [...] I’d agree that software designers and developers generally observe this rule except in the midst of a technological paradigm shift. During periods of rapid innovation and exploration, it’s tempting and more acceptable to violate the expectations of a particular environment”.

Actually what caught my eye was Atwood’s title in my feed reader, because I’ve previously mentioned the uncanny valley, “a hypothesis that when robots and other facsimiles of humans look and act almost like actual humans, it causes a response of revulsion among human observers” (Wikipedia).

It’s an interesting play on the UV concept, replacing “creepy” with “annoying”. I can see how this mismatch of expectations could cause a vague feeling of uneasiness/not-quite-right/frustration for someone who’s not thought much about interfaces.

Don Norman‘s classic The Psychology of Everyday Things (reissued as The Design of Everyday Things), extends the idea to all the “everyday things” we humans interact with.

So what’s your preferred software development metaphor?

image: Battage à Fléau, Wikimedia Commons

As told at New Otto malloc helps spot ancient bug, “Otto Moerbeek recently found and fixed an ancient bug (some 33 years old) in yacc(1). Here is his story….”

image: Dave Pape, Bos grunniens – Syracuse Zoo.jpg, Wikimedia Commons
h/t: Bob Plankers, links for 2008-07-14, The Lone Sysadmin

Echoing what I said in MySQL Backups, some situations may require more elaborate techniques, but these scripts are “good enough” for my needs. I hope you find these scripts useful, and welcome comments, critiques, or suggestions for alternative methods. Note: these are just examples. If you use them, you do so at your own risk. You’ll want to adjust them for your own situation.

I run these scripts ad hoc (via the at command), rather than via cron, for a couple reasons. First, these run on my laptop, so it may be down, or the backup media may not be avalable. Second, my personal schedule is very irregular and I prefer to back up when the system is quiescent. I’ve found the most convenient time for incremental backups is while I’m getting ready for work. I do full backups some time over the weekend.

My schedule, though not rigidly adhered to, is to run incrementals every morning. Because the name pattern includes the day of the week, I generally have seven copies available, although not necessarily the most recent seven days. Currently, they contain any files modified within 30 days.

Full backups use a similar naming scheme, which I should fix at some point. I pick one of the weekend backups each month to preserve long term, and rename it to indicate the specific date. I try to keep about a year’s worth of full backups available.

The first file backed up is /backid.txt; it is also included in the emailed logfile. The create-backid-file script (below) dumps several pieces of information useful in recovering a system into it, such as the output of fdisk -l and /etc/fstab. Because I use cpio, this text can be viewed directly from the archive file with less, head, or even cat. That could be handy if you have limited resources to work with during a recovery.

After the backup itself completes, a verify-crc pass is made over the archive. This may not be as assuring as doing an actual file-by-file comparison, but seems a good tradeoff in my situation between reliability and time. An occasional file restore should be done to confirm that what you think is getting backed up really is. I’ve also done bare-metal restores without problems using these archives.

The actual scripts are listed later, but first I’d like to point out a few things from the log file.

• the backid.txt file is included
• line counts from the backup pass and the verify pass are included; the vast majority of the time, these are the same
• the block count from the two passes are included; these should always be the same
• the size of the archive is calculated
• the logs are kept locally for review as well as being copied to the backup media
• timestamps are used at important steps so I can develop a feel for how the scripts perform

Here’s a sample log:

It’s kinda large because of a VMware disk image and a couple of dvd .iso files downloaded but not yet burned.

This is the full-backup script (text). Things to note:

• the DIRS variable specifies which filesystems to back up; the id file should be first in the list
• nul-terminated paths (options -print0, -va0) are used to avoid issues with unusual filenames
• the VOLNBR is just a holdover from tape and could be removed

The incr-backup script (text) only differs in two lines; next time I edit the scripts I’ll move the interval option to a variable.

Finally, the create-backid-file (text). I’ve stripped out the extra noise from the script for display in this post. Select the text link to get the whole thing. The grub.conf (or lilo) should probably be added as well.

Aside, check out the full-size version of the lead image — it’s high res and quite interesting, along with the Wikipedia article Hard disk drive where I found it.

image: Paul R. Potts, SixHardDriveFormFactors.jpg, Wikimedia Commons

MySQL is a pervasive, lightweight RDBMS that is used with popular applications such as WordPress and Bugzilla. The mysqldump utility which comes with MySQL provides a quick and simple way to create database backups.

While some situations my require more elaborate techniques, these scripts, using mysqldump, should be fine for the most common uses. They are run nightly via cron and keep a week’s worth of backups. Note: these are just examples. If you use them, you do so at your own risk. You’ll want to adjust them for your own situation.

This first script (text) runs on my laptop. Because it accesses the local server, has full permissions, and the databases are small, it locks and backs up everything at once.

This second script (text) runs on my blog host; the actual database is on another host. It differs from the previous script in that the flush-logs and lock-all-tables can’t be used, only a single database is backed up, and a password is required. I’ve also added the uptime command so that I can see how the server is performing when backups are run. Note: I don’t recommend embedding passwords in scripts, but it makes the example simpler.

In both cases, cron emails the log file to me. A typical message looks like this:

I periodically rsync my entire remote site to local storage — having the backups on the same remote host as your live data won’t do you any good if your host loses a disk (and have stale or corrupted backups), or falls off the ‘Net. Of course my own filesystem backups take care of having long term, off-the-machine backups of my local stuff. I’ll discuss the filesystem backups in a future post.

I hope you find these scripts useful, and welcome comments, critiques, or suggestions for alternative methods.

Update (7/7): added link to filesystem backups