Twitter and the Worst Passwords of All Time

This is a post from January 2010, replayed here to make it more accessible.

If you follow security news, you will have seen stories about Twitter and its banned password list, 370-odd passwords which Twitter has embedded in the source code of its registration page so that if you try to use them, it can say that they are "too obvious". Here's one such story. And here's the original source.

As it happens, I spent some of last week reading lists of common passwords (if this sounds grim to you, you should realize that I also spent a noticeable portion of last week reading spam, so you know, after all those misspelled drugs, common passwords were looking good). As a result, the list looked oddly familiar as soon as I saw it. It's "The 500 Worst Passwords of All Time" from Mark Burnett's book Perfect Passwords. You'll find it under that name all over the net, sometimes with proper attribution, and sometimes without.

The Twitter list isn't identical. For one thing, it's shorter, mostly by the removal of passwords under 6 characters; for another, it includes more duplicates (the original list includes "0" twice, probably a formatting error for "000000" and "00000000"). There are some other additions, like "twitter" and "abcdef", probably signs that another list was added in.

You may ask how I KNOW it's the same list. After all, maybe those really ARE the 500 worst passwords of all time, and Twitter came up with them independently, but if that's the case, it's a mystery that they don't match the data from any place else. Here's a summary of three compromised password lists. My first tip-off was "srinivas", eye-catching because it's the name of a colleague, and it's staggeringly improbable as a top 500 password. Sure, it's a pretty common Indian name, but it's the only Indian name on the list, it doesn't occur on any other frequent password lists, and it's not a pop-culture reference with the kind of extraordinary popularity shown by the other entries. There's no way it should beat out "krishna" or "ganesh", popular Indian names with lots more American popularity than "srinivas". So it's more than a little suspicious to see it on these two list. Couple that with the rest of the overlaps, and there's no way Twitter made this list up.

Furthermore, if you look at uncensored password frequency data, you're awfully likely to find exclamation points (particularly on obscenities, which are extremely popular in passwords -- if you see a top 100 list with nothing obscene in it, become very, very suspicious, as they usually make into the top 10). This list has no punctuation, presumably due to filtering at some point. It was almost certainly intended to be used with case insensitivity and common substitutions (Twitter is not using it that way; "srinivas" may be too obvious, but my local Srinivas was quite chagrined to discover that "Srinivas" is A-OK in Twitter's eyes.) The top 500 list, and its Twitter descendent, do include obscenities, although much press coverage edits them out.

In any case, there's no such thing as the 500 worst passwords of all time. The passwords on any given list reflect the password rules of the site, the pop culture of the time (that all-time list was before the band Blink 182 became popular), the popular names and sports teams of the regions the users come from… There are a lot of commonalities from list to list (apparently "password", "letmein", and "123456" never get old), but there's a lot of turnover, as well. My mystic password predictor says "avatar" is on the hit list with a bullet these days.

Looking at lists of common passwords is a fun exercise in social psychology, but if you want to know something about password security, use the longest password you're allowed, and put more than one piece of punctuation, and more than one number, somewhere in the middle. This advice is not original; here's Bruce Schneier's version, with explanation. That will not only keep you off the top 500 lists, it will also go some ways towards actually keeping you secure. The odds are that using your last name will keep you off a short banned list (even if you have a staggeringly popular last name) but it won't do a thing to keep away the password crackers, who really do use information about your login and your real name to decide what to try. Avoiding this kind of list is to actually being secure as avoiding getting onto the freeway in the wrong direction is to being a good driver. It is a start, but it's hard to call it even a good start.

Management Metaphors You Don't Need: The Pig and the Chicken

Do you know the difference between being committed and being involved? When you eat eggs and bacon, the chicken is involved, and the pig is committed!

I would like to note that my present management chain is not particularly fond of metaphors, and has never offered me this one, but it has come up before.

One advantage of this metaphor is that it allows you to say you want people to be pigs, which does briefly catch their interest. Or outrage them.

But what happens if you actually think about the implications? Well, the first interpretation I come up with is "It is not acceptable to merely be willing to sacrifice your children for the project; you have to be willing to die for it." That's icky. Even suggesting it's a good thing is icky. If you'd like a more professional phrasing than "icky", try "That's not a culture I want to work in."

But if you think about it for a bit longer, it doesn't get better, it gets worse. Ignoring the whole "first you die and then they eat you" thing, the chicken is not involved. The chicken and the egg-eater have a mutual interest in eggs, but different end goals; they are working at cross-purposes with an intersection point. The chicken wants eggs as a way to get more chickens, the egg-eater wants eggs for food.

As for the pig, the pig is not at all committed to bacon. The pig is opposed to bacon. The pig, if it could envision bacon, would surely take steps to prevent it. The pig is a purely unconscious participant in the breakfast, which does what gives it pleasure, unintentionally providing benefit to people who then eat it.

The commonality between the pig and the chicken is that they are both fools, being controlled by forces unknown to them and ultimately hostile to them. It is not in the best interests of my management chain to convince me to that I am in either of these positions.

You know who's involved in bacon and eggs? The waiter. And who's committed? Take your choice; the chef, the eater, the restaurant owner. As for chickens and pigs, their lack of active opposition to bacon and eggs indicates their lack of knowledge on the subject.

First Off The Ice Floe

Image from David Schroeder on Flickr

Another bird metaphor for software development.

Penguins and seals have more in common than being adorable. They also live in the same parts of the world, and take part in the same ecosystem. They do not find each other adorable, however. Seals think of penguins as delicious, if feathery; penguins think of seals as terrifying predators.

Luckily, seals don't go very far onto land, and penguins can nest out of their reach. Unluckily, all the food penguins eat is in the water, with the seals. The hungry seals.

This leads to a phenomenon which to a human eye looks like all the penguins standing at the edge of the water, trying to figure out who's going in first. "Is there a seal out there? I think there's a seal out there." "I'm hungry, somebody get in the water." "YOU get in the water." "No, you go first." Eventually a penguin gets in the water, and the rest of them watch: eaten by a seal, or not? If a seal eats the first penguin they all back up. If not, soon there's an avalanche of penguins into the water.

People contemplating whether they should upgrade their software do exactly the same thing. "You gonna upgrade?" "I dunno, you done it yet?" "I hear Joe did it, and he was eaten by a seal." "What, in the middle of our office?""Uhh... I mean it crashed a lot and ate an important file. I have no idea why I said that about seals."

For developers, this means that you need to distinguish very, very carefully between beta testers and the first real users of the application. You have to be certain that people's bad beta-test experiences, when you are sending them out to be eaten by seals, do not keep everybody else from upgrading. And you have to be certain that early and noisy adopters are happy penguins, swimming tranquilly in a threat-fee ocean of delightful little fish. Even if that means giving every one a personal escort.

For end-users, it means being aware of your penguin-like behavior. Not wanting to upgrade? Can you find somebody who did it first? Can you get somebody just like you to do it first? If you've heard about bad experiences, be sure to verify them -- was it really this version of the software? Was it really the software's fault? You wouldn't want to be scared off by some seal-shaped pool float.

Managers and IT people who need other people to upgrade need to use the penguin-like behavior. Start by making sure it's really safe. Pick quiet people who won't complain much, and get them to try it first. Work out all the kinks. Be sure you can guarantee a good experience. Then send in the star penguins and get them to show everybody how great it is. If you don't manage this carefully, you risk having a bad first experience turn into mutiny and misery all around.

They Made Me Do It

Security is a favorite excuse; "We can't do that, because of security". This is usually at least sort of true, but it is rarely entirely true. I present another rant in the form of an extended metaphor.

Suppose somebody takes the dog to the vet, and comes home empty-handed. "I'm sorry honey, the vet said we had to have him put to sleep." There are a lot of ways the vet can feel about that statement, which is almost never absolutely true.

It could be that the vet said "I strongly recommend putting the dog to sleep; otherwise he will just be in pain until he dies." In that case, the vet is going to feel that it's pretty fair to say "the vet said we had to have him put to sleep".

It could be that the vet said "There is a treatment available, but it is expensive, time-consuming, and uncomfortable for the dog, and there is no guarantee it will work. There are people who undertake it, and there are people who have the dog put to sleep; those are both reasonable choices." The vet is likely to feel less happy; the vet offered choices, the dog owner picked one, they were both hard choices. It's true that the vet did not produce any non-fatal choices the owner liked, but still, the vet did not doom the dog.

It could be that the vet said "Oh, we can treat that. It'll cost $100 and you need to give the dog pills every day for a week," and the dog owner said "Oh, I don't want to do that; what happens if I just don't treat it?" and the vet said "Then you'll have to put the dog to sleep." The vet is likely to be upset at the characterization "the vet said we had to have him put to sleep". The vet offered a different choice which most people would accept.

It could finally be that the vet said "The problem here is that you are not being a responsible dog owner; if you let the dog run loose unsupervised and it bites people, we'll have to put the dog to sleep." The vet is likely to be outraged at the idea that this is the vet's fault.

A lot of "we can't do that because of security" is like that. "I'm sorry; you can't do that thing you want to for security reasons" might mean "it's not safe" or it might mean "the safe options are really expensive and onerous" or it might mean "the safe options are more expensive than I'm willing to pay for" or it might mean  "the safe options involve actual effort, and I'm not willing to do anything at all" or it might even mean "I don't want to but I don't like taking the blame." (I would like to think this never happens to dogs, but I am very much afraid that it does.)

How Hard Can It Be?

Footnote: This is a revised version of a post with exactly the same point and the same opening from 2009. Also, it is very closely related to the standards problem, here explained by xkcd.

Of all the reasons that people get sucked into rewriting software, the one that drives me craziest goes like this: "Well, there are a bunch of versions out there, yes, but they all suck. So we decided to write our own. How hard can it be?"

Well, I dunno, it can be hard enough that a bunch of other people tried it, and their versions all suck. It's true, the world is full of mediocre programmers, but take a look at those versions. 

One commercial version might have been written by 2 bozos in a corner and then shamelessly exploited by a money-grabbing company trying to fill a vacuum. More than one? Longstanding products? Somebody who can do more than grunt was involved at some point. 

One open source version might have been written by a crackpot and released in a forlorn attempt at self-aggrandizement. More than 5? Some of which are longstanding projects with multiple contributors? Once again, a reasonable amount of brainpower has been unleashed here. 

If the result sucks, the problem is hard. And unless you have a brilliant insight they didn't have, your version is also going to suck. (If you do have such an insight, you do not ask "How hard can it be?" You ask "How come nobody else ever tried [fill in the blank]?" and let me warn you that this is not a rhetorical question, because sometimes there are very good reasons.)

Why do some problems -- time card systems, bug tracking systems, trouble ticket systems, monitoring software, to name some of my more frequent nemeses -- suck people into believing they cannot be hard when manifestly they are? In general, some combination of the following features:

1. Every environment has some customization that is, in itself, relatively trivial, but is sufficiently environment-specific that it is hard to implement in a program meant for other environments. When you contemplate the programming difficult, you calculate the difficulty of adding that customization, not the problem as a whole, forgetting all the difficult problems that the existing system has already solved.
2. Small and specific versions of the problem are easy to solve; scaling and customization are the hard parts. You can solve your organization's whole current problem easily and will not discover the lurking traps until later (the timecard programs works great until employees start moving between time zones, the bug tracking system works fine until a million bugs, the trouble ticket system works fine until the second group of people need to use it, the monitoring system deals with the network devices fine but stumbles when you need to manage systems).
3. The underlying parts of the problem are relatively tractable, but putting a user interface onto it that  isn't disgusting and hard to use for large portions of the user base is difficult. (Do user interface designer design beautiful interfaces for things that then nobody can program the backend to adequately? Probably.) In this case, even if your program works right, the users will think it sucks. See, for instance, every time card system I have ever been forced to suffer through, most of which seemed quite competent at dealing with my more normal time card needs, after I had learned to perform the arcane rituals needed to actually enter the data. Nevertheless, once your program has made me cry, I do not care that it was just a few hundred UI issues, not a fundamental algorithmic flaw. 

If you actually have a novel insight into one of these problems, more power to you, please do build your new system (although I have to warn you that at best a significant minority of users will still hate it). If it just seems logical to you that there must be a better way, please do not unleash your version onto the universe until you figure out exactly what that better way is. 

And please resist the temptation to believe that it will be cheaper to build yours than to try to beat somebody else's into submission until it is bearable. It may be more personally rewarding, and that may be a perfectly good reason to do it. But it will almost certainly be just as expensive. 

Programming Metaphors You Need, Part 1 of Birds

Some years ago, my mother was sick in bed, and the family cat, apparently feeling she needed perking up, went out and got the best present it could think of. It hunted vigorously for hours, and exercising all the smarts and power it could muster, it found the perfect thing, and brought it back to my ailing mother.

My mother, awoken from fever dreams to find a half-dead bird in her bed, was not appreciative. Actually, she was more horror-struck. There was shrieking. Cat and bird were both banished summarily.

My mother and the cat both sulked for days, furious at each other for the cruel way they were treated when each had behaved as well as one could possibly hope for.

I myself have had an eerily similar interaction with a programmer who proudly showed me a feature which he thought was incredibly useful, and which had also been very tricky to implement. He was outraged to discover that I did not adore it. In fact, my first response was to ask how to turn it off, out of fear that I would trip it accidentally. He felt that I was unappreciative, resistant to change, and failed to appreciate how life-changingly useful this feature would be. (Our disagreement was not improved by the fact that it was possible but not practical to disable.)

He did have at least one thing right; unlike the half-dead bird, it proved unobtrusive. In 10+ subsequent years of using software of the appropriate type, I have never once wanted this feature, and have often been in situations where it would have been dangerous, but I did successfully use his version for several years without accident.  I'm sure he continued to enjoy it, and his virtuousity, to no end. I just wish he hadn't given it to me.

Software is often full of dead bird features. It’s not valuable because it was difficult to implement, or because it makes developers happy; it’s only valuable if it makes the users happy. Save the dead birds for those who appreciate their excellence.


Edited to add:
It's not just programmers that come up with dead bird features, of course. For instance, the ultimate dead bird feature is almost certainly Clippy, the animated paper clip that used to offer to help you with your Microsoft Word documents. It was a masterpiece of technology, lovingly crafted, and beloved by its audience. But lots and lots of people found it not merely unattractive but actually repellent. There you were, working away, when AARGH! your eye was drawn to an animated paper clip, actively trying to distract you from your work in order to offer to help you do something you had no interest in.

Introduction

I'm working on blogging more often and splitting out topics. This blog is for my more computer-related rantings (there's something about computers that inspires ranting). The kid stories are staying at OTOH and the art is at Naming Cats is Easy, Naming Art is Hard.