“because it’s not compiled, it’s also very quick” [1]

It’s a different definition of quick of course, but somehow I begin to feel disconnected and trolling mode wells up from the darkness within.

But… even in that sense, is it really as quick as we think it is? In my experience it certainly makes it quicker to discover new and interesting types of bugs.

The quote relates to Ruby, but really, be it Ruby, Perl, or Python it seems much the same.

[1] I’ve quoted from the 3rd page of an article there. I’m still not quite sure why these sites bother to separate their articles into “pages.” Even SMH used to be one-page-per-article before this pagination fad caught on.

I play with the snake from time to time, Python having now almost entirely supplanted Perl as my hak-n-slash language. But I’m certainly still learning as I go. Today I’ve been trying to work out what’s gone wrong with something I’m doing. I thought I’d done something strange with inheritance, not understanding some case where inheritance causes data to become static (in the sense of static members in C++.) What was really the problem is that I didn’t realise that default parameters to members are kept and reused, and if you go and use a parameter you can end up with state being held on to where you don’t (well, I didn’t) expect it.

I’ve hit this at least once before I think, it is vaguely familiar. Anyway, it’s a bit of a tricky gotcha as far as I’m concerned. Intuitive? I wouldn’t say so. Here’s the example:

#!/usr/bin/python

class Parent:
    def __init__(self, stuff = []):
        self.stuff = stuff
    def doIt(self):
        print " ".join(self.stuff)
    def addStuff(self, stuff):
        self.stuff.append(stuff)

class Child(Parent):
    def __init__(self, stuff = []):
        Parent.__init__(self)
    def doIt(self):
        self.addStuff("foo")
        Parent.doIt(self)

c1 = Child()
c1.doIt()
c2 = Child()
c2.doIt()
c3 = Child()
c3.doIt()

Execute this and we see:

foo
foo foo
foo foo foo

What? Why is it accumulating “foo”s? The answer is in the “stuff = []” argument to __init__. What is going on, as far as I can garner from the documentation, is that that default argument (a list) is being instantiated once and then kept for future use. What’s more, I’m assigning self.stuff to this, which is kept as a reference and doesn’t create a copy. So I have ended up with a static value for self.stuff, well, within the functionality of this code – it isn’t entirely congruous to a static member.

How to fix it? I don’t know the definitive approach, but here’s a couple of ways. To achieve complete deep copying of the argument whether it be default or passed in use copy.deepcopy:

#!/usr/bin/python
import copy

class Parent:
    def __init__(self, stuff = []):
        self.stuff = copy.deepcopy(stuff)
...

Alternatively, you could use copy.copy for a shallow copy and I guess that might be analogous to this:

#!/usr/bin/python

class Parent:
    def __init__(self, stuff = []):
        self.stuff = []
        self.stuff.extend(stuff)
...

I’m sure there are great uses for this behaviour in Python. What are they? Something more fundamental than “neat tricks” involving incremental/changing default state? Am I the only one to think this somewhat of a gotcha?

Now that I’ve worked out what was wrong I can retrospectively build the right Google magic to get straight to the answer.

No time to read up on more casual chatter about it though. The documented formula is something like:

#!/usr/bin/python

class Parent:
    def __init__(self, stuff = None):
        if stuff is None:
            stuff = []
        self.stuff = stuff
...

In theory my ADSL finally goes live tomorrow. The frustration of trying to work without a reliable ‘net connection has been high. The O2 mobile broadband account has been a life-saver, although it certainly has its failings.

There’s one thing that has been rather a bother. While SSH connections are fairly manageable, the only problem being intermittent disconnections lasting 5 to 10 minutes, HTTP behaves very badly. Even when I did have an apparent working ‘net link, evidenced by the fact that my SSH sessions were still live, HTTP connections would time out and fail regularly.

I have HTTP working pretty well now, here’s what I did:

1. Forward all HTTP(S) to a remote proxy, in my case running on a machine I have in Germany.
2. Stop using the O2 allocated DNS addresses, I’m using OpenDNS‘s 208.67.222.220 and 208.67.222.222 instead, though I’m considering just using my own remote DNS server.

The former should be all that is required, however it looks like Opera uses local name resolution for something even when a proxy is set up. The latter alone may be sufficient too, since the ‘net is certainly still there but it looks like name resolution times out or fails.

I’ve been using Linux for quite some time now. It’s been more than 10 years since my first hesitant install of Debian (from the front cover of a tech mag) in early 1998, and a little longer since I first poked at my cousin’s Linux computers. In the earlier days of my Linux use there was rarely a time I had to power-cycle a locked up machine when I was running a ‘stable’ distro. Sure, sometimes a bleeding-edge X, or somesuch, would lock up – but I was nearly always able to switch to a console or get a serial terminal and kick it in the arse.

Things seem to be a bit different now. More and more often I’ll have to hold down that power button. Sometimes I can magic-sysreq a reboot, but often not. The culprit is usually web browsers, but sometimes other X applications. It does seem to be an X thing. I use an RC Firefox with a beta Flash, so I expect breakage. What I don’t expect is breakage in my userspace apps to lock up my machine irretrievably. Especially since I’m using a stable distro, aside from the Flash and Firefox my machine is 100% Ubuntu ‘gutsy’ (no proper ‘net yet so I haven’t done my dist-upgrade, the 13th I’m told now, Friday the 13th! Joy.)

Linux never used to do this to me. Has robustness suffered in the quest for a snappier user experience? Am I just unlucky? It could be my non-free video driver (nvidia) I guess. I haven’t had the time to try debugging the problem, I should really, since it can be replicated.

Maybe I’m just seeing the past through rose coloured glasses, could it have been worse than I recall? I’ve had to reboot twice this morning, so now I’ll stop trying to play with the Firefox RC and get back to browsing with Opera.

That aside, it looks like Firefox 3 could win me back from being an Opera user. If my bloody Linux stops hanging when I use it!

My donglenet is cool because I can post this from my laptop while travelling home on the train while passing between a couple of fields.

OK, so it cuts out from time to time, well… a lot, and can’t handle tunnels at all.

Yeah, it isn’t really that good.

But it’s still cool. 🙂

Of course, funny to think that I had access to this very same coolness nearly 5 years ago when I got my first 3G handset back in Sydney. Old tech in new packaging (though definitely higher bandwidth.)

First, these are more informed and to the point:

• metasploit page with information about the bug and pre-generated vulnerable keys
• The relevant Debian security advisory. Note that the ‘openssh-blacklist’ package only contains DSA-1024 and RSA-2048 keys, if, like me, you use RSA-4096 keys this blacklist isn’t much use. Download the one from the metasploit page. (Yes, one of my keys was vulnerable! But the ssh-vulnkey tool didn’t detect it because it was 4096 bit. I assume that in time the blacklist will be extended to include other keysizes.)
• Advisory on nessus.org. I like this one because it sums it up nicely “Consider all cryptographic material generated on the remote host to be guessable.” Joy.

Some people think that having a 1-in-131k chance of getting the right key for a susceptible account isn’t enough of a risk to cause much concern. I can’t agree, I think any reasonable risk is a concern and that this is certainly a reasonable risk. Given that the default is 2048 bit RSA keys and it is reportedly likely that the generating PID is low the risk is probably higher anyway, less than 1-in-32k.

There’s an easy solution for SSH authorized_keys: check the keys against the blacklists! The metasploit page has blacklists for the usual key sizes available as well as a few less common key sizes.

What’s the risk though? The main known factor is that there are a small number of possible keys. If we know our target is x86 (highly likely) and only go for RSA 1024, 2048, and 4096 keys plus DSA keys that’s only about 131k keys (as large as that number seems, I assure you that it really is rather small in this context.) For an attacker to successfully compromise your system he’ll have to “get lucky” with a key and username combination. There are a couple of general classifications for the attack scenarios I guess:

1. Global brute-force. (Low chance of successful attack on a specific system?)
2. Targeted attack with inside knowledge. (High chance of successful attack on a specific system?)

The first scenario is best executed with a botnet. If I was an evil botnet herder I’d probably consider devoting some of my resources to this. I’d probably select only 2048 and 4096 bit keys to attack, as in my experience these are the most recommended key sizes (in fact, it is probably a good bet to try for just 2048 bit RSA keys only, as this is the default for ssh-keygen and I expect most people stick to the defaults. I’m not sure about the data-point regarding PIDs on the metasploit site, but I can imagine it to be true. (I imagine many user keys are generated as an almost first-step after installing and booting a new system, thus a low PID.) Believing the data-point I’d assume that limiting the PID seed to 10k probably increases the likely rate of successful compromises.

The other variable from the point of view of a single system is user-names. I think trying for ‘root’ is a given, and may even consider that to be the only user-name worth trying. Not going for the lowest hanging fruit, going for the ripe and juicy windfall apples. (I know admins who prefer to give ‘root’ access via ssh keys, because then you can revoke an individual’s root access without having to change the password and update everyone. There’s sudo for that these days though, stop using ssh keys.)

So, block remote root logins – as you should – and you’d probably be safe from me. But there’s also common user-names, attackers have lists of these. (I get a large number of SSH login attempts daily for user/pass combinations like bob/bob, tom/tom, … etc.) These user-names are probably a definite risk. Like I said, I see a regular flow of brute-force attempts to SSH in brute-forcing on usernames and passwords, given that this activity is common on the ‘net I reckon it is a given that it’s already happening for the bad SSH keys.

How about the likelihood of your server actually being attacked? From the point of view of the attacker this is a third variable, server address. It is a big Internet. Assuming bot-nets are randomly testing all valid ‘net IPs individual machines are probably, statistically, fairly safe. Though if I were configuring a botnet I’d pare the IP range down to blocks owned by specific co-location providers. Places like EV1 where large numbers of machines with lots of bandwidth are administrated by large numbers of totally clueless gits. If you’re on such a network your risk is likely to be higher (and who doesn’t have a server that’s in a “bad neighbourhood” purely because such neighbourhoods are cheap?)

Overall, I’d rate the chances of becoming the victim of a global-brute-force as fairly low. Still, the bad guys are going to successfully compromise some machines, it could be your unlucky day! This bug still increases your overall chances of compromise and you decrease them by fixing it. At the very least ensure that remote root isn’t permitted (and fix the situation if it is) then check all user-keys without locking down SSH.

The second scenario could leave you much more vulnerable.

If your company is likely to be targeted then it is likely that the attackers can get hold of all kinds of information. If they know the names of your employees (easy to find out) they can probably work out a list of likely user-names. If any one of your uses logs in with key-based auth and has a susceptible key you’re probably screwed (especially if they chose a 2048 or 4096 bit RSA key, like most people probably do?)

If an employee has lost a laptop with a susceptible SSH key you have a new worry (amongst the many your probably have anyway): it no longer matters how good their passphrase is. The new owner of the laptop’s data now knows the employee’s login (well, it is likely, .bash_history for example) and the size and type of their key.

What if some employee logs in from a shared machine where somebody else has root? OK, this is already a risk. You should simply never do this and employee’s who use SSH should know better! Anyway, previously the untrusted root would have to set something up to snaffle the key when the user types in their passphrase – now they can narrow it down to one of 32k keys without intervention.

What if your employee logs into another server run by someone else using an SSH key that they also use on your server? The owners of the other system now have that employee’s private key, all they have to guess is the login. Vice versa, you now have the private key for systems that the employee may log into with that same key. It probably only takes a little bit of digging to find out what companies/institutions many of the users of your system uses. The user could be security concious and may not trust you, but it’s OK the only data they’ve given you is their SSH public key, right? But now you have their private key! It is likely that they use the same key for other systems, have a guess at their username and try logging into the IPs they’ve logged in from.

There’s more…

Anyway, this is all speculation. I’m just tossing around some of the obvious risk scenarios in my head, I don’t have the data to put any numbers on them so I can’t prove anything at all. The main point is that I think there are enough risk cases that taking this potential hole in your system lightly is probably a bad mistake. You should have fixed it already.

In the end I think this is a case where it is better to be paraniod.

And I’m only talking about SSH user keys here – server certificates are a whole other nightmare. Given that you publish your SSL public key as part of the transaction I assume it would be trivial for someone to generate your private key from this data – Mavis is going to be one very happy girl. IIRC signed keys are typically only 1024 or 2048 bit (last time I worked with CA-signed server keys, years ago, our CA would only sign 1024 bit SSL keys.)

Everyone is writing about the Debian & derivatives SSH issue.

[Update: For the sake of accuracy I did an s/32k/196k/ in a couple of places.]

I don’t think it can be written about enough, the more exposure the better. When I first saw the headlines I thought, “oh, probably just anther ones of those things where a heat-death-of-the-universe problem has become a 1-million-years problem” – ho ho! How wrong was I? It really is very serious. If you generated an SSH key on a Debian/etc machine while the bug was in place your private key is one of only 32k possible keys (for each key type for each key size, i.e. 32k possible 2048 bit RSA keys, etc. So for 1024, 2048, and 4096 bit DSA & RSA keys that’s 196608 possible keys [Update: oops, DSA keys can only be 1024 bits!]) This means that if someone knows a machine you log into and your username it’ll take them no more than 196k attempts to log in to the system as you (and probably less, since the Metasploit page linked to above claims most keys are generated by proceses with low PIds.) That’s a tiny number of attempts in the brute-forcing world.

Some of us have brute-force blocks on our gateways and servers, this is great up to a point. For example a typical set-up is to start blocking all traffic from an IP if it hits port 22 more than 10 ten times in 30 seconds. I do the this on my server.

This is only good so-far though. First, if yo aren’t in a hurry configure your brute-force script to try every 10 seconds. I doubt most firewall setups go as far as to notice that sort of thing.

Second, this is a bonanza for botnet owners! If yo have a modestly sized botnet (say, 32k nodes) yo just give each bot a set of keys, a list of common logins, and the Internet. Actually, you’ll probably do pretty well just testing the entire internet with ‘root@’. (AFAIC you should never permit remote root login anyway – but I suspect many servers do and they make it “secure” by permitting only key-based ath, heh heh.)

What I wold be doing right this moment if I was an admin:

1. Block SSH at the gateway! (Ouch!) And all other SSL protected servers.
2. Check your server keys, replace if needed. This could take a long time where re-signing for public keys is required.
3. Move all users .ssh directories to something like .ssh_suspect
4. Inform users, probably by phone. (They’ll probably call you when they loose SSH.)
5. Start scanning all .ssh_suspect directories for blacklisted keys, remove them, inform the users, reinstate SSH with good keys restored.
6. Continue mopping up the mess– probably mostly a case of chasing up server certificate re-signing and informing/handling users.

I’m not an admin and not an SSH expert, so that scheme is vague and probably needs further tightening. It surely must be better than doing nothing though. You probably need to audit all your logs too, especially auth and firewall. Essentially the security of all your systems is suspect until you are certain that all logins prior to the lock-down were kosher (probably requiring a lot of back-n-forthing with users.)

I haven’t even sat down and contemplated the full extent of this yet.

The fact that we could even have got into the situation is insane. Peer review?
How cold someone clueless enough to cluelessly mess with the PRNG in security
code even be permitted to make such a change?! It beggars belief.

Of course, anyone even vaguely interested in security has probably already reached the cynical point of believing that there isn’t any security. Awareness is key.