This is my somewhat long-winded reflection on the SANS GSSP-C certification. I signed up in September ’07, did the exam in London on December 5th, waited 8 weeks for my results, and eventually found out that I passed quite safely. The log below was written in parts, before taking the exam, then after taking the exam, then concluded once I got the results. It has been mixed up a little in the editing, but mostly retains this chronology.

The executive summary:

• I passed, answering 84 of 100 questions correctly.
• I barely studied, but I had read the “right book” previously.
• I’m a security-aware developer in the infosec sector who works with C regularly.
• I’m not convinced of the value of the certification to individuals.
• But I think that it can be a valuable benchmark tool for large companies.
• It worries me a lot that the “pass mark” is only 63.
• Don’t get me wrong, I do think it’s generally a good thing – read on…

Guhsspeesee?

“There’s just no amusing way to say, ‘I have a CISSP’.“
“MCSE is to computers as McDonalds Certified Chef is to fine cuisine.”

If you’ve had any technical exposure to the ‘net it’s likely you’ve come across snide phrases like these (especially in witty email sigs if you’re on mailing lists of a security or Linuxy nature.) In certain groups these “qualifications” have little respect. But are they really as bad as all that? Is it a case of there being a few bad apples in the barrel damaging the reputation of the good MCSEs and CISSPs? The MCSE seems to be almost universally mocked. Maybe I just know the “wrong people”, though it isn’t all hearsay, I’ve had to deal with a couple of MCSE-holders who had less of a clue about kicking XP into shape than I did (and I don’t know much about Windows.) I can’t comment on CISSP at all from any personal exposure to the breed, it’s certainly popular in the IT business sector and certainly unpopular in more underground security circles.

Personally I’m not going to judge people by the acronyms they choose, often these things are a business or work necessity. That they choose to publish their acronyms in email signatures? Why not? Sitting an exam, regardless of merit, seems more worthy of note to me than most of the other crap people put down at the butt-end of their messages.

You can probably gather that I’ve never really been “into” these certifications myself. On average “my group” (covering a group of pretty hard-core C/C++ applications/systems/kernel developers) really doesn’t have much time or respect for them to there’s never been any motivation to take an interest. But I can’t say that every opinion I’ve heard comes with complete and logical justification.

From about mid 2007 SANS started pushing the “GIAC Secure Software Programmer” certification, the “GSSP”. This initially comes in two flavours: Java and C (but with plans for Perl, PHP, C++, … Befunge?) That’s where the “C” comes from. This is the first time I’ve seen a certification that seemed particularly relevant to my day to day work. I decided to give it a whirl, since there’s no other way to really know what to think about these things.

A Rant on the Philosophy

Going into this I wasn’t sure what to expect. How do you measure up a coder’s security abilities with 100 multiple choice questions? How do you wrap something as complex as “secure coding” in this format? On lists there has already been some discussion showing up regarding the merits of the new certification. The main non-troll argument is “you just can’t measure this with a multi-choice exam”. I think, as seems to be the main defence, that the definition of “this” is what needs looking at. The detractors seem to take the definition as of “this” as “l33t security dude“. On the other hand I think, in agreement with the defence, the certification is best seen as a filter to sort people with a clue from the totally clueless. The value? In the shoes of someone hiring it’s a nice measure that you don’t have a total newbie sitting in front of you. For larger companies I think it could be a good tool to discover where weaknesses lie in your developer farm. (So, like Hyenas to a sick Zebra, your HR people can cull the weak! No, really: so you can properly target training and awareness programmes.)

Alas there is some marketing and up-speaking from SANS that does paint a little bit of a “silver bullet” picture around the certification. It’s easy to see where the detractors get their iffy feeling about the whole thing. Remember that SANS/GIAC aren’t charities, they have to sell this idea — and in this situation a little of the technical reality is lost to marketing drive.

I think this push has the potential to significantly weaken the value of the GSSP, “certify your coders and you’ve solved your security issues”. Having coders who can pass a test and who’re aware of bad practices is different from having coders who’re contentious in the application of their knowledge. It isn’t a replacement for peer reviews, regular code audits, code ownership, and plain old responsibility! I think that there is strength in the GSSP if viewed as one part of a more holistic approach to creating secure software. Don’t push it on your developers as yet another management hurdle shoehorned into their schedule! The last thing you want to do is say: “We’re hiring this expensive contractor in a suit you’ll talk at you for an hour every Friday for the next month, then you’ll take an exam. If you don’t pass your exam you’ll get a smaller bonus.” Yes, companies actually do do this, I’ve seen the insanity first-hand.

Is there an alternative? There must be! Start by getting actual developers behind the scheme, not some outsider in a suit. I could keep going on my thoughts here, but maybe another time. The point, as relevant to this entry, is these are my thoughts in the couple of months thinking about this before taking the exam. (In fact, in bulk, actually written prior to sitting the exam.) I wasn’t going into this as a fanboy, as usual my scepticism runs high.

Practicalities Prior to the Exam

The obvious starting points are the handbook that enumerates the exam content and (if still up) the webcast. The webcast is particularly useful as it involves Robert Seacord who’s one of the heads behind the exam and responsible for a highly relevant book on the topic. The ‘cast covers the sections you’ll find in the exam, the content, and the topic weightings.

The content of the blueprint seemed straightforward, the everyday issues that any C developer should be thinking of. I resolved not to worry much about study since it would be most interesting to see how I’d “rate” just going into this thing and giving it my best shot. So, in essence, my pre-exam “practicalities” were minimal. Just like in my Uni days, study is something for other people to fret about. (I don’t claim this is a good philosophy!)
Claiming a total lack of study would be dishonest, a few months ago I read Robert Seacord’s Secure Coding in C And C++.

Through several weeks of morning coffees I gradually made my way through this volume, it’s a good size for a morning-espresso(s) book. The content is a little dry, so it took a while to get through despite being fairly short (early in the morning it can seem far more profitable to stare into your espresso than to read about buffer overflows!) This is particularly relevant to the exam given the author’s involvement with the GSSP, and it turns out there is a strong symmetry between the content of his book and the topics enumerated in the GSSP-C handbook. So in a vague and fortuitous sort of way I’d covered some study content. For potential studiers it is a good starting point, but not all details covered in the exam are covered in this one book.

I can’t comment on the other books recommended on the SANS site as I’ve read none of them. On the website front one stands out, the CERT Secure Coding Initiative and especially the related Secure Coding standards web site. The latter is a wiki aimed at developing a secure coding standard. The content of the wiki covers much of the GSSP-C exam blueprint, in fact one of the major contributors is Robert Seacord. (You’ll see a prominent advertisement for his book in the sidebar.)

My Qualifications

So minimal study hey? What are my “qualifications” going into this exam then, what sort of person is being tested here?

I “learnt” C in around 2nd year of Uni, though IIRC none of the courses I took ever taught C specifically (we had a 2nd year C++ course). That was about 7 years ago. Through Uni I played with C a lot, mostly through an interest in some Linux systems and applications. Before completing Uni I also taught practical classes in “Programming Practice” to 2nd years, that was all in C (what a nightmare!), I learnt a lot of little details then to keep ahead of the syllabus.

Since Uni, as a developer, I’ve coded in C fairly often but not in a continuous or hard-core sense. I’ve used C++ and Python more often, but in the months leading up to the exam was mostly dealing with plain C. I also spend a fair bit of time auditing C code and working with/in 3rd party code, which is often pretty terrible. The context of this work has been that of being a development/research/integration(/pre-sales shudder) engineer for a startup/research/OEM company in the network security sector for several years (approaching 5.)

In essence, I consider myself a competent and security-aware C coder who still has a lot to learn. Neither a security expert or a C expert though, in my opinion expert is a pretty strong word.

The Big Day

So on Wednesday December 5th, just before 9AM, I walk into a room at the EcXeL centre in London. I wasn’t sure what to expect, being at a huge convention centre I was thinking it’d be a uni-eqsue exam situation, a huge cold room with a couple of hundred people sitting at little desks. I figured there’d be a pile of finance sector wage-slaves sucked into the process by this time. But the bandwagon, if it is to become such, had only just started rolling — this was the first exam held in Europe after all. I expect the big corporates will have their own in-house sessions arranged anyway. I walked into a room where only 12 people would be sitting an exam, and, if I counted right, just 5 of them were there for the C. The others doing the Java version, but that’s no surprise since there’s a pretty high demand for Java people in the City.

Of the exam content itself there isn’t much I’ll say as, logically, you agree to an NDA as part of taking it. Don’t worry about it branching out horribly from the “blueprint” in the handbook, it didn’t. You might want to worry a little that some questions lean a little towards the qualitative, rather than quantitative, side. But this might merely be the delineation between a good secure coder and someone who can merely recognise instances of bad practice. Given an example with multiple flaws, all of which make you shudder, which flaw is the worst? I had some difficulty with some of these and similar questions that required rankings of flaws and solutions, and classification of flaws. There were also a couple of questions that essentially required knowledge of security “glossary terms”, this is one area where a bit reading up on things in the suggested references is really going to help (I winged it as best I could since the terms tend to be fairly self explanatory.)

Reflecting on the Exam

Just going through the questions was worthwhile in a couple of ways. First, it highlighted some small gaps in my knowledge right away when either a question completely stumped me or I saw something and realised I didn’t really know what should happen with any certainty. Second, I recognised my strong reliance on manpages, I simply don’t commit much detail to memory when it is always handy the ubiquitous “dev” manpages. Over years, gaining more C coding experience, I imagine the manpage reliance will diminish. But anyway, you’re unlikely to find yourself working without manpages — so long as you don’t go making assumptions surely you’re OK?

Other gaps include: “terminology” and “severity”. The former is just knowing the right collection of glossary terms, best gained through more reading I supposed. The latter is a bit of a funny one, when presented with code containing several of the worst mistakes a coder can make I think “this code is crap, it’s all bad”. But you’re asked to pick the most severe flaw. Which one is it? The one that, in the right circumstances, could give an attacker arbitrary-code execution? The one that can reliably make the program crash (DoS) 100% of the time with the right input? I can’t answer these ones comfortably, as far as I’m concerned all the flaws are bugs that must be removed, none are acceptable.

It does seem they had some trouble coming up with their 100 questions as there was quite a lot of repetition and a few questions that felt like “filler.” I assume they need a corpus much larger than 100 in order to randomise the exam content between sittings.

The Results

“Results will be sent in the post 6 weeks after the exam. Results will not be made available over the phone or Internet.”

Waiting, waiting, waiting… Given it was filled in on one of those machine-marked sheets I’m surprised there’s a six week waiting time. Especially surprised that I didn’t get the results until more than 8 weeks later. The whole Christmas/NY thing was in the middle though, so I guess we can write off two weeks. But even then it seems a long time to deal with less than 20 exams.

The results arrive in an envelope with SANS on the front and even a real stamp from the USA. I know what it is right away and eagerly tear it open right away, “Congratulations!” is says. “Ah,” I think, “so I didn’t totally waste my 200 quid.” The letter is short and to the point, I passed the exam and a certificate “will follow under separate cover.” Whatever that means, I assume the translation is “in a separate envelope.”

The envelope also includes a separate sheet with your final results and summary of how you performed in the different parts of the exam. In the end I got 84 questions right, I guess you could call that 84%. It turns out that the “passing point” is 63, so I passed pretty safely. Looking down the breakout there’s no one area where I did particularly badly, and also none where I answered all questions correctly.

So, that’s it. Passed.

Personally I think getting 16 questions wrong is pretty bad. “Secure coding” is mostly rather black and white, getting things wrong at all is insecure coding. I have to admit that it kind of scares me that the pass threshold is 63! That means that someone can be 37% insecure and still pass. In other words, people getting this certification could be more than twice as insecure as I am.

Is is Worth It?

Too early to tell really. I got a couple of good things out of doing the exam: I noticed some gaps in my knowledge of what it was testing, and I found out what sort of barrier the exam sets. Whether it “furthers my career” in any tangible way is going to be hard to measure (just like measuring “secure coding”!) It probably won’t make much difference, since most places I’m likely to work will be pretty “geek” (so possibly fairly dismissive of “certifications.”)

Is it worth US$499? Maybe not to the individual, unless it becomes wide-spread for companies to require this qualification for new hires I don’t think it matters much. On a CV, going into the right sort of job, it probably makes for a reasonably good differentiator. (Or may mean nothing, depends on the sort of shop you’re applying for — I know of people in security who’ll filter people out if they seem to make a big thing of having a CISSP.) As far as judging the value goes, I’m a “special case” — I place a high value on satisfaction of curiosity and doing this exam did that for me (I’d probably have to give myself a good slap for not taking study seriously if I’d not passed though.) I also think there’s value in it having highlighted a few things that I didn’t know, but, as the results above attest, I don’t seem to be missing any whole zones of knowledge. It would be nice to know more about the questions I answered incorrectly, since otherwise it is hard to pick up on what I might need to research to bridge the gaps. Anyway, it helps that the GBP is nice and strong against the USD!

If one thing concerns me above all else it is that 63% pass mark. I seriously don’t think that someone getting 37 questions wrong should be considered a “secure software programmer.” In fact, I’d be more comfortable with an 85% cut-off (which would rule me out, so maybe 80%.) As an employer, if someone was talking up their GSSP-C I’d want to see their itemised results before giving much weight to it. Given my experience from taking the exam I’d consider a mark of 70 to be pretty borderline, but I’d consider it a good starting point for someone in a more junior C development role. (Update: 2008-04-08: My name is finally on the list, along with 18 others at this point in time. A rather short list! It’d be interesting to know how many people have sat the exam. Of the 19 “analysts” at this time, 4 people scored higher than me, one with 85 and three with 86.)

I think that the GSSP-C can have a much higher value to a large company wanting to gauge the abilities of its herd of coders. I imagine that if you’re pushing 100 people through the certification you’d get a discount!

It seems a little funny that they’re going to have a different exam for the GSSP-C++ … while there are some C++ specific security concerns most issues are the same as for C (no surprise, right?) It smells a little like milking the “secure programmer certification” for all the $$$ they can get. I’m a cynical bastard. I’m far from the first person to be critical about the GSSP though, in fact SANS SSI has a page devoted to critics, and I’m glad of that. The existence of that page makes me more confident in the certification and the process that gave birth to it. Though it was last updated in April 2007, I’m sure there must have been more critical feedback since then! (Note also that the page covers the C/C++ split and discusses why it was split and the related difficulties.) It’s also clear that the technical people behind this test, and SANS in general, are serious, concerned security professionals. If anything we can hope that the existence of things like the GSSP will raise awareness of secure programming methods and give the world more such professionals.