Couldn’t Resist: Cheat-Proofing Certifications
Stumbled upon this paragraph on Russ White’s blog:
I don’t really know how you write a certification that does not allow someone who has memorized the feature guide to do well. How do you test for protocol theory, and still have a broad enough set of test questions that they cannot be photographed and distributed?
As Russ succinctly explained the problem is two-fold:
How do you test for understanding of theory? Test questions along these lines are really hard to create, and should probably involve a lot of “considering this scenario, what will likely happen” stuff. Unfortunately, you have to pay people who understand the said theory (and therefore happen to be expensive) instead of review question miners.
Yes, in case you didn’t know, as I described in 2009, some managers believe you can hire someone (cheap) that will create review- or test questions on any topic just by somewhat-intelligent copy-pasting of source material.
How do you have broad enough set of questions? This one is simple. Generate the pool that’s impossible to memorize (excluding few individuals gifted with photographic memory) due to its size even if it’s made public. As I wrote in the same blog post, you can randomize the questions, or generate a dozen questions out of a single scenario, or hire more experts to write questions.
However, you have to invest heavily in the quality of your certification exam, which is hard if you treat a niche certification as a profit-generating machinery instead of a brand-awareness exercise that should result in significantly larger increase of profits of whatever it is the certification is all about.
You probably had the “privilege” of going through at least one certification exam in your career. What would you say the driving force behind that exam was?
I've done SME work in the past so I know how difficult it is to produce good content. It's fairly simply to come up with a couple of questions and fairly simple to work out one alternate answer to a question that seems reasonable. It's much more difficult to think of two or three reasonable answers that are wrong.
I definitely agree that certifications shouldn't have to carry their own cost. How much gear has Cisco sold through the years because of certifications? Billions and billions again. For some of the higher level certs I wouldn't be against some form of interview but it scales poorly and is open for interpretation and people will sue the vendors because they think they have been mistreated *sigh*.
I don't think there's a silver bullet to create good certs. It takes a lot of hard work and because people cheat the exams that lashes back on the people trying to take them honestly because the tests are filled with trivia type questions.
I spent so long trying to get my CCNP, but now i just see it as a weight around my neck that needs money throwing at it, for no real benefit to me personally.
I enjoyed the journey through it all and the learning that was required to get the certification, after that it all became a bit hollow.
I am also suffering from Dyslexia and I can't retain memory but I make notes plus hands on practice. I did my CCIE in 2010 after two years of passing my lab I went through divorce which made my Dyslexia into worse condition and end up losing my job. Two years to recover and now I am preparing for CCIE Wireless and it took me 1.50 years to pass written. I hope I will pass Lab exam and I am fighting against Dyslexia but most employer don't understand difficulties we are experiencing to learn and retain memory. It take us double or triple time to beat those exams but we don't give up. I hope some people will understand pain of Dyslexia. My passion for technology and learning never die even though I am nearly 50 years old.
I think the real question is how do you prevent people cheating? Easy, give them an incentive not to. Make learning fun and targets achievable. Perhaps vendors and certification bodies should invest more time in a quality structured material. Stop capitalizing on education, and make exam re-takes affordable. When it costs several hundred dollars to take a test, you are always going to breed a sub-industry that will undermine honest achievement.