Stumbled upon this paragraph on Russ White’s blog:
I don’t really know how you write a certification that does not allow someone who has memorized the feature guide to do well. How do you test for protocol theory, and still have a broad enough set of test questions that they cannot be photographed and distributed?
As Russ succinctly explained the problem is two-fold:
How do you test for understanding of theory? Test questions along these lines are really hard to create, and should probably involve a lot of “considering this scenario, what will likely happen” stuff. Unfortunately, you have to pay people who understand the said theory (and therefore happen to be expensive) instead of review question miners.
Yes, in case you didn’t know, as I described in 2009, some managers believe you can hire someone (cheap) that will create review- or test questions on any topic just by somewhat-intelligent copy-pasting of source material.
How do you have broad enough set of questions? This one is simple. Generate the pool that’s impossible to memorize (excluding few individuals gifted with photographic memory) due to its size even if it’s made public. As I wrote in the same blog post, you can randomize the questions, or generate a dozen questions out of a single scenario, or hire more experts to write questions.
However, you have to invest heavily in the quality of your certification exam, which is hard if you treat a niche certification as a profit-generating machinery instead of a brand-awareness exercise that should result in significantly larger increase of profits of whatever it is the certification is all about.
You probably had the “privilege” of going through at least one certification exam in your career. What would you say the driving force behind that exam was?