Readers who commented on some of my previous certification-related posts have complained about the vagueness of exam questions. I have to agree with them; I’ve seen my fair share of dubious questions in the exams I’ve taken. For example, when I was developing EIGRP and BGP courses for Cisco, my lowest scores on the CCIE recertification exams were in those two categories. I knew too many details and was confused by the vagueness of the questions.

Writing good exam questions is not easy. To start, you have to be a subject matter expert. Using professionals specializing in generic learning theories to write exam questions simply doesn’t work; the best they can do is try to squeeze meaningful questions from the course materials. (You can recognize this type of question immediately: the correct answer is a direct quote from the student notes.) Furthermore, the expert writing the questions should have a somewhat devious mind and focus on testing the student’s understanding, rather than retention of trivial facts. For example, instead of asking students to select TCP-based applications from a list, you could ask them to select the correct access list (or NAT translation) from a set of lists using TCP or UDP.

There are also simple tricks that can significantly expand the question pool: randomize the answers (this will hinder at least some of the users of brain dumps) and randomize all the parameters in the test questions. It’s quite easy to change IP addresses or port numbers in the exam questions, and if someone can answer a pattern of questions correctly, he or she might have gained some useful knowledge in preparation the process.

Finally, you have to be prepared to invest serious money and lots of resources into the exam questions. A small group of experts can only take you so far; the more people you involve, the better. As a side effect, you lower the risk of leaks – if each writer produces only a few questions, there is little harm if those questions are leaked. Organizations like the Institute of Internal Auditors (IIA) have already recognized this opportunity and actively encourage members to contribute their questions. We can only hope that Cisco will follow the same path instead of focusing on high-tech methods (such as exam forensics).

Let me conclude with a simple financial calculation: using the IIA pricing model ($50 per accepted question), you can get a pool of 1,000 questions for $50,000—the cost of 200 CCNA exams (currently $250). Last year, Cisco announced that it has issued more than a million certifications. You can also assume that a few students probably had to take an exam more than once (and you need to pass multiple exams to gain some certifications). How many questions could you generate if you would devote just 5% of the exam-generated revenue to this activity?