Couldn’t Resist: Cheat-Proofing Certifications

Stumbled upon this paragraph on Russ White’s blog:

I don’t really know how you write a certification that does not allow someone who has memorized the feature guide to do well. How do you test for protocol theory, and still have a broad enough set of test questions that they cannot be photographed and distributed?

As Russ succinctly explained the problem is two-fold:

How do you test for understanding of theory? Test questions along these lines are really hard to create, and should probably involve a lot of “considering this scenario, what will likely happen” stuff. Unfortunately, you have to pay people who understand the said theory (and therefore happen to be expensive) instead of review question miners.

Yes, in case you didn’t know, as I described in 2009, some managers believe you can hire someone (cheap) that will create review- or test questions on any topic just by somewhat-intelligent copy-pasting of source material.

How do you have broad enough set of questions? This one is simple. Generate the pool that’s impossible to memorize (excluding few individuals gifted with photographic memory) due to its size even if it’s made public. As I wrote in the same blog post, you can randomize the questions, or generate a dozen questions out of a single scenario, or hire more experts to write questions.

However, you have to invest heavily in the quality of your certification exam, which is hard if you treat a niche certification as a profit-generating machinery instead of a brand-awareness exercise that should result in significantly larger increase of profits of whatever it is the certification is all about.

You probably had the “privilege” of going through at least one certification exam in your career. What would you say the driving force behind that exam was?

Recent posts in the same categories

10 comments:

  1. Daniel Dib 10 April 2018 11:39
    For me certs have always been about learning. So whether someone argues that the CCIE is less valuable now or not is not very relevant to me. I went through certifications to learn and to become a better engineer. Certifications are a guided learning path. If you treat them as such, they will always be valuable.

    I've done SME work in the past so I know how difficult it is to produce good content. It's fairly simply to come up with a couple of questions and fairly simple to work out one alternate answer to a question that seems reasonable. It's much more difficult to think of two or three reasonable answers that are wrong.

    I definitely agree that certifications shouldn't have to carry their own cost. How much gear has Cisco sold through the years because of certifications? Billions and billions again. For some of the higher level certs I wouldn't be against some form of interview but it scales poorly and is open for interpretation and people will sue the vendors because they think they have been mistreated *sigh*.

    I don't think there's a silver bullet to create good certs. It takes a lot of hard work and because people cheat the exams that lashes back on the people trying to take them honestly because the tests are filled with trivia type questions.
    Replies
    1. Salman Naqvi 11 April 2018 19:21
      I second this x 1,000,000. Certifications area great "guided learning path". Not all certifications are good at this aspect, but most are. Certification also shows discipline and dedication. And those who cheat in certifications are easily outed in interviews. I tell everyone to always do certifications. They are great ROI in almost all IT careers, not just networking.
  2. John 10 April 2018 12:17
    Had all good intentions of progressing to CCIE, but got as far as CCNP managed to get through the CCIE written (suffering from Dyslexia, this was hard enough) and when faced with the additional cost and stress of going through the lab where the timings were so tight and with little prospective improvement to my current existence I simply just couldn't face it.

    I spent so long trying to get my CCNP, but now i just see it as a weight around my neck that needs money throwing at it, for no real benefit to me personally.

    I enjoyed the journey through it all and the learning that was required to get the certification, after that it all became a bit hollow.
    Replies
    1. Unknown 04 May 2018 13:55
      @John
      I am also suffering from Dyslexia and I can't retain memory but I make notes plus hands on practice. I did my CCIE in 2010 after two years of passing my lab I went through divorce which made my Dyslexia into worse condition and end up losing my job. Two years to recover and now I am preparing for CCIE Wireless and it took me 1.50 years to pass written. I hope I will pass Lab exam and I am fighting against Dyslexia but most employer don't understand difficulties we are experiencing to learn and retain memory. It take us double or triple time to beat those exams but we don't give up. I hope some people will understand pain of Dyslexia. My passion for technology and learning never die even though I am nearly 50 years old.
  3. Anonymous 10 April 2018 12:42
    Maybe the author of this post can answer his own question (as a CCIE Emeritus). I would be highly interested in the answer. For me certifications are just for salary increase and only useful at the start of a career. Then after some years it's all about successful projects and experience.
  4. Unknown 10 April 2018 17:52
    I took the CCDE exam back in the early days because most of my work at the time was design/architecture and the CCIE didn't seem like the best investment at the time. (incidentally, I was in the same exam as Scott Morris who was on his 3rd attempt). The CCDE was interactive - a wrong answer led you further down the rabbit hole in the scenario. To be honest, I felt that exam did test your theory because it framed scenarios, let you select which supporting docs would help you, and it was brutal. Wasn't perfect, but I think it is a good starting point. Do they still have value? Not so much... the world is changing so quickly, people are pushing to cloud, building private clouds and using hybrids with overlaying management. Plus you may find large enterprises migrating from one platform to another so your CCIE may help with your ACI deployment but not so much with a large scale NSX deployment. Certs are like a college degree at this point... it simply indicates to an employer that you put some time in. We interviewed a CCIE for a position a couple years ago and hired the other non-CCIE candidate instead... he had far more real-world experience and therefore answered some of the tricky questions much better. YMMV.
  5. SMV 10 April 2018 19:06
    Many years past now, Cisco once did a pretty decent job with the CCIE Written exam (at least R&S). There were clearly SMEs involved in the development of test scenarios and questions (e.g. "Review the following topology. When the R5<->R6 link goes down, how many queries are necessary in order for EIGRP to fully converge?"). They now apparently employ professional test writers exclusively who scour the online product literature for questions such as "The following product/solution that Cisco is currently championing (but will soon be poo-pooing in a few short years as a horrible idea in need of replacement) has which of the following benefits (pick five out of seven)?" To make matters worse, my understanding is that some sort of ISO certification that Cisco pursued for their certification program bars any sort of "collusion" between the cert guide authors (SMEs) and the test writers, only further dumbing things down. Since the written is no longer a test of knowledge, and I have reached the 10-year mark with my digits, it's Emeritus all the way from here on out. The challenge I face now is educating folks as to just how little it means to pass (or fail) the marketing quiz that the CCIE Written has devolved into.
  6. Anonymous 10 April 2018 19:54
    Don't focus on vendors, focus on technologies. Vendors come and go. Understand the technologies well, test them and share your experience with the community.
  7. Anonymous 11 April 2018 16:21
    I lost every interest in certifications, as my colleagues hold tons of them, obviously by means of braindumps. If vendors cannot protect the value of their certification and everyone can achieve one, then why bother anymore?
  8. Steve R 12 April 2018 10:42
    As Daniel Dib stated, certification, like any qualification should simply be a structured learning path. This is something that in my experience always been pretty poor within IT, regardless of vendor.

    I think the real question is how do you prevent people cheating? Easy, give them an incentive not to. Make learning fun and targets achievable. Perhaps vendors and certification bodies should invest more time in a quality structured material. Stop capitalizing on education, and make exam re-takes affordable. When it costs several hundred dollars to take a test, you are always going to breed a sub-industry that will undermine honest achievement.
Add comment
Home
Sidebar