A look at the process of creating a vendor certification exam, and what we can learn from it to help prepare for these kinds of exams.
Disclaimer: I am not a vendor certification kind of guy. Until recently, I have never taken a vendor certification exam, never wanted to, never thought I would. I’m still not a cert kind of guy, but at least I now have a little more respect for those who are.
Heck, I’ve never even taken the famous Brian Madden Class on Citrix because that just isn’t how I learn things. I learn by popping open the hood and pulling wires to see what breaks. Thank goodness I deal in software so I can re-image rather than trying to figure out just where that wire is supposed to go afterwards.
Sure, vendor certifications are important for many people in this field. They are a shorthand that shows that, at least at some level, you know something about a particular technology. Or at least cared enough to read the crib sheets. Employers, while not necessarily wanting to pay for exams, love certifications for the hiring process. Many employers pay for them as a way to further educate their workforce hoping that they don’t jump ship after passing them.
So this spring when I had an opportunity to participate in the development of a certification test, I decided to do so to learn more about that whole process. I entered this experience deadly biased against what they stood for. In my mind, the tests consist of a bunch or archaic questions that proved nothing. I mean, is it really important that I know off the top of my head if it is port 1434 or 1443? I’ll look it up if I need to know. This guy named Al invented this neat thing called “the internet”.
My NDA prevents me from talking about specifics, so I’m not going to name the exam, nor even the vendor. But I do want to describe my experience and what I learned about the exams. While this experience might be specific to this test and this vendor, I believe that it fairly represents most exams created by vendors. I entered in the middle of the exam development process, but we should start at the beginning.
The Vendor Creates an Exam
It costs money to create a good exam. It isn't simply getting a couple of developers and field engineers together to slap some questions together.
Prior to my involvement, the vendor had put together a syllabus that represented areas of the test to be covered by both the test and a test preparation class to be developed. The syllabus is an outline of the products and feature areas to be covered in the exam. The initial syllabus provided by the vendor will have an additional level of detail beyond what you will see posted describing the content of the exam when the exam is advertised.
Enter the 3rd Party Exam Development Company
Next, they contracted out the test development to a third party dedicated to the creation of this test. There is a whole industry of folks who do nothing but create exams, and training classes for the exams. The third party gets a fixed amount of money to produce the test, and they need to make a profit. Sure, they very much care about producing a quality test, but when push comes to shove, you have a manager with budget line responsibility making the decisions. These people are clearly experts well trained in the art of developing a test. Every test is special in some way, but the development of the test always shares a similar set of problems that may be overcome using standard techniques.
Enter the Exam Question Writer
The third party locates experts to produce appropriate questions that would allow test takers to demonstrate their command of the material, based upon the syllabus. This person (or group of people) are provided the detailed syllabus and a set of guidelines on how to write questions. The question creation and editing is run on a tight schedule not unlike the book writing process, managed by someone who is experienced in writing tests .
Enter the Subject Matter Experts
When this part of the process is complete, the questions are then scrubbed by a different group of other subject matter experts (SMEs). This is where I came into the process.
In our case, a handful of SMEs were flown in a met for most of a week to scrub the questions. To ensure we completed this process in the available time period, the test development company had several employees present. One was in charge of leading us through the test. This person was chosen for having good skills at group dynamics and the ability to keep flaring egos in check. (What, you say, “experts” my not agree on the correct answer?!¿) The expert that wrote the questions was not present, but the person who managed that part of the test was present. Also present was a note taker, who would produce the final test copy, and someone who managed the SME that produced the questions and could talk to some extent regarding the intent of a question or answer on the test. Finally, there was the manager, whose job it was to keep us moving.
Like any test, ours was flawed. In our case, the test had to be written against a Beta version of the actual product (several products, actually) and the SMEs had already seen newer versions that worked very differently is so many ways that you simply could not avoid the changed areas. We were told we had to produce the test against the already outdated version so that a test could get out and that the test would be updated at a later time (and obviously a separate contract). We went through each potential question on the test. In each case, we first each answered the question independently. Then discussed the question. In some cases we modified the question, or some of the answers. In others, we threw the question out completely, and were then asked to create a replacement question.
The guidelines were very important in this process. There really is no intent in these tests (any more) to test memorization of archaic facts. There are also no trick questions. And all answers must be actual things that someone could try to do – you can’t invent a product or feature or menu item that doesn’t really exist to put it into one of the incorrect answers. One of the overriding things we were asked to keep in mind was to stop and ask ourselves two questions:
1) Is this a fair questions?
2) Is this something a hiring manager would want to know a potential employee has knowledge of?
We certainly had our share of compromises driven by the need to keep moving. In particular, I remember one question that was in the form of “given this situation what action should you take next”. But, the answer I would have given was not on the list. Sure, only one of the multiple possible answers provided with the question was a reasonable step, but it was definitely not what I would do as a next step. Ultimately, however, I had to let this question go as is. The purpose of the question was to test a particular area of the syllabus and my answer would have made the question fit a different area of the test. And frankly, there was not a better question we could propose (without a very convoluted question setup) to cover this syllabus item, so we had to let it go. Of course, when you take the exam they don’t tell you what part of the syllabus is being covered. We were also restrained from “trying to teach” with questions, and were trained to watch for questions and answers that would provide an answer to a different question on the exam.
There are other parts of this process that were impressive to see. The person I described earlier as the “note taker” was an expert on test grammar and terms. This keeps product names and acronyms consistent and ensures that questions are not ambiguous due to phrasing. While the test generation company had a budget and time constraints to live with, they still acted professionally and clearly deeply cared about producing a quality result. The SMEs involved also cared deeply about the test, even downloading the old version and running a test over night to see if one of the incorrect answer s might actually solve the problem by accident.
Enter the Exam Training Class
There was also a training class official curriculm developed by the vendor (likely also outsourced) I was not involved in the development of the official curriculum training class for this exam. I believe that was a separate process completed independently and in parallel using the same detailed version of the sylabus. In our case I think the class had been written prior to our test scrubbing, however none of the SMEs had seen the class materials.
Enter the Exam Taker
I recently went to a testing center to take the final version of the exam. This was my first time through that process, and I was better impressed at the test taking process than I expected. Considering what goes into generating the test and providing a independent quality facility to take the test in, what you pay to take the test is a lot less it could be.
In taking the test, I could clearly see how the editing I mentioned improves the test taking experience. After only a few questions, the style of language on a question quickly led me to understand what was being asked for on the question. Ultimately I cringed at some of the questions that we came out with. While awful questions, at least they were fair.
I did learn along the way here that some of the questions that appear on the test will be weighted in the scoring process. Once the test was produced out of our effort, the test was taken by a number of invited people to test the fairness of the test. Based on that testing, some questions may be reduced in weight or even given a weight of zero. Why don’t they remove the just remove question altogether? Because they hope to improve awareness over time and use this non-counted question as a gauge of success.
How I Would Prepare in the Future
While I’m still not going to be taking a lot of vendor tests (hopefully, no more), I at least have a newly found respect for this whole testing industry and the folks involved with it.
From this experience, I think I have a better idea of how I would “study” for a certification exam in the future. If you look at the posted version of the syllabus, I would work through that. I would put myself in the shoes of that test writer. What kinds of questions would I write against each item? If I can’t come up with a question, this becomes an area that I need to work on more.