by Alan Mead, PhD, Chief Psychometrician
Anyone with access to a word processor can write an exam, but if it covers industry-specific or technical material, non-experts probably can’t look at it and determine if it’s doing the job. So, how do you test a test?
The Standards for Educational and Psychological Testing list over 170 general standards and sixteen more for credentialing exams. We use these to ensure that an exam is psychometrically reliable, valid, fair, and is implemented correctly.
In general, a good exam has these parts:
-
Job Task Analysis: Know what you’re asking. An exam should be based on some thorough analysis of a domain of knowledge, curriculum, job, or role. This usually means that a JTA has been conducted.
-
Blueprint: Build your exam on a solid foundation. Good exam content adheres to a blueprint that specifies the items and topics for reliable and valid measurement. A good blueprint is typically based on a curriculum (where one exists) or else the results of a job/task analysis.
-
Subject matter experts: Write what they know. High-quality exams are authored by qualified experts who know the material well, adhere to the blueprint, and follow item-writing best practices. Peer review review is also important to effective item writing.
-
Psychometrics: Data reveals truth. Pretesting and psychometric data analysis are used to identify and eliminate poorly performing items. You don’t need to understand the statistics to know that your exam is of high quality when it’s constructed using psychometric best practice.
-
Standard-setting: Keep it fair. The passing scores should not be chosen arbitrarily and should reflect the same performance across forms, even if those forms differ in difficulty. A standard-setting study and psychometric activities like equating ensure that performance standards are equitable across forms.
-
Customer service: Keep an eye on the candidate. An exam must be easy to schedule, easy to navigate, and provide clear feedback that supports candidates who need to retake the exam.
-
Documentation: Show your work. Part of the test development process is documenting that these steps were completed according to professional standards.
While these standards don’t represent every aspect of a quality exam, they provide a key to evaluating if a certification measures up.
