It’s hard to overstate how important item-writing is to support exams that are reliable, valid, and fair. A well-crafted test item accurately measures knowledge and skills while minimizing ambiguity and bias. All the later steps that add quality to an exam do so by removing bad items, so it is critical to write a good pool of items.
Training SMEs for Effective Item Writing
Subject matter experts (SMEs) bring essential applied knowledge but are usually novices at writing and reviewing items. And formal item-writing training is time-consuming with limited effectiveness. Item writers routinely write items that are unclear, overly simple, overly complex, rely on trivia or opinion, or otherwise fail to align with best practices. (It is interesting to note that in all the studies conducted in the past century, there is not a single study of what method of training is most effective in training item writers. The Handbook chapter on item writing has no citations and emphasizes feedback on items written.) The Certiverse exam platform provides a short, embedded video and written training materials regarding the basics of the item-writing process, but we feel strongly that the interface and workflow are the reasons that the median survival rate for items in beta is 92.5%.
Guided Item Development with Smart Tools
Certiverse’s item-writing wizard breaks down the process of writing items to make each step easier and to better ensure alignment to psychometric best practices. SMEs are prompted to craft clear, concise stems, supported keys, well-balanced distractors with rationales, and a reference. In the same way that you use spell-check, our platform is evaluating the item against the syntactic aspect of Haladyna & Downing’s (1989) rules for writing multiple-choice items and highlighting issues when they are detected.
The Certiverse platform also offers exam programs the option of enabling AI-assistance. When enabled and chosen by the item author, the SME supplies the item topic, and the system transparently and securely contacts a private instance of a SOTA AI model to write drafts of the item. We show the first three drafts and ask the SME to pick the best one. SMEs can dismiss these drafts to see as many as ten drafts of the item. Subsequently, the SME is sent to the item-writing wizard with pre-filled fields to review and revise.
These built-in supports help SMEs write high-quality items, even if they have little experience with test development. And our cloud-based platform is equally accessible from Berlin and Hyderabad, with the asynchronous workflow facilitating far-flung team members working together seamlessly.
Applying Best Practices in Question Design
Haladyna and Downing’s item-writing rules for multiple-choice items are ubiquitously used to guide item writing and have been largely supported by subsequent research. In fact, most lengthy item-writing training processes go through their 43 rules, showing positive and negative examples based on each rule. Here are just a few of their rules:
Iterative, Distributed, Automated Peer Review
Even experienced item writers sometimes write flawed items (and inexperienced item writers frequently write flawed items). Certiverse’s platform automatically places newly written items into a review queue for review by their peers. Typically, review is configured to require two passes to become Accepted and the average item in most campaigns requires three reviews to achieve Accepted status. We therefore encourage SMEs to adopt a cadence of writing one item and then reviewing three items (assuming there are other authors’ items available for reviewing). This workflow not only formally reviews each item, but also ensures that each item author gains the perspective of having to review others’ items.
The review itself is simplified by the UI which asks the reviewer to focus on a specific aspect of the item (overall question, stem, key, distractors, etc.). For each focus area, the reviewer answers a yes/no question derived from Haladyna & Downing (1989). If their answer indicates a violation of the guidelines, they are asked to provide feedback to the author. If the item passes review, it is immediately requeued for the next review, and if it does not pass, it is sent back to the author for revision.
By the way, the same review process is used for both AI-assisted items and items authored entirely by SMEs.
Building Better Certification Exams
Effective test item creation is a balance of SME expertise and psychometric principles. By following structured guidelines and leveraging collaborative exam development tools, organizations can create certification exams that are fair, reliable, and valid. Certiverse’s integrated item-writing tools, including an AI-assisted wizard and embedded training, help SMEs produce high-quality questions that accurately measure professional competence.