Easing Angoff: How To Streamline Your Standard Setting

The purpose of an exam is to prove knowledge or skills, but it’s rarely expected that test takers will need to answer every question correctly to pass. So how do you decide the threshold for success? Some programs may simply set the cut score for a certification exam to 70%. That’s a bad idea because a fixed percentage like 70% ignores the difficulty of individual test items and the required level of competence for the profession. Why not 69%? Or 71%? Or 75%? Arbitrarily setting a passing score risks passing underqualified candidates on an easy test or failing competent ones on a harder test. Instead, the cut score should reflect the specific content and difficulty of the exam form, ensuring it accurately assesses the competence of candidates based on the required skills for the field (potentially resulting in different passing scores across forms). It is important that the process of setting the passing score is as transparent, replicable, and objective as possible. 

Standard setting is the process of determining the minimum score or level of performance required to pass a certification exam, and is critically important for ensuring that certification decisions are fair, reliable, and valid. Many methods have been devised for this process, including:  

• Bookmark method: requires that items are sorted by difficulty and panelists simply pick a cutscore (where a minimally competent candidate would start to struggle) 

• Nedelsky method: panelists estimate how many incorrect options a minimally competent candidate could eliminate and then the cutscore is calculated based on an assumption  

• Contrasting groups method: sets the cut score based on performance data from two selected groups (e.g., experienced and novice candidates). 

The most widely used, however, is the modified Angoff method. This method is favored for several reasons. It is highly flexible and can be applied to any type of test composed of “items” regardless of subject matter, and it does not depend on having item data so it can be used before the exam has been administered. The modified Angoff method focuses directly on the minimally competent candidate, which ensures that the passing score is directly linked to the required level of competence. It also can be applied to a single form or to an entire item pool. Finally, the iterative process of expert judgment and discussion increases the reliability of the passing score, while its item-by-item focus allows for a detailed, transparent standard-setting process. As a panel method, it minimizes the subjective judgment of any one subject matter expert. 

One perception, however, is that the modified Angoff standard setting method is a cumbersome, expensive process that takes weeks or months to complete. Some exam programs have elaborate face-to-face meetings involving time, travel, and expense. Often these meetings are a big deal and serve important purposes, like building relationships with key subject matter experts and offering a reward for the work that these committees do for an exam. But just as there are different standard setting methods, there are different ways that the modified Angoff method can be implemented, and it is possible to streamline this process without sacrificing rigor. 

The standard Certiverse implementation of the modified Angoff method involves a one-hour virtual kickoff meeting with the panelists, in which they are oriented to their role, familiarized with the minimally qualified candidate, and practice making the evaluations. The panel then uses a Certiverse interface to complete the individual item evaluations independently and asynchronously. Typically, we might allow a few days or a week for the SMEs to complete their evaluations at their own pace. Finally, a reconciliation meeting of 60-09 minutes is arranged in which the panelists discuss items that have excessive disagreement (as shown by statistical analysis of their evaluations).  

This implementation of the modified Angoff method still requires a panel of SMEs, but having the meetings be brief and virtual and the work asynchronous makes this process considerably easier and more cost efficient than a one- or two-day in person meeting but without sacrificing quality. By adopting efficient yet rigorous approaches, certification programs can maintain high standards while minimizing unnecessary complexity and expense.