Psychometricians: Who, What, and Why?

I once worked for an economist who sold wage data; he told me that the position of psychometrician doesn’t exist because no one had ever asked him for wage data for that position. A professor of engineering interested in measuring ethical standing claimed I’d invented the term “psychometrics!”  

Clearly, psychometrics is an obscure area of specialization, but we have deep roots with some of the seminal papers being published in the early 1900s. Our field is old enough that we have both modern and classical test theories (both are now widely used). Today, psychometricians are eagerly borrowing from machine learning to make exams better. 

In fact, psychometricians have a function for exam programs similar to that of Certified Public Accounts for corporations. Some CPAs are business or process experts and help organizations function better in big ways, some do taxation work, and others audit corporations.   

Many psychometricians are consultants, working with exam programs to invent new exam programs or transform existing programs. We also apply our expertise to advise exam programs on applied problems (e.g., “Will the exam work as well if it’s taken on a mobile device?”) and conduct important steps, like training item writers or facilitating standard setting meetings. 

Other psychometricians conduct applied research, such as evaluating the effect of taking an exam on a mobile device (perhaps comparing three different responsive designs to see which is the best, or which is best liked by candidates).  

Finally, psychometricians perform a critical auditing function. No item writer has ever intentionally written a poor item or allowed one to pass review, but psychometric analysis of pretest data routinely detects a small percentage of items that either don’t work well, or which are disruptive to the exam (e.g., accidentally having two correct answers). 

As chief psychometrician for Certiverse, I work with our team of developers to build rigor into our system to ensure that partners can easily access industry best practices, while also collaborating with our test sponsors to provide program feedback and analysis. Certiverse hasn’t eliminated the need for psychometricians, but it has made it easier for them to manage more or larger programs, or for our own staff to provide these services at scale.   

Regardless of the setting, psychometricians apply our knowledge of exam programs and psychometric theory to advise exam programs and to improve exams. All our research methodologies are focused on ensuring that exam scores are reliable, valid, and fair. At Certiverse, we share this commitment to rigor, but we also focus on making exam development as efficient as possible.