Calling All Experts: How to Leverage Your Community to Build a Better Test

Over recent months I have spoken to many organisations that certify candidates in their sector or on their software product. Although they represent a range of industries, the commonality among them is a desire to  both create more content and reduce the time it takes to develop exams. There are many ways Certiverse is able to support these goals: just-in-time Subject Matter Expert (SME) onboarding; more accurate and quicker content creation using natural language processing (NLP); integrated surveys and analysis; and asynchronous content development and workflow optimising SME time. Who wouldn’t want to create more content more quickly? This is, of course, a great differentiator and is well-received by potential users as I would have expected, but something that consistently generates as much interest is the capability of the platform to effectively facilitate the use of external SMEs.  

Perhaps it is my naivety of not having been involved in the content creation aspect of certification exams for a while, but I just hadn’t realised that a majority of the organisations I have spoken to are only using internal staff, or long-serving SMEs, to create their exam content. Perhaps they do this because it’s just the way it has always been done or maybe it is perceived to be more secure, as they know their SMEs and theoretically have more control over them. There is also the financial component: If you’re using your own staff, it is a ‘free’ resource. The challenge with that approach is that SME time always gets reprioritised, and sadly content writing is rarely at the top of the organisation’s priority list. Of course, some organisations are returning to in-person item-writing workshops post-COVID. However, given the expense of this approach, organisations only want to invite participants who they know will do a good job. Inviting new SMEs to participate is a good way to add diversity and fresh thinking to your program, but it may be costly to the organisation if they don’t work out. Often these obstacles make it difficult for an organisation to gather enough items for the exam in the needed timeframe, and corresponding practice tests are delayed even further.  

So, what is it that most impresses people about Certiverse’s SME management component? Enabling SMEs, including internal SMEs, to work asynchronously is an obvious benefit, as it allows them to contribute to the exam from anywhere in the world while working around their other responsibilities. Plus, they are guided by natural language processing and supported by the platform’s automated workflows, so they are given real-time item feedback and prompted when they need to interact with the system rather than requiring even more staff to manage every step. The Certiverse platform also has several built-in gamification features that can be enabled, which recognize SME activity with a leader board of the most prolific and accurate contributors.  Making progress through levels and getting a high score are real motivators for achievement-oriented individuals. It’s also possible to link productivity to digital badges for item writers to demonstrate their expertise externally, and Certiverse even has different remuneration options to choose from, if you need to provide your SMEs with some financial motivation.  

Another benefit of Certiverse’s remote, asynchronous platform is how it limits item exposure among SMEs, with item writers only seeing the questions they are writing or reviewing, backed up by a full audit trail. Real-time dashboards of your item writers provide instant feedback on their accuracy and output so you can manage them more effectively. Seeing these dashboards highlighted something to me: if someone only writes a few items, does it really matter? With just-in-time training you aren’t wasting time and money onboarding people to be item writers. So, isn’t it much better to have 50 item writers from a diverse range of backgrounds and experiences who write 10 questions each, rather than 10 item writers who write 50 questions each? Much quicker, too. Incidentally, Certiverse tags questions that are written by the same item writer so we can prevent too many items from one author appearing on a form.  

If using external SMEs is going to help your organisation create more content, then it can only be a positive thing. Apart from increasing the variety of expertise in your item pool, Certiverse makes it easier to write more content. That is a big bonus. The more content there is, the more options you have for more frequent testing, the less risk of exposure of exam forms, and – the big one for me – the greater the likelihood of making practice tests available when you launch your live exam.