Curious About Crowdsourcing?

As the pandemic has shifted a lot of programs to remote exam development (often a toxic combination of scheduling headaches, Google sheets and Zoom fatigue), more test sponsors are considering crowdsourcing. But can it really work?

Certification and licensure exams exist to validate the skills and competencies needed to perform a certain job role, so it follows that those creating the exam should have knowledge of those skills and competencies, right? After all, nobody wants their local butcher choosing topics for a cloud-security exam or an art history professor determining criteria for medical board licensure.

Assuming we’re agreed on that point, let’s look at the definition of crowdsourcing:

If crowdsourcing is to happen in exam development, it needs to be done by a community of experts in the topic area. That’s why we generally refer to Certiverse’s process as “community-sourced” or “expert-sourced” rather than just “crowdsourced.” However, finding a large number of experts for your exam topic area may be another challenge altogether. To succeed, it’s imperative that you make the whole process as easy and enticing for them as possible.

For example, are all the potential subject matter experts for your exam located in the same city as your exam administration staff? Unlikely. Are they all in the same time zone? How about the same country? When scheduling busy professionals across multiple geographies and time zones, there will never be a convenient time for ALL participants. That’s an unnecessary barrier to participation.

The second barrier is training. Most would agree that your SMEs must be experts in your topic area. However, if you’re also expecting them to be experts in exam development, you’re adding another unnecessary obstacle to the process.

The last SME recruitment hurdle is one I’m going to describe as “scope of project.” This is the entirety of the “ask” when you’re looking to engage a subject matter expert in the process of building your new exam. It encompasses everything that goes into completing their part of the project—number of meetings to attend, number of hours spent on training, number of weeks, months (dare I say years?) that the exam development process takes, requirements to travel, compensation, etc. The greater the “scope” of commitment, the fewer willing participants you are likely to find. We’ve heard anecdotes from many programs about SMEs quitting mid-process, frustrated because they didn’t realize all that was expected when they initially committed to the process.

Because you can’t crowdsource without a crowd, Certiverse removes as many barriers to the process as possible to engage a much larger pool of SMEs:

Scheduling/Travel – Built for asynchronous, remote participation, the platform allows SMEs to log in and perform their work based on when it’s most convenient for them.

Training/Facilitation – Certiverse uses AI, machine learning and natural language processing to guide participants through the system, providing prompts and alerts as they go (think of how TurboTax works).

Scope of Project – Certiverse streamlines the scope by automating routine processes and handling all exam development functions within one platform. As a result, the overall time commitment for SMEs is dramatically reduced. Royalty options and gamification further incentivize SMEs to participate.

We’ve actually moved away from using the word crowdsourcing, because it has such a wide interpretation and doesn’t quite capture the goals of our process. More accurately, we’re applying an effective expert-sourcing approach to your exam — one that strengthens your professional community and also pays dividends in the form of lower exam development costs and faster time-to-market. Read The Linux Foundation case study to learn more.