How do I develop an effective screening test?

In this article, we take you through the TestGorilla test development process in five steps. These guidelines will help you create high-quality tests.

Interested in creating a screening test for your company? Great! You'll be applying your expertise and experience in a new and exciting way, and we're here to help you through the entire process. This step-by-step guide gives an overview of our process for creating the most effective screening tests.

Step 1: Define the skill areas for a test

General idea: Every test is meant to test candidates on something specific. It’s important to take time to clarify what exactly you're trying to measure and it will be structured. This will form the foundation of your test.

What this means in practice: Clearly define the skill areas within your test. These areas break the test down into different domains of knowledge or skill. Each test should contain 3 or 4 skill areas that are mutually exclusive and collectively exhaustive (MECE). Organizing your skill areas by the MECE principle means finding the optimum arrangement whereby you organize all the relevant information into clearly distinct groups without any repetition.

For example, let's say you want to organize a group of people into a few subgroups. Organizing the main group into smaller nationality groups (i.e. European nationals in one group, African nationals in another, Asian nationals in another, etc) is a non-MECE organizing system. People who have dual nationalities will create "repetition" as to which groups they belong to, and people who have no nationality won't fit into any subgroup. A MECE way to organize the same group would be by age groups. Every person is of a certain age and that age is unique (you can't be 30 and 31 years old at the same time), so every person in the main group will belong to one subgroup and one subgroup only. 

Depending on the job role your test applies to, you may also be able to refer to official standards to help you with the organization. It's also helpful to research and review current job descriptions to see what employers expect.

For example: Let's say you intend to develop a "Customer Service" test at the intermediate level. First, review current industry standards (like the ones from the NOS). You can also find job descriptions for customer service representatives elsewhere on the web (for example here or here). Based on these standards, you could define skill areas that look like this:

  1. Ability to understand customer concerns or requests, and uncover root causes
  2. Ability to interact with customers in a positive and constructive way
  3. Ability to accurately and efficiently determine appropriate solutions
  4. Ability to avoid negative or unintended consequences

        Essentially, defining the skill areas for a test is about crafting a high-level structure for the knowledge and skills that the test will assess.

        Step 2: Define the categories (skills to test) within each skill area

        General idea: Each skill area tests candidates in a specific section of the overall scope of the test. It's important to define the specific skills, or categories, that each skill area covers. This will give you a framework for writing questions in the next step.

        The categories within each skill area must still follow the MECE organizational principle. This means that the categories must collectively cover all the information in a specific skill area without creating any repetition within that skill area or with categories in other skill areas. 

        Going back to our earlier examples to see how this works, breaking down age groups into specific ages results in a MECE system. Each number can only exist in one age group once and exists only in that age group. Breaking down nationality groups, however, into nationalities isn't as clear. Should the Egyptian nationality, for example, be organized under African nationalities (based on geography) or under a Middle East group (based on language)? 

        When breaking skill areas into categories begins to create repetition or categories that cannot be clearly (and logically) categorized into one skill area along, it's a good idea to take a step back and consider approaching the topic from a different perspective.

        What this means in practice: Clearly define the categories (specific skills) within each skill area. These categories break your skill areas down into different skills. Each skill area should have about 5 or 6 categories; they should be mutually exclusive and collectively exhaustive.

        For example: In the "Customer Service" test from above, you would further define the key categories covered in each of the main skill areas. For the first skill area, it might look something like this:

        1. Ability to understand customer concerns or requests, and uncover root causes:

          • Understanding the main issue in a stated problem
          • Uncovering the root cause of an issue
          • Empathizing with concerned customers
          • Understanding the potential impact of an issue for the customer
          • Understanding the potential impact of an issue on the company

        The skill areas and categories together form a comprehensive outline of your test. Following this outline, you can create questions with specific objectives in the next step.

        Step 3: Write questions based on the defined skill areas

        General idea: Your test's questions should relate to the skill areas and categories that you've defined in the previous step. Each question should address only one skill area and one category. Another expert in your field should be able to read your questions and immediately understand which skill area they relate to. Your questions should be based on real-world scenarios that the candidates have to solve, avoiding generic, theoretical questions that can be answered from a manual without having the necessary skills.

        What this means in practice: To come up with relevant and effective questions, first determine what a test-taker should be able to demonstrate for each defined category. Ask yourself what knowledge or skill would be good evidence of the test-taker's abilities.

        You need to create at least as many questions as your test will include, evenly spread across all skill areas. For example, for a 12-question question with four skill areas, you need to create at least three questions per skill area. If you plan to include this test in many different assessments, then it's a good idea to create more questions per skill area. Every time you add the test to a new assessment, a new set of questions will be selected by our algorithms to protect the integrity of your questions and the test results you receive, as candidates in each assessment will see a different set of questions. 

        The difficulty of the questions should vary somewhat within each skill area, while still maintaining the overall level of difficulty determined for the test. That way, the test can accurately distinguish a range of ability levels.

        Most of your questions should be scenario-based questions (also called "situational-judgment questions) that ask test-takers to solve a specific scenario. This will help evaluate their skills in dealing with a relevant issue or situation (versus just their theoretical knowledge). In general, avoid questions of the type "What does X stand for?" or "What does Y do?" that can be easily answered with a simple Google search.

        For example: In the Customer Service test, a question in the fourth skill area ("Ability to avoid negative or unintended consequences") could be:

        Which action can have a negative impact while dealing with an angry customer?

        image-95  Showing your compassion about the inconvenience experienced.

        image-94  Allowing the customer to vent and calm down.

        image-94  Asking the customer to come back when they have calmed down.

        image-94  Explaining what the best solution is for this moment.

        Important: Do not copy questions from an existing source, either online or offline. Your questions should be your own original content.

        To learn more about the question types available on our platform, how to use them, and see examples of questions, read our article on which question types to use. Our article on how to create good situational judgment questions will also help you create comprehensive yet concise scenarios for your questions.

        Step 4: Write for clarity—don't confuse test-takers

        General idea: The goal of each question is to measure a specific skill, ability, or knowledge. Therefore, you should make sure your questions are straightforward and easy to understand. Candidates shouldn't have to spend time figuring out what exactly a question is trying to ask, nor should they be purposely confused by overly ambiguous answers.

        What this means in practice:

        1. Be clear and concise in your wording. If not, you're measuring someone's language skills, vocabulary, or reading ability rather than the skill area to which the question belongs.
        2. Do not use negatives. Avoid words like NOT, FALSE, LEAST, WORST, and EXCEPT. These can confuse candidates about the action they need to take.
        3. Do not try to trick test-takers. In multiple-choice questions, for example, have one correct answer and several distractor choices. The distractors should be somewhat tempting, but not overly tricky.
        4. Do not give your answer away. Don't make one answer much longer or shorter or very different from the others, and avoid adverbs or other qualifies that signify whether an answer is desirable or not. (Words like just, only, immediately, right away, etc.) Don't give the answer to one question in another question.
        5. Limit questions that have multiple correct responses or use "all of the above" and/or "none of the above" as options. You can include a few of these but should avoid them whenever possible.

        Step 5: Perform a thorough final check

        General idea: Every test should be a good predictor of job performance and should also provide a good experience for job candidates. Thoroughly reviewing your finished test questions is critical for quality control.

        What this means in practice: Make sure you can say "Yes" to each of the following questions:

        1. Do your questions cover the skill areas appropriately and completely?
        2. Are the questions sufficiently varied (to avoid duplication of knowledge tested)?
        3. Is every question and answer option crystal clear?
        4. Is all provided information relevant to the answer (to avoid confusion)?
        5. Are the image, audio, and/or video materials you're using clear and relevant? Are graphs and diagrams properly and correctly labeled?
        6. Did you check grammar and spelling (U.S. English) for all questions?
        7. Did you write a clear and compelling description of the test? Does it clarify to customers and test-takers when to use the test and what to expect from it, respectively?