How do I create a coding test?

Looking to create a live coding test that screens developers? This article explains how.

Coding tests on TestGorilla assess a candidate’s ability to create or debug code in a specified programming language within a certain time limit (10 or 30 minutes). We have 8 coding languages available: Python, JavaScript, PHP, Java, C, C++, C#, and SQL.

For the main coding languages, you can develop a test in the following categories:

  • Algorithmic thinking (e.g. searching, sorting, recursion, iteration)
  • Language-specific concepts (e.g. data structures, object-oriented programming)
  • Debugging

When it comes to SQL, you can create a test in which candidates write a query (using SELECT) or a database update (using INSERT, UPDATE, or DELETE).

If you haven’t already discussed with TestGorilla’s test development team what kind of coding test you intend to create, please take the time to do so now.

Getting Started

After registering for our platform, click "Create new test." You'll be taken to the Landing Page of your new test. Here, you can clarify what your screening test is about and which job roles it tests for:

Next, select “Programming skills” as the test type. Determine if your coding test will have a 10 or 30 minute time limit in the box below. This is the total time that candidates will have to complete the question (coding task). You can also determine the difficulty level (entry-level, intermediate, or advanced). Please revisit and review this page after you've created your questions to make sure it’s still accurate.

In the Questions tab, you can create 1 Preview Question and 5 actual Coding Questions. The preview question is what our customers will have access to. It should be representative of the other questions in the test, as customers will use it to determine if the test is relevant to them. The preview question will not be used in an actual candidate assessment.

Next, you'll create 5 actual coding questions. Only one question at a time will be used for each candidate assessment. We cycle through the questions as an anti-cheating measure. To add a question, select “Add first question” and then select a language.

Test Description

The first key part of creating a question is to define the coding task on the left-hand side of the screen (“Test description”):

Please follow these steps:

Step 1: Give some context and the overall goal of the coding task.

This brings the question to life and is necessary for the candidate to understand the intent of the coding task. Here's a good example of what this would look like:

Assume you are in a crew of bank robbers, and one of you has secured the key to the vault of your local bank branch. The key can only be inserted into the lock at certain moments. This is a security measure.

You know that the key has to pass a number of pins that can obstruct the entry of the key. Those pins are lined up behind each other and can move away from their position to open their part of the key slot. If the key hits one of the pins, the alarm goes off. So you can only enter the key when all pins are extracted.

You are the hacker in the crew, and your task is to determine the precise moment when the key should be entered. At time T=0, the key is initially held right in front of the lock (position 0) and should go all the way in (position X). Your buddy is measuring X, and he’s still working on it.

Array A, consisting of N integers, represents the pins that are extracted to allow entry of the key. A[T] represents the position of the extracted pin at time T (in seconds). The objective is to find the first moment T when all pins up to position X are extracted. There is no pin at position X itself.


Step 2: Articulate in clear terms what the candidate needs to do, for example: “Write a function: …”

A. Specify the output: what should the function return?
B. Specify the arguments of the function (the inputs)


Here is a good example:

Write a function:

        int solution(int X, int[] A)

that, given a non-empty array A consisting of N integers and integer T, returns the earliest time when the key can be entered into the slot. N and X are integers within the range [1-100]; each element of array A is an integer within the range [1..X].


Step 3
: Give an example of input and output that satisfies the requirements:

For example, given X = 6 and array A such that:
A[0] = 3
A[1] = 2
A[2] = 2
A[3] = 4
A[4] = 1
A[5] = 5
A[6] = 3
A[7] = 4
the function should return 6, as all integers 1-5 are covered at T=6.


Step 4
: Any detailed requirements. In the example of the vault, it would be:

If the key can never be entered, the function should return −1.


Step 5
: Any potential hints or considerations. Note that this is not necessary in most cases.

After you've finished the test description, you can move on to filling out the tabs on the right-hand side of the screen. Here, you'll find the following tabs (for all languages except SQL):



Function Signature

Here you can set the name of the function, what kind of variable it returns, and the input parameter(s). You can choose from the following parameter types: String, Integer, Number, Float, Boolean, and Array. Arrays are entered as comma-separated-values, such as “1,2,3,4,5” (without the quotation marks). Objects are not supported as function parameters.

Initial Code

The initial code is the starting point for candidates. It shows the function signature. You can add some code already if your coding task requires it. Please indicate where candidates are expected to insert their code. This is not necessary if you’re developing a debugging task.

Test Cases

Test cases are combinations of input and output parameters that satisfy your requirements. They determine how successful the candidate is in completing the coding task. The first set of test cases is available to the candidate. If they click "run," their code is executed with the input parameters specified in each of these test cases. If the code returns the same result as you have specified in the test case, the test case is passed. This result is shown to the candidate. You can find these under the header “Test cases to be used during the test.” These test cases should be somewhat straightforward. Create 3-4 of them.

Additionally, there are separate test cases that will be used to produce the test score. These are not used during the assessment, so candidates cannot check their code against these test cases. You can find them under the header “Test cases to be used for validation.” These test cases should cover all requirements, exceptions, corner cases, etc. Create 6-10 of them, depending on the complexity of the question. The Correctness Score of the test will be the percentage of passed validation test cases.

You can also add test cases to measure the efficiency of the code. These test cases produce the Performance Score, showing the % of performance tests passed. The performance is defined by the runtime required to execute the test case, in milliseconds. The candidate’s code passes the performance test case if the runtime of the code is fully executed within the time limit you have set. A performance test case is relevant for coding tests where scalability is important. You can see how much time it took to run a performance test case when you run your verification code (see below):


Verification Code

The verification code is basically the model answer. You need to code it and run it to ensure all test cases you entered are passed. Only then can you save the question. The customer (i.e. the company that's testing candidates) will be able to inspect the candidate’s code only (not your verification code). However, it's still important that you create clean and efficient code.


SQL tests

SQL tests are obviously a bit different, given the additional database component in the test.

The first tab is “Database structure." Here, you choose whether you ask the candidate to write a query (using SELECT) or a database update (using INSERT, UPDATE, or DELETE).

This tab is also where you'll also script the initial database by creating the table(s) and inserting the appropriate values. In most cases, you should copy your initial database structure as a code block into the test description. Otherwise, candidates have no idea how the database is structured and what’s in it.

In the “Initial code,” you can include some first lines of code if your coding task requires it.

Here, too, the test cases are split up between those that will be available to candidates (“Test cases to be used during the test”) and those that will be used to determine the Correctness Score (“Test cases to be used for validation”).

The difference is in the structure of the test cases. In the “Test case setup,” you can insert additional values into the database. This script is executed right after the creation of the database structure but before the test taker code.

In the case of asking the candidate to create a query, you define the expected result of the query given the database structure and the test case setup, as per figure 1 below.

Sequence of SQL scripts being executed in when asking a candidate to write a query (using SELECT)

Figure 1: Sequence of SQL scripts being executed when asking a candidate to write a query (using SELECT)


In the case of asking the candidate to write a database update, you should also write a checking query. This query will be executed after the test taker code. In the “Result,” you specify what the checking query is supposed to return in this case. See figure 2.

Figure 2: Sequence of SQL scripts being executed when asking a candidate to write a database update (using INSERT, UPDATE, or DELETE)


Please note that every test case recreates the database. Test cases are therefore isolated from each other.

The “Verification code” works in the same way as in other coding questions: this is the "model answer" and should be clean and efficient.

Versions

We use the following types and versions of each coding language:

  • Java: OpenJDK 12
  • C: GCC 10
  • C++: G++ 10
  • C#: SDK 3.1.300
  • JavaScript: Node 13
  • PHP 7.4
  • Python 3.8
  • SQL: SQLite 3.31.1

Note that all your questions must be your own original work.


Good luck!