This article explains how to create your own coding test as a custom question in your assessment.
Coding questions allow you to test the following:
- Algorithmic thinking (e.g. searching, sorting, recursion, iteration)
- Language-specific concepts (e.g. data structures, object-oriented programming)
When it comes to SQL, you can create a test in which candidates write a query (using SELECT) or a database update (using INSERT, UPDATE, or DELETE).
The first key part of creating a coding question is to define the coding task on the left-hand side of the screen (“Test description”):
Please follow these steps:
Step 1: Give some context and the overall goal of the coding task.
This brings the question to life and is necessary for the candidate to understand the intent of the coding task. Here's a good example of what this would look like:
Assume you are in a crew of bank robbers, and one of you has secured the key to the vault of your local bank branch. The key can only be inserted into the lock at certain moments. This is a security measure.
You know that the key has to pass a number of pins that can obstruct the entry of the key. Those pins are lined up behind each other and can move away from their position to open their part of the key slot. If the key hits one of the pins, the alarm goes off. So you can only enter the key when all pins are extracted.
You are the hacker in the crew, and your task is to determine the precise moment when the key should be entered. At time T=0, the key is initially held right in front of the lock (position 0) and should go all the way in (position X). Your buddy is measuring X, and he’s still working on it.
Array A, consisting of N integers, represents the pins that are extracted to allow entry of the key. A[T] represents the position of the extracted pin at time T (in seconds). The objective is to find the first moment T when all pins up to position X are extracted. There is no pin at position X itself.
Step 2: Articulate in clear terms what the candidate needs to do, for example: “Write a function: …”
A. Specify the output: what should the function return?
B. Specify the arguments of the function (the inputs)
Here is a good example:
Write a function:
int solution(int X, int A)
that, given a non-empty array A consisting of N integers and integer T, returns the earliest time when the key can be entered into the slot. N and X are integers within the range [1-100]; each element of array A is an integer within the range [1..X].
Step 3: Give an example of input and output that satisfies the requirements:
For example, given X = 6 and array A such that:
A = 3
A = 2
A = 2
A = 4
A = 1
A = 5
A = 3
A = 4
the function should return 6, as all integers 1-5 are covered at T=6.
Step 4: Any detailed requirements. In the example of the vault, it would be:
If the key can never be entered, the function should return −1.
Step 5: Any potential hints or considerations. Note that this is not necessary in most cases.
After you've finished the test description, you can move on to filling out the tabs on the right-hand side of the screen. Here, you'll find the following tabs (for all languages except SQL):
Here you can set the name of the function, what kind of variable it returns, and the input parameter(s). You can choose from the following parameter types: String, Integer, Number, Float, Boolean, and Array. Arrays are entered as comma-separated-values, such as “1,2,3,4,5” (without the quotation marks). Objects are not supported as function parameters.
The initial code is the starting point for candidates. It shows the function signature. You can add some code already if your coding task requires it. Please indicate where candidates are expected to insert their code. This is not necessary if you’re developing a debugging task.
Test cases are combinations of input and output parameters that satisfy your requirements. They determine how successful the candidate is in completing the coding task. The first set of test cases is available to the candidate. If they click "run," their code is executed with the input parameters specified in each of these test cases. If the code returns the same result as you have specified in the test case, the test case is passed. This result is shown to the candidate. You can find these under the header “Test cases to be used during the test.” These test cases should be somewhat straightforward. Create 3-4 of them.
Additionally, there are separate test cases that will be used to produce the test score. These are not used during the assessment, so candidates cannot check their code against these test cases. You can find them under the header “Test cases to be used for validation.” These test cases should cover all requirements, exceptions, corner cases, etc. Create 6-10 of them, depending on the complexity of the question. The Correctness Score of the test will be the percentage of passed validation test cases.
You can also add test cases to measure the efficiency of the code. These test cases produce the Performance Score, showing the % of performance tests passed. The performance is defined by the runtime required to execute the test case, in milliseconds. The candidate’s code passes the performance test case if the runtime of the code is fully executed within the time limit you have set. A performance test case is relevant for coding tests where scalability is important. You can see how much time it took to run a performance test case when you run your verification code (see below):
The verification code is basically the model answer. You need to code it and run it to ensure all test cases you entered are passed. Only then can you save the question.
Once the candidate has completed the test, there will be a coding report that shows the candidate's code and the result of the validation test cases that determine the score. You can read more about the coding test report here.
SQL tests are obviously a bit different, given the additional database component in the test.
The first tab is “Database structure." Here, you choose whether you ask the candidate to write a query (using SELECT) or a database update (using INSERT, UPDATE, or DELETE).
This tab is also where you'll also script the initial database by creating the table(s) and inserting the appropriate values. In most cases, you should copy your initial database structure as a code block into the test description. Otherwise, candidates have no idea how the database is structured and what’s in it.
In the “Initial code,” you can include some first lines of code if your coding task requires it.
Here, too, the test cases are split up between those that will be available to candidates (“Test cases to be used during the test”) and those that will be used to determine the Correctness Score (“Test cases to be used for validation”).
The difference is in the structure of the test cases. In the “Test case setup,” you can insert additional values into the database. This script is executed right after the creation of the database structure but before the test taker code.
In the case of asking the candidate to create a query, you define the expected result of the query given the database structure and the test case setup, as per figure 1 below.
Figure 1: Sequence of SQL scripts being executed when asking a candidate to write a query (using SELECT)
In the case of asking the candidate to write a database update, you should also write a checking query. This query will be executed after the test taker code. In the “Result,” you specify what the checking query is supposed to return in this case. See figure 2.
Figure 2: Sequence of SQL scripts being executed when asking a candidate to write a database update (using INSERT, UPDATE, or DELETE)
Please note that every test case recreates the database. Test cases are therefore isolated from each other.
The “Verification code” works in the same way as in other coding questions: this is the "model answer" and should be clean and efficient.
We use the following types and versions of each coding language:
- Java: OpenJDK 12
- C: GCC 10
- C++: G++ 10
- C#: SDK 3.1.300
- PHP 7.4
- Python 3.8
- SQL: SQLite 3.31.1
- Typescript: 4.3.4
- Ruby: 3
- Go: 1.16
If you have any questions, feel free to reach out to our support team.