This article explains how to create your own coding test as a custom question in your assessment.
|Applies to:||Scale, Business, Enterprise plans; Owner, Admin, Recruiter roles|
Coding questions allow you to assess a candidate’s ability to create or debug code in a specified programming language. Coding questions allow you to test the following:
- Algorithmic thinking (e.g. searching, sorting, recursion, iteration)
- Language-specific concepts (e.g. data structures, object-oriented programming)
We currently offer tests in eleven programming languages. We use the following versions of each type (so be sure to write your question with this in mind).
- Java: OpenJDK 12
- C: GCC 10
- C++: G++ 10
- C#: SDK 3.1.300
- Go: 1.16
- PHP: 7.4
- Python: 3.8
- Ruby: 3
- SQL: SQLite 3.31.1
- Typescript: 4.3.4
This article goes into detail on creating custom coding questions. If you'd like to jump ahead, the topics covered are:
- Writing the test description
- Code specific information
Writing the test description
The first key part of creating a coding question is to define the coding task on the left-hand side of the screen (Test description) and selecting how long the candidate will have to complete the task.
You have two options for time limits on coding questions: 10 minutes and 30 minutes
The below steps will help you create the test description.
Step 1: Give some context and the overall goal of the coding task.
This brings the question to life and is necessary for the candidate to understand the intent of the coding task. Here's a good example of what this would look like:
Step 2: Articulate in clear terms what the candidate needs to do
For this example, let's say the candidate needs to write a function. You'll want to
- Specify the output: what should the function return?
- Specify the arguments of the function (the inputs)
Working with our above scenario, here is an example:
Step 3: Give an example of input and output that satisfies the requirements
This might look something like this:
Step 4: Any relevant details.
In our vault example, this would be:
Step 5: (Optional) - Add any potential hints or other considerations.
This is not necessary in most cases, but might be something you want to add.
Code specific information
After you've finished the test description, you'll need to fill out the tabs on the right-hand side of the screen.
Here, you'll find the following tabs (for all languages except SQL, that will be discussed later in the article):
Here you can set the type of variable the function returns, it's name, and the input parameter(s). You can choose from the following parameter types:
- Array (entered as comma separated values such as 1,2,3,4,5
|Note: Objects are not supported as function parameter|
The initial code is the starting point for candidates. It shows the function signature. You can add some code as a starting point if your task requires it. Please indicate where candidates are expected to insert their own code.
|Note: This is not necessary if you're developing a debugging task|
Test cases are combinations of input and output parameters that satisfy your requirements. They determine how successful the candidate is in completing the coding task. The first set of test cases is available to the candidate. If they click "run," their code is executed with the input parameters specified in each of these test cases. If the code returns the same result as you have specified in the test case, the test case is passed. This result is shown to the candidate. You can find these under the header “Test cases to be used during the test.” These test cases should be somewhat straightforward. Create 3-4 of them.
Additionally, there are separate test cases that will be used to produce the test score. These are not used during the assessment, so candidates cannot check their code against these test cases. You can find them under the header “Test cases to be used for validation.” These test cases should cover all requirements, exceptions, corner cases, etc. Create 6-10 of them, depending on the complexity of the question. The Correctness Score of the test will be the percentage of passed validation test cases.
You can also add test cases to measure the efficiency of the code. These test cases produce the Performance Score, showing the % of performance tests passed. The performance is defined by the runtime required to execute the test case, in milliseconds. The candidate’s code passes the performance test case if the runtime of the code is fully executed within the time limit you have set. A performance test case is relevant for coding tests where scalability is important. You can see how much time it took to run a performance test case when you run your verification code (see below):
The verification code is basically the model answer. You need to code it and run it to ensure all test cases you entered are passed. Only then can you save the question.
Once the candidate has completed the test, there will be a coding report that shows the candidate's code and the result of the validation test cases that determine the score. You can read more about the coding test report here.
SQL tests are obviously a bit different, given the additional database component in the test.
The first tab is “Database structure." Here, you choose whether you ask the candidate to write a query (using SELECT) or a database update (using INSERT, UPDATE, or DELETE).
This tab is also where you'll also script the initial database by creating the table(s) and inserting the appropriate values. In most cases, you should copy your initial database structure as a code block into the test description. Otherwise, candidates have no idea how the database is structured and what’s in it.
In the “Initial code,” you can include some first lines of code if your coding task requires it.
Here, too, the test cases are split up between those that will be available to candidates (“Test cases to be used during the test”) and those that will be used to determine the Correctness Score (“Test cases to be used for validation”).
The difference is in the structure of the test cases. In the “Test case setup,” you can insert additional values into the database. This script is executed right after the creation of the database structure but before the test taker code.
In the case of asking the candidate to create a query, you define the expected result of the query given the database structure and the test case setup, as per figure 1 below.
Figure 1: Sequence of SQL scripts being executed when asking a candidate to write a query (using SELECT)
In the case of asking the candidate to write a database update, you should also write a checking query. This query will be executed after the test taker code. In the “Result,” you specify what the checking query is supposed to return in this case. See figure 2.
Figure 2: Sequence of SQL scripts being executed when asking a candidate to write a database update (using INSERT, UPDATE, or DELETE)
Please note that every test case recreates the database. Test cases are therefore isolated from each other.
The “Verification code” works in the same way as in other coding questions: this is the "model answer" and should be clean and efficient.
If you still need help, you can always reach out to the support team. We're happy to answer your questions.