How do I create a coding test?

This article explains how to create a coding test on the TestGorilla platform.

Applies to: Subject matter experts

 

 

This article is for subject matter experts who are creating coding tests for our platform. If you are a customer creating your own coding test, please see this article.

 

Coding tests on TestGorilla assess a candidate’s ability to create or debug code in a specified programming language within a certain time limit (10, 30, or 60 minutes). We currently have 15 coding languages available: PythonJavaScriptTypescript, PHP, SQL, Ruby, Java, C, C++, C#, Go, Kotlin, R, Scala, and Swift.

For most coding languages, we develop tests in the following categories:

  • Algorithmic thinking (e.g. searching, sorting, recursion, iteration)
  • Language-specific concepts (e.g. data structures, object-oriented programming)
  • Debugging

When it comes to SQL, tests ask candidates to write a query (using SELECT) or a database update (using INSERT, UPDATE, or DELETE).

 

 

Getting Started

Once we get set up to start a project, you will receive your credentials for accessing the platform from a member of our test development team. In your account, you should find your test title, as discussed with your test development specialist. Note that your test will have a specific time limit seen on the test's Questions tab. Each test you create should be solvable by candidates within that timeframe. 

In the Questions tab, you can create 1 Preview Question and 5 actual Coding Questions. The preview question is shown to customers as a sample when they are trying to decide which test (and test level) to use for their candidates. Thus, the Preview Question should be representative of the other questions in the test, even though this question will not be used in actual candidate assessments.

Next, you'll create 5 actual coding questions. Only one question at a time will be used for each candidate assessment. We cycle through the questions as an anti-cheating measure. To add a question, select “Add first question” and then select a language.

 

 

Test Description

The first key part of creating a question is to define the coding task on the left-hand side of the screen under the heading Test description.

 

 

Please follow these steps:

Step 1: Give some context and the overall goal of the coding task.

This brings the question to life and is necessary for the candidate to understand the intent of the coding task. Here's a good example of what this would look like:

 

Assume you are in a crew of bank robbers, and one of you has secured the key to the vault of your local bank branch. The key can only be inserted into the lock at certain moments. This is a security measure.

You know that the key has to pass a number of pins that can obstruct the entry of the key. Those pins are lined up behind each other and can move away from their position to open their part of the key slot. If the key hits one of the pins, the alarm goes off. So you can only enter the key when all pins are extracted.

You are the hacker in the crew, and your task is to determine the precise moment when the key should be entered. At time T=0, the key is initially held right in front of the lock (position 0) and should go all the way in (position X). Your buddy is measuring X, and he’s still working on it.

Array A, consisting of N integers, represents the pins that are extracted to allow entry of the key. A[T] represents the position of the extracted pin at time T (in seconds). The objective is to find the first moment T when all pins up to position X are extracted. There is no pin at position X itself.

 

Step 2: Articulate in clear terms what the candidate needs to do, for example: “Write a function: …”

A. Specify the output: what should the function return?
B. Specify the arguments of the function (the inputs)


Here is a good example:

 

Write a function:

        int solution(int X, int[] A)

that, given a non-empty array A consisting of N integers and integer T, returns the earliest time when the key can be entered into the slot. N and X are integers within the range [1-100]; each element of array A is an integer within the range [1..X].

 

Step 3: Give an example of input and output that satisfies the requirements:

 

For example, given X = 6 and array A such that:

A[0] = 3

A[1] = 2

A[2] = 2

A[3] = 4

A[4] = 1

A[5] = 5

A[6] = 3

A[7] = 4

the function should return 6, as all integers 1-5 are covered at T=6.

 

 

Step 4: Add additional requirements that will help create some variety in test scores, meaning that the possible scores are not only 0% (if you don't solve the main parameters correctly) or 100% (if you solve them correctly). Additional parameters help better evaluating candidates by distinguishing between those who score, for example, 30% vs 60%, etc.

In the example of the vault, an additional requirement would be:

 

If the key can never be entered, the function should return −1.

 


Step 5
: Include any potential hints or considerations. This may not be necessary for all tests.

After you've finished the test description, you can move on to filling out the tabs on the right-hand side of the screen. Here, you'll find the following tabs (for all languages except SQL):

 

 

 

Function Signature

Here you can set the name of the function, what kind of variable it returns, and the input parameter(s). Each language has a set of parameter types available based on its specifications, such as string, integer, number, float, boolean, array, etc.

Arrays are entered as comma-separated values such as “1,2,3,4,5” (without quotation marks). Objects are not supported as function parameters.

 

 

Initial Code

The initial code is the starting point for candidates. It shows the function signature. The basic initial code is automatically shown on the platform, based on your function signature, but you can add additional code if your coding task requires it, or if you're creating a debugging test. Please indicate where candidates are expected to insert their code. This is not necessary if you’re developing a debugging task.

 

 

Test Cases

Test cases are combinations of input and output parameters that satisfy your requirements. They determine how successful the candidate is in completing the coding task. There are two kinds of test cases: tests cases to be used during the test and test cases to be used for validation.

 

Screen_Shot_2021-11-19_at_1_44_08_PM

 

The tests cases to be used during the test are available to candidates during the test and are meant should cover the core test requirements only as a way for candidates to check that they are on the right track while solving the exercise. When candidates click Run during the test, their code is executed with the input parameters specified in each of these test cases. If the code returns the same result as you have specified in the test case, the test case is passed. You should create 4 of these cases. 

 

The test cases to be used for validation are not available to candidates during the test, so candidates cannot check their code against these test cases. These test cases determine the candidate's final score in the test and are executed after the candidate submits their test. These test cases should cover all requirements, exceptions, corner cases, etc. of the test and increase in complexity. You should create at least 8 of these test cases. 

 

You can also add test cases to measure the efficiency of the code. These test cases produce the Performance Score, showing the % of performance tests passed. The performance is defined by the runtime required to execute the test case, in milliseconds. The candidate’s code passes the performance test case if the runtime of the code is fully executed within the time limit you have set. A performance test case is relevant for coding tests where scalability is important. You can see how much time it took to run a performance test case when you run your verification code (see below):

 

 

 

Verification Code

The verification code is basically the model answer. You need to code it and run it to ensure all test cases you entered are passed. Only then can you save the question. The customer (i.e. the company that's testing candidates) will be able to inspect the candidate’s code only (not your verification code). However, it's still important that you create clean and efficient code.

 

 

SQL tests

SQL tests are obviously a bit different, given the additional database component in the test.

The first tab is “Database structure." Here, you choose whether you ask the candidate to write a query (using SELECT) or a database update (using INSERT, UPDATE, or DELETE).

This tab is also where you'll also script the initial database by creating the table(s) and inserting the appropriate values. In most cases, you should copy your initial database structure as a code block into the test description. Otherwise, candidates have no idea how the database is structured and what’s in it.

 

In the “Initial code,” you can include some first lines of code if your coding task requires it.

Here, too, the test cases are split up between those that will be available to candidates (“Test cases to be used during the test”) and those that will be used to determine the Correctness Score (“Test cases to be used for validation”).

The difference is in the structure of the test cases. In the “Test case setup,” you can insert additional values into the database. This script is executed right after the creation of the database structure but before the test taker code.

In the case of asking the candidate to create a query, you define the expected result of the query given the database structure and the test case setup, as per figure 1 below.

 

Sequence of SQL scripts being executed in when asking a candidate to write a query (using SELECT)

Figure 1: Sequence of SQL scripts being executed when asking a candidate to write a query (using SELECT)

 


In the case of asking the candidate to write a database update, you should also write a checking query. This query will be executed after the test taker code. In the “Result,” you specify what the checking query is supposed to return in this case. See figure 2.

 

Figure 2: Sequence of SQL scripts being executed when asking a candidate to write a database update (using INSERT, UPDATE, or DELETE)

 


Please note that every test case recreates the database. Test cases are therefore isolated from each other.

The “Verification code” works in the same way as in other coding questions: this is the "model answer" and should be clean and efficient.

 

 

Versions

We use the following types and versions of each coding language:

  • Java: OpenJDK 12
  • C: GCC 10
  • C++: G++ 10
  • C#: SDK 3.1.300
  • JavaScript: Node 13
  • PHP 7.4
  • Python 3.8
  • SQL: SQLite 3.31.1
  • Typescript: 4.3.4
  • Ruby: 3
  • Go: 1.16

Caution: All questions must be your own original work.