Writing assignment tests

Tests run in an offline sandbox against a student's files and produce a score, passed/total, that the student sees and the gradebook records. Set them up in Test settings: open the assignment's Settings tab and click Edit beside Tests and solutions. The dialog has two tabs.

• Input/output tests say what the program should print for a given input. You write no test code. Most assignments need nothing more.
• Test command runs pytest, go test, JUnit, dotnet test, or your own checker script.

Input/output tests

Say students write a program that prints the first n Fibonacci numbers, where n is typed in.

1. Open Test settings. A new assignment opens on the Input/output tests tab.
2. Set the Run command, the command that starts the program, such as python3 main.py. The Language list fills it in for Python, Node.js, Go, Java and C#.
3. Add cases. Each case has an optional Input, typed into the program one line per prompt, and an Expected output.
4. Click a case's eye icon to hide it. Students run the visible cases with their Run Tests button. Hidden cases run only when you grade, so students can't code to their exact answers. Use a couple of visible cases to teach the format and a few hidden ones to decide the grade.
5. Under Try it, run against a solution and confirm every case passes before you publish.

For the Fibonacci program, add a visible case with input 5 and expected output 0 1 1 2 3, and a hidden case with input 10 and expected output 0 1 1 2 3 5 8 13 21 34.

How input reaches the program

The input box goes to the program's standard input, as if someone ran the program and typed at it. A case with input 5 behaves like running python3 main.py and typing 5, then Enter. The usual reading calls take it: input() in Python, Scanner.nextLine() in Java, fmt.Scan in Go, Console.ReadLine() in C#.

Put one value per line, in the order the program asks for them. A program that reads a name and then an age needs two lines:

Ada
36

The last line needs no trailing newline. Once the whole box is written, input closes. A program that keeps reading gets end-of-input instead of hanging, so a loop over input lines ends on its own.

Prompts are output. input("Enter n: ") prints Enter n: , so the program prints Enter n: 0 1 1 2 3. That passes under the default comparison, Output contains the text. Under Output matches exactly, which ignores one trailing newline on each side, it fails unless the expected output includes the prompt. Change the comparison in How output is compared under Advanced options.

Only standard output is compared. Anything the program writes to its error stream, a traceback included, is never matched. A failing case shows it under Errors from the program, which usually explains the failure. The exit code isn't checked either, so a program that prints the right text and then crashes still passes. If that matters, add a case that reaches the crash.

The run command goes through a shell, so redirection and && work. It runs once per case, from the assignment folder.

Each case is worth one point. Time limit (seconds) under Advanced options covers all cases together: 120 by default, which is also the maximum. Each case gets at most 15 seconds. Dependencies work as for a test command; see Dependencies.

Walkthrough: a pytest assignment

Students write a function add(a, b) in add.py. They get a public test they can read and run. A hidden test they never see also counts toward the grade.

1. Create a draft with four files

On the class's Assignments tab, click New assignment and upload a folder. Start from scratch or use a lesson if you prefer. A new assignment is a draft, invisible to students until you publish. Give it these four files.

README.md tells students what to build:

Create a file called add.py with a function add(a, b)
that returns the sum of its two arguments.

add.py is your reference solution. You hide it in step 3:

def add(a, b):
    return a + b

test_add.py is the public test. Students can read it and run it against their own code:

from add import add

def test_add_small_numbers():
    assert add(2, 2) == 4

def test_add_negative():
    assert add(-1, 1) == 0

test_add_hidden.py is the graded test. It stays hidden, so students can't see it or change it:

from add import add

def test_add_large_numbers():
    assert add(10**9, 1) == 10**9 + 1

def test_add_floats():
    assert add(0.5, 0.25) == 0.75

pytest finds any file named test_*.py on its own.

2. Set the test command

In Test settings, open the Test command tab and pick the Python (pytest) preset. It fills in:

python3 -m pytest -q

3. Hide the answer key and the graded test

Under File visibility, click the eye icon beside add.py and test_add_hidden.py. They move into the assignment's .teacher folder, which is never copied to students. Students get README.md and test_add.py.

4. Mark the solution and check it

Under Try it, choose A solution, tick add.py, and click Test it. It should score 4/4: two public tests and two hidden ones. If it doesn't, fix the tests or the solution before you publish.

Ticking add.py also keeps it out of graded runs. Grading lays your hidden files over each student's copy, so a hidden add.py that isn't ticked as a solution would replace every student's own add.py.

5. Publish

Click Publish at the top of the draft's page. Every student gets the public files. While a file from the assignment is open, their editor shows a Run Tests button. It runs python3 -m pytest -q on their own copy, where only test_add.py exists, so a working add.py scores 2/2. Grading runs the same command with test_add_hidden.py laid over their copy, so the gradebook records out of 4.

6. Keep editing after publishing

Everything in Test settings stays editable after you publish. A change applies to the next run, whether a student clicks Run Tests or you grade.

Hidden files live in the assignment's .teacher folder, which your file explorer marks with an eye-off icon. To fix a hidden test, edit it there, check it with Try it, then click Regrade all on the assignment's Student work tab, or Run all tests on the class's Gradebook tab.

Editing a public test file doesn't change the copies students already have, and grading runs their copy.

How a score is read

With Result parser on Automatic, the default, the score comes from the first of these that the output has:

1. A last line of standard output in the form N/M, such as 3/5. Print this from your own checker script.
2. The summary printed by pytest, go test -json, Maven Surefire, dotnet test, or Python unittest. Skipped tests count as failed.
3. The exit code: 0 scores 1/1, anything else 0/1. This step is skipped when the command runs a test framework. A framework run that prints no summary scores 0, and the run notes say why.

Python unittest results add up across files, so python3 test_a.py && python3 test_b.py reporting 1 and 26 passing tests records 27/27. An N/M score is read only from the last line, so a command that runs two scripts printing scores records the second one.

To force one reading, pick it as the parser instead: Score (N/M), Exit code, pytest, unittest, Go test, JUnit, dotnet or nbgrader.

Test settings

On the Test command tab, type a Test command or pick a Language preset. The command runs from the top of the assignment folder, with the student's files and your hidden files in the working directory.

Advanced options has three more settings:

• Timeout (seconds) limits the whole run. It is 120 by default, which is also the maximum.
• Result parser sets how the score is read, as described above.
• Graded command replaces the test command at grading time. Leave it blank to use the test command for both the student's Run Tests button and grading.

Public and hidden files

Every file in an assignment is public or hidden. Set it under File visibility in Test settings. Public files go to students, and their Run Tests button runs against them. Hidden files stay in the assignment's .teacher folder and never reach students.

A graded run starts from the student's copy and lays your hidden files over it, minus any you ticked as solution files. Each public file, a public test included, comes from the student's copy, which they can edit. Keep tests students should read in public files. Keep answer keys and every test that must not change in hidden files.

Moving a file into or out of .teacher from the terminal changes what new students receive. Only File visibility also updates copies students already have.

You can change visibility after publishing. Each change asks you to confirm, then applies to every student's copy. Hiding a file deletes it from their copies, but anyone who opened it may have read or copied it, so hide answer keys before you publish. Showing a hidden file adds it for every student who doesn't already have a file at that path. A student's own file is never overwritten.

Check a test with Try it

Try it in Test settings saves your test settings, then runs them the way grading does, with no student involved. It works on drafts and published assignments. Choose what to run against, then click Test it.

• Starter code runs against the files students receive, plus your hidden tests. The score is usually low. This confirms the command runs and reports a score.
• A solution adds the files you tick as the complete solution, and should score full marks. Ticked files are left out of graded runs. Ticking doesn't change their visibility.

Jupyter notebooks (nbgrader)

A Python notebook written with nbgrader cell metadata grades in the same sandbox as every other language. You write one source notebook holding the solutions and the autograded test cells with their point values. Students fill in a copy with the solutions removed. Only Python notebooks are supported.

Setting one up

1. Write the source notebook with nbgrader's cell toolbar (View > Cell Toolbar > Create Assignment in Jupyter, or the nbgrader labextension). Mark the solution cells, and mark the test cells as autograded with point values. nbgrader's own guide covers the metadata.
2. Create a draft assignment with that notebook, solutions still in it, plus any data files. You don't make a student version.
3. In Test settings, open the Test command tab and pick the notebook under Source notebook. This hides it from students, sets the test command and parser, and generates the student copy, with the solutions and hidden tests removed.
4. Use Try it before you publish. Starter code scores 0 out of the full total, since it grades the student copy. A solution should score full marks. If it scores less, fix the tests before students see the assignment.

Notebooks follow the same dependency rule as every other language. List anything the notebook imports beyond the standard library, such as numpy, pandas or matplotlib, in a requirements.txt at the top of the assignment folder. A package you installed only in your own workspace is missing when grading runs, so every cell scores 0 even though the notebook runs in the editor. The run output names the missing module.

Scores

Point values can be fractional. The gradebook stores whole numbers, so a fractional score is scaled to fit: 7.5 out of 10 records as 75/100. The run notes and the per-cell breakdown show the real points.

Manually graded cells count toward the total but earn nothing automatically. Leave them out of a notebook you want fully autograded.

Language reference

Python (pytest)

Put tests in test_*.py files.

python3 -m pytest -q

Python (script)

Print the score as passed/total on the last line of output. The Python (script) preset runs test_main.py:

from add import add

passed = total = 0
for args, expected in [((2, 2), 4), ((-1, 1), 0)]:
    total += 1
    if add(*args) == expected:
        passed += 1
print(f"{passed}/{total}")

python3 test_main.py

Go

Put tests in _test.go files. Keep -json in the command; the score is read from it.

go test -json ./...

Java (Maven)

mvn -o -q test

Pin maven-surefire-plugin to version 3.x in your pom.xml. Maven's built-in default predates JUnit 5, and with it your Jupiter tests don't run: the build succeeds with zero tests. Add this plugin block:

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.5.2</version>
    </plugin>
  </plugins>
</build>

C# (.NET)

dotnet test --nologo

Dependencies

Tests run with no network access, but you don't need to vendor or preinstall anything. Declare dependencies in the language's usual manifest at the top of the assignment folder. Before a test runs, they are resolved once, in a separate step that has network access, and cached. The test then runs offline against the cache. Resolution reruns only when the manifest changes.

• Python: requirements.txt. Packages are installed and importable in the tests.
• Go: go.mod and go.sum. Modules are fetched and vendored.
• Java: pom.xml. Dependencies go into a read-only repository the test run reads from.
• C#: your .csproj. Packages are restored into a read-only package folder the test run reads from.

An assignment that uses only the standard library has nothing to declare. JUnit 5, xUnit, NUnit and MSTest are already in the image, so a test framework alone needs no resolution step. Gradle isn't installed; use Maven for Java.

What students see

When tests are set up, students get a Run Tests button in the editor while a file from the assignment is open. For input/output tests it runs the visible cases against their current files and shows each case's input, expected output and what the program printed. If every case is hidden, there is no button. For a test command it runs the command on their own copy, without your hidden files, and shows the score, the command and the output. Hidden cases and hidden files never run for students, so a student whose visible tests all pass can still lose points at grading.