Check out this page for more detailed tutorials, tips, and code samples for creating Python Autograders.

Overview

This is a guide to programming a Python autograder and which tests are available on the CodeHS platform. This guide assumes prior knowledge of the basic functioning of an autograder, and provides further details on available parameters, test methods, and test options.

For more information on creating/editing autograders, feel free to check out this article: Creating Autograders to Check Student Code

Building a Python Autograder

Autograder Starter Code

When you add a Python autograder to an assignment, it comes preloaded with starter code containing example test cases. These examples are designed to help you get started quickly.

Screenshot of the autograder code with information on the test class is structured.

Understanding the Parameters

All parameters of the beforeRun() and afterRun() functions are strings (except self, which is the calling object).

beforeRun Function

student_code → The student’s program text.
solution_code → The solution program text.

afterRun Function

student_code → The student’s program text.
solution_code → The solution program text.
student_output → Output from the student’s program (excluding text printed via input).
solution_output → Output from the solution program.

Key Notes

To pass multiple sets of inputs, create separate classes.
Access the input list with self.inputs.
Store inputs as lists of strings to avoid type issues.
Place code analysis tests in beforeRun() and output checks in afterRun().
Remove the pass statement when implementing your functions.

Autograder Test Methods

To create a test, use expect( … ) and then call one of the following test methods. Let res be the student’s output/code.

## Example
expect(res).<test_method>(expected)

Available Methods

Method	Description	Example
`.to_contain(expected)`	Checks if `expected` is in `res`.	expect(student_code).to_contain("while")
`.not_to_contain(expected)`	Checks if `expected` is NOT in `res`.	expect(student_code).not_to_contain("pass")
`.to_be_less_than_or_equal_to(value)`	Checks if `res <= value`.	num_fors = student_code.count("for") expect(num_fors).to_be_less_than_or_equal_to(4)
`.to_be_greater_than(value)`	Checks if `res > value`.	expect(len(lines)).to_be_greater_than(3)
`.to_be_less_than(value)`	Checks if `res < value`.	expect(num_fors).to_be_less_than(5)
`.to_be_greater_than_or_equal_to(value)`	Checks if res >= value.	expect(len(lines)).to_be_greater_than_or_equal_to(4)
`.to_equal(value)`	Checks if `res == value`.	expect(lines[0]).to_equal("My Fun Program")
`.not_to_equal(value)`	Checks if `res != value`.	expect(lines[-1]).not_to_equal("Change this output")
`.to_be_truthy()`	Checks if `res == True`.	expect(loops_right).to_be_truthy()
`.to_be_falsey()`	Checks if `res == False`.	expect(code_changed).to_be_falsey()
`.to_be(obj)`	Checks if `res` is `obj`.	expect(res).to_be(expected)
`.not_to_be(obj)`	Checks if `res` is not `obj`.	expect(res).not_to_be(expected)

Test Options

After a test method, you can call .with_options() to customize:

Option	Purpose	Default
`test_name`	Name shown to the student.	String representation of the test.
`message_pass`	Message if the test passes.	(none)
`message_fail`	Message if the test fails.	(none)
`student_output`	What’s shown as the student’s output.	Passed `expect` value.
`solution_output`	What’s shown as the solution output.	Empty.
`show_diff`	Shows the difference between student and solution output.	`False`

Note: Unless formatting is critical, show_diff often confuses students more than it helps

Example: Loop Check

## This test checks if the student used a for loop and not a while loop

used_for = "for" in student_code
not_use_while = "while" not in student_code
loops_right = used_for and not_use_while

expect(loops_right).to_be_truthy().with_options(
    test_name="You should use a for loop for this program",
    message_pass="Great!",
    message_fail="You should not use a while loop!",
    student_output=student_code
)

## !! It's a good idea to set the student_output here; otherwise, student output would be the value of loops_right !!

Example: Check Last Line of Output

## This test checks if the student changed the last line of output

lines = student_output.split_lines()

expect(lines[-1]).not_to_equal("Change this output").with_options(
    test_name="You should customize the last line of output",
    message_pass="Great!",
    message_fail="Check your last line!"
)

## !! In this case, student_output will be the last line that the student printed !!

Testing with Input

To set the input to a program, values will need to be stored in the inputs list as strings.

inputs = ["3", "4"]  # 3 ft, 4 inches

If you want to test a multiple sets of inputs, you will need to write another test class:

## Test the first set of inputs.
class Suite01(PythonTestSuite):
    
    # Any values that should be passed to any call to `input`
    inputs = ["1", "2"]

    # Write any tests that should run before the code is evaluated
    def before_run(self, student_code, solution_code):
        expect(student_code).to_contain(self.inputs[0])

    # Write any tests that should run after the code is evaluated
    def after_run(self, student_code, solution_code, student_output, solution_output):
        expect(student_output).to_contain(self.inputs[1])
        

Suite01()

## Test the second set of inputs.
class Suite02(PythonTestSuite):
    
    # Any values that should be passed to any call to `input`
    inputs = ["3", "4"]

    # Write any tests that should run before the code is evaluated
    def before_run(self, student_code, solution_code):
        expect(student_code).to_contain(self.inputs[0])

    # Write any tests that should run after the code is evaluated
    def after_run(self, student_code, solution_code, student_output, solution_output):
        expect(student_output).to_contain(self.inputs[1])
        

Suite02()

💡 To use the inputs list in the before_run or after_run functions, use self.inputs.

Example

Problem: The student is supposed to ask the user for a number of feet and number of inches, then print out the number of inches.

Test Cases:

3 ft, 4 inches → 40 inches
0 ft, 6 inches → 6 inches
12 ft, 1 inches → 145 inches

The Autograder:

Tips for Input Tests:

Use self.inputs inside beforeRun() or afterRun().
Reuse test classes by copying them and changing only the inputs list and class name.
You can name the test classes anything you want; ensure to inherit from PythonTestSuite.

General Best Practices

Avoid using expect(student_output).to_equal(solution_output). Check key output elements instead.
Use my_string.lower() or my_string.replace() for flexible string matching.
Remove whitespace with: "".join(my_string.split())
Use strip_comments(my_string) to ignore comments when analyzing code.

Common Issues

Many autograders check for specific text within a student’s code (syntax). However, some may fail if the student uses a different spacing or indentation style.
To avoid this issue, you can remove whitespace from the code before performing text checks.

Example: Remove All Whitespace

## Removes all whitespace (spaces, tabs, newlines, etc.) from a string 
def remove_whitespace(my_string): 
     return ''.join(my_string.split())

You can also remove only spaces (while keeping tabs/newlines) with:

cleaned_variable = string.replace(' ', '')

Still have questions? Contact our team at hello@codehs.com to learn more!

Creating a Python Autograder