Contents Overview

This is an advanced guide to programming an autograder on the CodeHS platform. This guide assumes that you know how an autograder fundamentally works and details what tests are available.

To Create an Assignment with an Autograder

From your Assignments page, click into a module and lesson, and choose Add Assignment
Choose Coding Exercise and choose which program type you would like to create (i.e. Javascript Graphics or Console)
Add an assignment title, description, and any starter code for the assignment. Save your work as you go.
In the lower left corner of the modal, choose Advanced Settings
Navigate to the Autograders tab and click Add Autograder. A boilerplate autograder with example tests and comments will be added.
Write your code to create your Autograder and save your changes.

Animated gif showing how to create an autograder via the steps above.

Building Your Autograder

The Parameters

beforeRun

code is an object with keys student and solution

code.student is a String containing all of the student’s code
code.solution is a String containing all of the solution’s code

afterRun

output is an object mapping {String:object}. The keys are student and solution

output.student is an object mapping {String:array}
- output.student.graphics is an array of all graphic objects created by the student’s program
- output.student.console is an array of Strings. Each element is one line of output from the student’s program
  VERY IMPORTANT: this does not contain any text that was printed using readInt, readLine, etc
- output.student.runnerData contains information about the execution of the program
output.solution is an object mapping {String:array}
- output.solution.graphics is an array of all graphic objects created by the solution program
- output.solution.console is an array of Strings. Each element is one line of output from the solution program
  VERY IMPORTANT: this does not contain any text that was printed using readInt, readLine, etc
- output.solution.runnerData contains information about the execution of the program

Tests Methods

You can see the full test options in the Javascript Autograder Documentation.

To create a test, use expect(res) and then call one of the following test methods. Let res be the student’s output/code:

toBe(expected)

Checks that res == expected
Code: expect(res).toBe(expected)
Example: checks the first line of output prints a title
- expect(output.solution.student[0]).toBe(“Karel’s To Do List”);

toEqual(expected)

Checks that res.isEqual(expected)
We suggest using this method instead of toBe()
Code: expect(res).toEqual(expected)
Example: checks the first graphics object added to the Canvas is a circle
- expect(output.student.graphics[0].type).toEqual('Circle');

toContain(expected)

Checks that the string expected is contained in res
Code: expect(res).toContain(expected)
Example: checks the student has a while loop
- expect(code.student).toContain(“while”);

toBeGreaterThan (expected)

Checks res > expected
Code: expect(res).toBeGreaterThan (expected)
Example: checks the student printed at least 4 lines of output
- expect(output.student.console.length).toBeGreaterThan(3);

toBeGreaterThanOrEqualTo (expected)

Checks res >= expected
Code: expect(res).toBeGreaterThanOrEqualTo (expected)
Example: checks the student printed at least 4 lines of output
- expect(output.student.console.length).toBeGreaterThanOrEqualTo(4);

toBeLessThan(expected)

Checks res < expected
Code: expect(res).toBeLessThan(expected)

Example: checks the student used no more than 2 for loops

var fors = code.student.match(/for/g); 
// match takes a regular expression 
var numFors = fors ? fors.length:0;
 //make sure fors isn’t null
expect(numFors).toBeLessThan(3);

toBeLessThanOrEqualTo(expected)

Checks res <= expected
Code: expect(res).toBeLessThanOrEqualTo(expected)

Example: checks the student used no more than 2 rectangles

var rects = output.student.graphics.filter(function(elem){ return elem.type == "Rectangle"});// rects will be an array with only Rectangle objectsexpect(rects.length).toBeLessThanOrEqualTo(1);

toBeUndefined()

Checks res === ‘undefined’
Code: expect(res).toBeUndefined()

Example: checks the student did not use any for loops

//str.match returns null if there are no matches
var fors = code.student.match(/for/g);
expect(fors).toBeUndefined();

Test Options

With each of the previous methods, you can then call withOptions() on the expect object. This determines the message students will see in the Test Cases. The parameter is an object with the following possible keys:

testName is the test name displayed to the student

defaults to a string representation of the test. For example, something like “Expected “print(“hello world’)” to contain “print””

messagePass is the message displayed if the test passed.

defaults to “Success”

messageFail is the message displayed if the test failed.

defaults to “Try Again”

studentOutput is what is shown as the student output (labeled “your result”).

defaults to what’s passed to expect

solutionOutput is what is shown as the solution output.

defaults to the value passed to the expectation function (e.g expected or value from the examples above)

showDiff shows the difference between the student’s output and the solution output

defaults to False
Note: unless the program only passes with very specific formatting, students usually find this more confusing than helpful

Example that checks the student used a for loop and did not use a while loop.

This test produces the following test results. The code shown in Your Result is the student’s program. Note that there is no expected output shown.

General Tips

We suggest looking for key pieces of code instead of using expect(output.student.console).toEqual(output.solution.console);
To help debug your tests, you can set studentOutput or solutionOutput to see the values of your variables or print them using console.log();
- Remember studentOutput and solutionOutput should be String values
The browser has a built in JavaScript console. You can use the console to work out syntax errors or make sure your array functions are behaving as expected.
String comparisons are case sensitive and whitespace sensitive
- use str.toLowerCase() to make str all lowercase
- use str.replace(‘ ‘, ‘’) to remove spaces (but not all whitespace)
  - Note: the first parameter is a single space and the second is an empty string

Still have questions? Contact our team at hello@codehs.com to learn more!