This is an advanced guide to programming an autograder on the CodeHS platform. This guide assumes prior knowledge of the basic functioning of an autograder, and provides further details on available parameters, test methods, and test options.

Configuring an Assignment with a JavaScript Autograder

From the Create App

Access the Create Assignments tool by clicking Create from the left menu.
1. If you haven't done so yet, Create a Coding Exercise and choose one of the JavaScript options.
2. Add an assignment title, description, and any starter or solution code for the assignment.
3. Save your changes, and close the popup window, and refresh the page.
Click the three dots "..." next to an assignment you created and choose Edit.
Scroll down to the bottom of the window and choose Custom Autograder Edit.
At the bottom of the list, click the JavaScript General Autograder toggle to enable it. An autograder with example tests and comments will be added. You can also enable any of the other premade autograder checks.
Write your code to create your autograder and click Save Autograder Config.

Animated GIF demonstrating how to access the Autograder configuration window from the Create page in CodeHS

From the Assignments App

Click Assignments from the left menu. Choose the course where you'd like to add or edit an assignment by clicking the dropdown menu in the upper left corner
1. If you haven't done so yet, Create a Coding Exercise and choose one of the JavaScript options.
2. Add an assignment title, description, and any starter or solution code for the assignment.
3. Save your changes, and close the popup window, and refresh the page.
Click [Edit] next to an assignment you created.
Scroll down to the bottom of the window and choose Custom Autograder Edit.
At the bottom of the list, click the JavaScript General Autograder toggle to enable it. An autograder with example tests and comments will be added. You can also enable any of the other premade autograder checks.
Write your code to create your autograder and click Save Autograder Config.

Animated GIF demonstrating how to access the Autograder configuration window from the Assignments page in CodeHS

Building Your Autograder

The Parameters

beforeRun

code is an object with keys student and solution

code.student is a String containing all of the student’s code
code.solution is a String containing all of the solution’s code

afterRun

output is an object mapping {String:object}. The keys are student and solution

output.student is an object mapping {String:array}
- output.student.graphics is an array of all graphic objects created by the student’s program
- output.student.console is an array of Strings. Each element is one line of output from the student’s program
  VERY IMPORTANT: this does not contain any text that was printed using readInt, readLine, etc
- output.student.runnerData contains information about the execution of the program
output.solution is an object mapping {String:array}
- output.solution.graphics is an array of all graphic objects created by the solution program
- output.solution.console is an array of Strings. Each element is one line of output from the solution program
  VERY IMPORTANT: this does not contain any text that was printed using readInt, readLine, etc
- output.solution.runnerData contains information about the execution of the program

Tests Methods

To create a test, use expect(res) and then call one of the following test methods. Let res be the student’s output/code:

toBe(expected)

Checks that res == expected
Code: expect(res).toBe(expected)
Example: checks the first line of student output prints a title
- expect(output.student.console[0]).toBe(“Karel’s To Do List”);

toEqual(expected)

Checks that res.isEqual(expected)
We suggest using this method instead of toBe()
Code: expect(res).toEqual(expected)
Example: checks the first graphics object added to the Canvas is a circle
- expect(output.student.graphics[0].type).toEqual('Circle');

toContain(expected)

Checks that the string expected is contained in res
Code: expect(res).toContain(expected)
Example: checks the student has a while loop
- expect(code.student).toContain(“while”);

toBeGreaterThan (expected)

Checks res > expected
Code: expect(res).toBeGreaterThan (expected)
Example: checks the student printed at least 4 lines of output
- expect(output.student.console.length).toBeGreaterThan(3);

toBeGreaterThanOrEqualTo (expected)

Checks res >= expected
Code: expect(res).toBeGreaterThanOrEqualTo (expected)
Example: checks the student printed at least 4 lines of output
- expect(output.student.console.length).toBeGreaterThanOrEqualTo(4);

toBeLessThan(expected)

Checks res < expected
Code: expect(res).toBeLessThan(expected)

Example: checks the student used no more than 2 for loops

var fors = code.student.match(/for/g); 
// match takes a regular expression 
var numFors = fors ? fors.length:0;
 //make sure fors isn’t null
expect(numFors).toBeLessThan(3);

toBeLessThanOrEqualTo(expected)

Checks res <= expected
Code: expect(res).toBeLessThanOrEqualTo(expected)

Example: checks the student used no more than 2 rectangles

var rects = output.student.graphics.filter(function(elem){ return elem.type == "Rectangle"});// rects will be an array with only Rectangle objectsexpect(rects.length).toBeLessThanOrEqualTo(1);

toBeUndefined()

Checks res === ‘undefined’
Code: expect(res).toBeUndefined()

Example: checks the student did not use any for loops

//str.match returns null if there are no matches
var fors = code.student.match(/for/g);
expect(fors).toBeUndefined();

Test Options

With each of the previous methods, you can then call withOptions() on the expect object. This determines the message students will see in the Test Cases. The parameter is an object with the following possible keys:

testName is the test name displayed to the student

defaults to a string representation of the test. For example, something like “Expected “print(“hello world’)” to contain “print””

messagePass is the message displayed if the test passed.

defaults to “Success”

messageFail is the message displayed if the test failed.

defaults to “Try Again”

studentOutput is what is shown as the student output (labeled “your result”).

defaults to what’s passed to expect

solutionOutput is what is shown as the solution output.

defaults to the value passed to the expectation function (e.g expected or value from the examples above)

showDiff shows the difference between the student’s output and the solution output

defaults to False
Note: unless the program only passes with very specific formatting, students usually find this more confusing than helpful

Example that checks the student used a for loop and did not use a while loop.

This test produces the following test results. The code shown in Your Result is the student’s program. Note that there is no expected output shown.

General Tips

We suggest looking for key pieces of code instead of using expect(output.student.console).toEqual(output.solution.console);
To help debug your tests, you can set studentOutput or solutionOutput to see the values of your variables or print them using console.log();
- Remember studentOutput and solutionOutput should be String values
The browser has a built in JavaScript console. You can use the console to work out syntax errors or make sure your array functions are behaving as expected.
String comparisons are case sensitive and whitespace sensitive
- use str.toLowerCase() to make str all lowercase
- use str.replace(‘ ‘, ‘’) to remove spaces (but not all whitespace)
  - Note: the first parameter is a single space and the second is an empty string

Still have questions? Contact our team at hello@codehs.com to learn more!