While we wish we could write autograders to check everything that the student does, there are limitations to what we can check and it is important to understand these limitations. For some lessons, the autograders may not thoroughly test the student code and teachers should make sure to review student submissions using our grading tools.
Autograders check program output
When a program outputs to the console, autograders are able to capture this output and compare it to expected results. Autograders cannot check the student code, which means that if the instruction say to add 5 and 5, the autograder can only look to see if the output is 10. Autograders cannot tell if the student adds 5 and 5, multiple 5 and 2, or just printed out 10. Lessons that focus on comments, or other changes to the code that are not reflected in the output cannot be tested.
Also, autograders can only test output from one method/function at a time. It is possible that two functions/methods are writing to the console, but the autograder only checks one at a time.
Autograders can interact with methods/functions
Once courses get into functions/methods, autograders are able to pass parameters and receive and test any return values. This allows the autograder to test multiple calls to the function to verify that it responds correctly to different parameters. For example, if an exercise asked the students to create a function/method to take miles as an input and return kilometers, the autograder can pass different miles to the student's function and verify that the return value is correct each time.
Autograders can input to a prompt
When a program prompts the user, the autograder can provide input to the prompts. This is helpful for many problems as it allows the autograder to check multiple runs with different inputs to prevent students from hardcoding answers. Unfortunately, programs that don't take input or use functions can limit the ability to test. Problems at the start of the course tend to fall in this category and it can be easy for a student to side step the problem.
Have questions or feedback? Contact our team at firstname.lastname@example.org!