Offer Automated Programming Labs to Students

Autolab is a course management service at CMU that enables instructors to offer programming labs to their students over the Internet. The two key ideas in Autolab are autograding, that is, programs evaluating other programs, and scoreboards.

Autolab also provides other services that instructors expect in a course management system, including gradebooks, rosters, handins/handouts, lab writeups, code annotation, manual grading, late penalties, grace days, cheat checking, meetings, partners, and bulk emails.

Autograding. Each time a student submits their work, the system autogrades it on a secure Linux virtual machine, and stores the results in a gradebook. Autograders and the programs they evaluate can be written in any language and use any software packages. The model for a traditional programming class is that students work on their code, hand it in once, and then get feedback a week or two later, at which point they've already moved on to the next assignment. Autograding, on the other hand, allows students to get immediate feedback on their performance.

Scoreboard. The latest autograded scores for each student are displayed, rank ordered, on a real-time scoreboard. The scoreboard is a fun and powerful motivation for students. When coupled with autograding, it creates a sense of community and a healthy competition that seems to benefit everyone. Students anonymize themselves on the scoreboard by giving themselves nicknames. A mix of curiosity and competitiveness drives the stronger students to be at the top of the scoreboard, and all students have a clear idea of what they need for full credit. In our experience, everyone wins.

Interested in using Autolab for your CMU course? Here are the instructions.


Who uses Autolab?

Each semester, Autolab serves over 2,500 CMU students in many different courses. In a typical semester it autogrades over 100K jobs. We're still actively developing the system, and we hope to eventually make it available to other schools.

CMU Courses

02-713: Algs and Data Structs for Scientists | s13 | s14
10-601: Machine Learning | f13| s14
10-601a: Machine Learning | s14
10-605: Machine Learning with Large Datasets | s14
10-701: Machine Learning | s14
15-110: Principles of Computing | f10| s11| m11| s12| f12| s13| f13| s14
15-112: Fundamentals of Programming | f11| s12| m12| f12| s13| m13| f13
15-112q: Fundamentals of Programming (Qatar) | f13
15-121: Introduction to Data Structures | f13| s14
15-122: Imperative Programming | s12| m12| f12| s13| m13 | f13| s14
15-122q: Imperative Programming (Qatar) | s13| s14


15-150: Functional Programming | s13| f13| s14
15-150q: Functional Programming (Qatar) | f13
15-210: Parallel and Sequential Data Structs and Algs | s13| f13| s14
15-213: Intro to Computer Systems | f10| s11| m11| f11| s12| m12| f12| s13| m13| f13| s14
15-213q: Intro to Computer Systems (Qatar) | f13
15-213p: Intro to Comp Sys (Peking Univ) | n12|f12|f13
15-237: Cross-Platform Mobile Web Apps | f12| s13
15-239: Software Development for Social Good | s14
15-312: Foundations of Programming Languages | s14
15-317q: Constructive Logic (Qatar) | s14
15-348q: Embedded Systems (Qatar) | s14
15-359: Probability and Computing | s13
15-411: Compiler Design | f11| f12| f13
15-424: Foundations of Cyber-Physical Systems | f13

15-440: Distributed Systems | f11| f12| s14
15-440w: Distributed Systems (Qatar) | f12
15-451: Algorithms | f13
15-732: Secure Software Systems | s13
15-746: Storage Systems | s12| s13| s14
18-732: Secure Software Systems | s14

Programming Contests

Companies recruiting at CMU use Autolab for various programming contests:

Citadel Programming Contest | f13
Delphix Puzzle Challenge | f13

The Autoloab Team

Tom Abraham

Tom Abraham: Lead developer, graduated May, 2013, currently at Yelp.


Dan Bucci

Daniel Bucci: Front-end developer, leading the upgrade to Rails 4, and working on the UI redesign.


Ilter Canberk

Ilter Canberk: Managing Autolab3 beta deployment and public web page refresh.


Ian Huang

Ian Huang: Front-end developer, working on the UI redesign.


Ian Huang

Yashas Kumar: Back-end developer, building a new RESTful autograding service on AWS.


Dave O'Hallaron

Dave O'Hallaron: Faculty lead.


Mihir Pandya

Mihir Pandya: Back-end developer, building a new RESTful autograding service on AWS.


An Wu

An Wu: Front-end developer, adding external authentication to the front-end.


Jimmy Zong

Jimmy Zong: Front-end developer, creating an open-source distribution of the Autolab service.