Saturday, January 28, 2012

Meaningful feedback on 50,000 programming assigments?

Previously I mentioned that we’re offering the first five weeks of Berkeley’s CS169 course through the Coursera education portal (the startup formed to commercialize Andrew Ng and Daphne Koller’s online education system).

I had already been thinking of autograding technology even for the on-campus course, since enrollment has grown from a low of about 35 when I first took over the course to a high of 108 this semester, and we continue to have chronic TA shortages.  For 50,000+ students, there’s no choice but to automate everything.

Given the apprenticeship aspect of learning to write good code, can we really give automated feedback on programming assignments in a meaningful way, beyond just testing whether the code runs and computes the right answer?

Coursera already had autograding for multiple-choice and short-answer (match a regexp) questions, but we wanted to be able to autograde programming assignments too.  To that end, Coursera created an API that allows an autograder to run arbitrary code to evaluate student homework submissions.  We’ve used that ability to create a tool that runs RSpec in a sandbox, so that we can do an automatic but detailed evaluation of student code.  For example, if an assignment specifically calls for them to use map() rather than writing a loop to express an operation on a collection, RSpec allows writing fine-grained tests that would check this.

Our goal is to extend the tool to also run reek, which looks for code smells, flog, which estimates cyclomatic complexity, and flay, which looks for duplicated code across many , so that we can provide automated feedback on coding style and best practices as well as code correctness.  We also expect to hook up SimpleCov or cover_me to give students feedback on their code coverage.  We’ll release this tool on Github as soon as it’s been field-tested.

Since we teach BDD, I’m also looking for a way to automatically run integration tests against students’ homeworks, which they deploy on Heroku.  So my next adventure is to get in touch with someone at Saucelabs and see what we can work out.

Anyone want to introduce me?

No comments:

Post a Comment

Comments are disabled because the only commenters are spammers, despite Google's best efforts. But I welcome actual comments: Google my name and you can easily direct an email to me, and I'll publish your comment here.

Note: Only a member of this blog may post a comment.