The rationale was that we’d have a chance to field-test the homeworks and quizzes on real students, debug the questions and answers, and fix them up in time for the online class.

We just got through grading the first programming homework using the autograder.  Tellingly, students made a number of (understandable) errors that we hadn’t thought of, so we had to change our specfiles to give partial credit for cases where some of these unforeseen errors were made.  (The autograder runs a bunch of specs on each submission, possibly with different weights toward the overall score, and reports the overall score and which specs failed.)

We also found, not surprisingly, quiz questions that we thought were unambiguous but actually needed fixing.

And of course, we commonly make minor errors in lecture slides that are corrected after lecture, or add clarifications to lecture slides based on questions received in class.

So the bottom line is the online students will benefit from having had the lectures, homeworks and quizzes pre-tested by a talented (and patient!) group of students on campus.  The students of CS 169 Spring 2012 say “you’re welcome.”

• Every code example in the textbook, and many additional ones that are part of the lectures or the homeworks, are up on Pastebin.  So even students who doggedly refuse to learn an editor with syntax highlighting can see what they’re missing, plus Pastebin provides 1-click copy-and-paste so they can try the code themselves.  We created automation to keep the Pastebin links in sync with the URLs that appear in the book and ebook whenever the examples are updated and/or the book is revved.
• Many book chapters include screencasts to show how to use specific tools.  Anyone can watch these for free on Vimeo (albeit out of context), and we’ll be doing an iBooks version of the book in which the screencasts are embedded right into the ebook.
• There’s no glossary.  With Wikipedia, who needs one?  Instead, important terms in the book are linked to the corresponding Wikipedia entries.  (In the print book, they appear as URIs in the endnotes of each chapter, but in the Kindle book, the links are live as long as you’re connected to the Internet).
• At the moment there’s no index.  This may be a hardship if you only own the print version, but the electronic version is searchable.  We anticipate it will be common for print book owners to also own the electronic version (though sadly there’s no way for us to bundle the two purchases, since the print book is created and distributed by the CreateSpace print-on-demand shop and the Kindle ebook is distributed by Amazon, which ironically owns CreateSpace).
• Separately from term definitions via Wikipedia, the book includes various links to interesting news articles, YouTube videos, and other online materials related to the text.  If you’re reading the ebook on a device like the iPad or Kindle Fire, you can just click on the videos and watch them.  The print book has URIs that you have to type in manually.

So that’s what we’ve been doing and why.  The alpha edition, with some missing chapters, is now available (January 2012) at saasbook.info.  By March there will be some minor revisions to it and by Fall we hope to have a beta edition that is content-complete.

Fine for the 108 students in CS 169 this semester.  But how can we provide over 50,000 students the necessary software infrastructure (and our in-progress textbook) to participate in the class?

Simple answer—we don’t.  The on-campus class has been relying more and more on existing commercial SaaS tools and less on Berkeley’s own IT infrastructure, in part because we thought it was important for students to use the same services and tools they’d use if they were full-time developers.  The transition is now complete:

• We provide a VirtualBox-compatible VM image with all courseware (Ubuntu+ Ruby+ Rails+ gems+ ….) Thanks to Maggie Johnson for offering to have Google underwrite the nontrivial bandwidth charges for distributing a 1.5GB VM image to over 50,000 people via Google AppEngine.
• More than half the students signed up at saas-class.org are non-USA.  The print version of our book is beyond the reach of many of these, both because of shipping and because of its $20 price (low for a textbook, high for India).  Fortunately we had always planned on doing a Kindle edition anyway, which is priced at $9.99 (for now), can be obtained instantaneously, and can be updated on demand (we’re planning a new Kindle release just before the online course begins, and probably one more before the Beta Edition comes out in Fall 2012).
• Every student will open a free Heroku account for deploying their projects, so the autograders (and someday a Saucelabs-based grader, I hope) can check deployed code running on a public server.  Coursera apparently uses EC2 to run many instances of the autograder.
• For code that doesn’t get deployed, we’re developing an autograder that runs rspec, reek, flay and flog, so students’ code is evaluated for both correctness and good coding style.  The feedback may not be as deep as you’d get from an experienced TA looking over your shoulder, but the approach scales way better.
• Every student opens a Github account.  Since we need to protect homeworks against plagiarism, students need private repos for those.  The Github free account doesn’t allow that, but we’ve been working with Github to offer a free time-limited Micro account to the online students (many of whom are not traditional full-time students and so don’t qualify for Github’s free student micro plan).  Shout out to Chris Wanstrath and Kami Lott for making that happen!

Whew!  That’s the list so far.  In another post I’ll write up what we did in the textbook both to embrace the coming ebook tsunami and to make the book more interactive and connected to other online course materials, even if you get the dead-tree version.

I had already been thinking of autograding technology even for the on-campus course, since enrollment has grown from a low of about 35 when I first took over the course to a high of 108 this semester, and we continue to have chronic TA shortages.  For 50,000+ students, there’s no choice but to automate everything.

Given the apprenticeship aspect of learning to write good code, can we really give automated feedback on programming assignments in a meaningful way, beyond just testing whether the code runs and computes the right answer?

Coursera already had autograding for multiple-choice and short-answer (match a regexp) questions, but we wanted to be able to autograde programming assignments too.  To that end, Coursera created an API that allows an autograder to run arbitrary code to evaluate student homework submissions.  We’ve used that ability to create a tool that runs RSpec in a sandbox, so that we can do an automatic but detailed evaluation of student code.  For example, if an assignment specifically calls for them to use map() rather than writing a loop to express an operation on a collection, RSpec allows writing fine-grained tests that would check this.

Our goal is to extend the tool to also run reek, which looks for code smells, flog, which estimates cyclomatic complexity, and flay, which looks for duplicated code across many , so that we can provide automated feedback on coding style and best practices as well as code correctness.  We also expect to hook up SimpleCov or cover_me to give students feedback on their code coverage.  We’ll release this tool on Github as soon as it’s been field-tested.

Since we teach BDD, I’m also looking for a way to automatically run integration tests against students’ homeworks, which they deploy on Heroku.  So my next adventure is to get in touch with someone at Saucelabs and see what we can work out.

Anyone want to introduce me?

As with previous times I’ve taught the class, students will learn fundamental SW engineering techniques in the context of using agile methods to develop SaaS using Rails.  (High buzzword quotient for that sentence.)  This is the fifth time we’ve done a SaaS-oriented class and the second time that the official CS 169 offering has been taught this way, and we’ve been happy with the results.

But some major things are new this time around.  First, Dave and I are writing a textbook that we hope will allow others to teach this course.  The alpha edition, which is missing some chapters, is available now in print and Kindle formats (I’ll blog later about our experience self-publishing it).  Writing the book has also helped better organize the lecture material.

Second, Github and Google join the list of companies who are being super-cool in connecting the students in the course with great services/products at their own expense, and/or providing great guest speakers to come talk about life in the real SW world.  (Current list already included Pivotal Labs and Heroku.)

Third and scariest, Andrew Ng at Stanford persudaded us to offer the first 5 weeks of the course using the online-learning infrastructure he and Daphne Koller pioneered last year.  Anyone can sign up for free at saas-class.org —and so far over 53,000 people have!  Holy s**t.  Even with 90% attrition, 5,300 people would be more than I’ve cumulatively taught in my whole life.

Basically, we’ve refactored our 70-minute on-campus lectures into 6-7 ten-minute chunks each, which Andrew recommended as a near-optimal quantum.  In their system, each video chunk is supposed to include one or two self-assessment questions (multiple choice) that the student must answer correctly before proceeding, kind of like when you go to traffic school online to get a ticket wiped from your record.  I had already been getting good results using peer instruction sporadically in lecture, so we just systematized it to include a peer instruction question at the end of every chunk of lecture material; these same questions become the in-video self-assessment questions.

At this scale, everything has to be optimized for automatic grading and evaluation.  (Although the on-campus enrollment for CS 169 right now is 108, which is about triple its typical size, so I had already been thinking about auto-grading and design-for-gradability even before the online course happened.)  I’ll write more about that shortly.  But meanwhile, if you’re off-campus and interested, go ahead and sign up—hey, what’s one more?

We’ve always intended to deliver it primarily as an e-book, and while we have a very promising prototype of a highly interactive tablet version, we recognize that in the near term the mass market for e-books will be the Kindle format.

We‘ve been assuming many (most?) people will use the free Kindle Cloud Reader since it displays color and can take advantage of a larger screen than the Kindle hardware devices.  So we were excited when the Kindle Fire was announced—a color e-book reader that doubles as an Android tablet at an attractive price point ($199) might be something that gets real student uptake.  We ordered a couple to evaluate internally.

First, people are invariably going to compare it to the iPad.  I’d say there’s no comparison.  Besides Fire’s smaller size, its user experience is downright clunky compared to iOS.  (I’m new to Android, but if this is representative of the gap between iOS and Android, they’re two different products.)  Using the tablet is far from intuitive.  Not all of my books appear under the Books tab; some appear only under the Home tab.  And the Home tab is different from the Apps tab.  In trying to review the apps I have, there’s a bunch on there I don’t even understand (hidden apps that came with the device but aren’t visible on the Apps screen), and I can’t imagine how a nontechnical user could figure out what to do with them.  Frequently, dialogs will pop up that require typing, yet the on-screen blind-up keyboard will cover some of the fields, so that it’s impossible to see what you’re typing except by repeatedly hiding and re-showing the keyboard.  There are ever-present buttons along the bottom of the screen for Back, Home, and Search, but these buttons don’t work the same way (or at all) in every app.  For example, I downloaded a YouTube viewer app and was puzzled when the Search button didn’t present a search bar. Turns out the app had its own Search function that doesn’t use the Android-standard everpresent search button.  Big lose.

I didn’t try the media-consumption features (watching TV/movies, downloading songs, etc.), but they’ve gotten only lukewarm reviews from others.

The touch interface is flawed.  The multitouch hardware is fine, and the build quality of the device feels solid, but the sensitivity to actual gestures is badly off.  It’s nearly impossible to open a book from the “carousel view” without flipping past it.  Page turns are oversensitive, and you can’t hold the device in just one hand because every part of the screen responds to touch and you end up turning pages without meaning to.  (The original Kindle had this problem with button placement.  It’s amazing they didn’t learn from that experience.)  The page turning animation is jerky rather than smooth, and if you balk partway through the page-turning gesture, it’s quite hard to detect whether the page was actually turned or not except by actually re-parsing the words—big lose.

So it’s not an iPad—is it a better Kindle?  Maybe.  I have 3 other black-and-white Kindles (1st gen, DX, and graphite-colored 3rd gen), and my position has always been that those devices are one-trick ponies, but it’s a really good trick.  The 3G in particular is ultralight, can be read in bright sunlight, can be easily held in one hand, is no strain on the eyes, has a battery that lasts for days or weeks, etc.  In contrast, the Fire is too bright to read in a dark bedroom, even at the lowest brightness setting; eyestrain sets in quickly.  You can’t hold it in one hand, not only because of the misdesigned UI that results in gratuitous page turns, but because it’s just too heavy to hold comfortably despite its small size.  Its battery life is a few hours—by the end of my playing around with it on the first day for a couple of hours, it was down to 60% charge.  It seems to combine the disadvantages of the iPad with the disadvantages of a smaller screen (about 7”, or only slightly larger than the 6” screen on the 3G).

What about the rendering of actual Kindle books on the Fire?  It seems to be identical to the rendering in Kindle Cloud Reader, which isn’t great news since the layout possibilities are extremely limited.  (They would be on a small screen anyway, even with more powerful layout facilities.)  The Kindle Format 8 is supposed to improve on this, but I’ve been on their mailinglist since the preannounce and nothing has been announced to developers on what new formatting is possible in KF8.  For the moment, though, the device I’ll always throw in my bag is still the less-expensive black & white Kindle 3G.

Having said all that, the Fire will probably sell well.  Android phones have sold well even though the user experience is clunky compared to the iPhone.  Windows killed Apple on the desktop even though its user experience is a far cry from Mac OS X (or for that matter Mac OS 1 through 9).  Both sold because they were cheaper and because they looked good enough from afar, until you try to really get intimate with them.  As some technology columnist recently wrote, the Fire doesn’t have to be better than the iPad—it just has to be good enough.

I miss Steve Jobs.

1. Get a Clipper card at any Walgreens; you have to put a minimum of $5 on it.  You can add value at any Walgreens and any BART ticket machine.
2. Register the card at clippercard.com.
   Why:  Though not necessary to get the Berkeley benefits, this lets you add value online from a credit card or bank account, setup “autoload” (card reloads itself if balance falls below threshold), and protects your balance if your card is lost or stolen.
3. Register yourself at wageworks.com, which manages UCB’s transit benefits program.  You’ll need your SSN and possibly your UCB employee ID number.
   Why: so you can get $10 of free money on your Clipper card every month, and put additional pre-tax money on it.
4. Once you’re registered at WageWorks and logged in:
   1. click on the Commuter tab
   2. click Place Commuter Order
   3. select Public Transportation (not Parking or Vanpool)
   4. select “Clipper“ as your transit vendor
   5. select “E-cash” as the product
5. Enter the serial # of your Clipper card and select an amount to be loaded onto the card each month.  (You can change the order month to month, or leave it standing.)
   • The first $10 per month is free money (doesn’t affect your paycheck)
   • The next $230 per month is deducted from your pre-tax income on your paycheck
   • Anything beyond that is deducted from your post-tax income
   • $2 of your total monthly transfer amount is taken as a service fee, so if you sign up for just the $10 “free money” option, you really only get $8/month

That’s it.  I recommend setting the pay-with-pretax option based on your estimated usage, and using Autoload on the Clipper site as a backup with a threshold of $20.

Advanced version, to avoid paying $2/month service fee

If you’re OK leaving $24 of free money on the table per year to avoid some hassle, then just use the procedure above.  If you’re cheap or resent that $2 service fee, here is an alternative.

In step 4.5 above, instead of selecting E-cash, select WageWorks Commuter Card.  This will cause your full monthly transfer amount (ie no $2 fee deducted) to be put onto a debit MasterCard that WageWorks will send you, instead of onto Clipper.

You can use this debit MasterCard to:

• add value to your Clipper at BART machines and Muni machines
• add value to Clipper online, including via autoload

The downside is that you have to keep track of two balances—Clipper and debit card.  Especially if you have Autoload setup, if your debit card balance is too low to satisfy Autoload, Autoload fails, etc.  As well, you have to arrange to load your Clipper card as needed.

It would be great if there was a “pay-as-you-go deduction” option for funding Clipper, but so far there isn’t.

Various large libraries, including the New York Public Library and my local library the SFPL, are starting to allow loans of Kindle (and ePub) ebooks.

I used to think ebooks were the future, but actually, a better statement is that (most) print books are the soon-to-be-past.  I still appreciate the artisanal craft represented in some print books, though most exhibit little to none; but in terms of getting to the content, ebooks are the way to go.

I’m sufficiently convinced of this that the software engineering textbook being created by Dave Patterson and me will be available primarily as an ebook, with versions optimized for iOS, Android, and Kindle.  There will be a print version, but the e-versions will be enriched with reader-friendly features you can’t do in a print format.  And our goal is to have the ebook version rationally priced relative to the print version, and both versions much less expensive than a typical textbook.

Stay tuned.

Besides being one of the most poignant documentaries I’ve seen in some time, it drove home to me how ingrained is our instinct as a species to tell a story.  Here’s a group of people who, in general, have serious trouble driving a car, playing an instrument, writing poetry, doing athletics… yet they relish the opportunity to tell a story, and they rise to the challenge.

Tonia pointed out that two other behaviors we saw in the rehearsals—singing and laughing—also cut across levels of mental ability.

So there it is.  Telling a story, singing, and laughing—the universal constants.  I knew there was a reason I did this theater thing.

And if you harbor even the least bit of idealism about how theater can affect people, watching Yellow Brick Road will be an hour well spent.

“… and your little dog, too!”

Here’s my new proposed Terms Of Use agreement that such firms can use as a CYA in the future:

TERMS OF USE: You understand and agree that:

• Inevitably, this website will be compromised. I mean, sh** happens.
• We store much more of your confidential information than is really  necessary, so when the inevitable compromise occurs, you’ll be f**ked eight ways from Sunday.  But it won’t be our fault.
• When the media descends on us after a breach, we will loudly protest that we use “the strongest industry-standard security measures” and that we “monitor our site constantly”, all of which won’t matter because you’ll be f**ked eight ways from Sunday.
• We will quickly wring our collective hands and apologize that “this should never have happened”, that we “are investigating the incident vigorously”, and/or that we “will notify customers whose accounts appear to have been accessed improperly” (how we would actually know which ones, given that our site was compromised, will remain unclear).  In any case, though, what it will boil down to is: you’re f**ked eight ways from Sunday.
• If you have half a brain, you are already doing end-to-end encryption on stuff you care about (for example, as 1Password does when backing up your secret passwords database to Dropbox).
• We will encourage you to check the “remember this credit card for Romanian hackers^H^H^H^H^H^H^H^H^H^H^H^H^H^H future purchases” checkbox, but you probably shouldn’t.