The rationale was that we’d have a chance to field-test the homeworks and quizzes on real students, debug the questions and answers, and fix them up in time for the online class.

We just got through grading the first programming homework using the autograder.  Tellingly, students made a number of (understandable) errors that we hadn’t thought of, so we had to change our specfiles to give partial credit for cases where some of these unforeseen errors were made.  (The autograder runs a bunch of specs on each submission, possibly with different weights toward the overall score, and reports the overall score and which specs failed.)

We also found, not surprisingly, quiz questions that we thought were unambiguous but actually needed fixing.

And of course, we commonly make minor errors in lecture slides that are corrected after lecture, or add clarifications to lecture slides based on questions received in class.

So the bottom line is the online students will benefit from having had the lectures, homeworks and quizzes pre-tested by a talented (and patient!) group of students on campus.  The students of CS 169 Spring 2012 say “you’re welcome.”

• Every code example in the textbook, and many additional ones that are part of the lectures or the homeworks, are up on Pastebin.  So even students who doggedly refuse to learn an editor with syntax highlighting can see what they’re missing, plus Pastebin provides 1-click copy-and-paste so they can try the code themselves.  We created automation to keep the Pastebin links in sync with the URLs that appear in the book and ebook whenever the examples are updated and/or the book is revved.
• Many book chapters include screencasts to show how to use specific tools.  Anyone can watch these for free on Vimeo (albeit out of context), and we’ll be doing an iBooks version of the book in which the screencasts are embedded right into the ebook.
• There’s no glossary.  With Wikipedia, who needs one?  Instead, important terms in the book are linked to the corresponding Wikipedia entries.  (In the print book, they appear as URIs in the endnotes of each chapter, but in the Kindle book, the links are live as long as you’re connected to the Internet).
• At the moment there’s no index.  This may be a hardship if you only own the print version, but the electronic version is searchable.  We anticipate it will be common for print book owners to also own the electronic version (though sadly there’s no way for us to bundle the two purchases, since the print book is created and distributed by the CreateSpace print-on-demand shop and the Kindle ebook is distributed by Amazon, which ironically owns CreateSpace).
• Separately from term definitions via Wikipedia, the book includes various links to interesting news articles, YouTube videos, and other online materials related to the text.  If you’re reading the ebook on a device like the iPad or Kindle Fire, you can just click on the videos and watch them.  The print book has URIs that you have to type in manually.

So that’s what we’ve been doing and why.  The alpha edition, with some missing chapters, is now available (January 2012) at saasbook.info.  By March there will be some minor revisions to it and by Fall we hope to have a beta edition that is content-complete.

Fine for the 108 students in CS 169 this semester.  But how can we provide over 50,000 students the necessary software infrastructure (and our in-progress textbook) to participate in the class?

Simple answer—we don’t.  The on-campus class has been relying more and more on existing commercial SaaS tools and less on Berkeley’s own IT infrastructure, in part because we thought it was important for students to use the same services and tools they’d use if they were full-time developers.  The transition is now complete:

• We provide a VirtualBox-compatible VM image with all courseware (Ubuntu+ Ruby+ Rails+ gems+ ….) Thanks to Maggie Johnson for offering to have Google underwrite the nontrivial bandwidth charges for distributing a 1.5GB VM image to over 50,000 people via Google AppEngine.
• More than half the students signed up at saas-class.org are non-USA.  The print version of our book is beyond the reach of many of these, both because of shipping and because of its $20 price (low for a textbook, high for India).  Fortunately we had always planned on doing a Kindle edition anyway, which is priced at $9.99 (for now), can be obtained instantaneously, and can be updated on demand (we’re planning a new Kindle release just before the online course begins, and probably one more before the Beta Edition comes out in Fall 2012).
• Every student will open a free Heroku account for deploying their projects, so the autograders (and someday a Saucelabs-based grader, I hope) can check deployed code running on a public server.  Coursera apparently uses EC2 to run many instances of the autograder.
• For code that doesn’t get deployed, we’re developing an autograder that runs rspec, reek, flay and flog, so students’ code is evaluated for both correctness and good coding style.  The feedback may not be as deep as you’d get from an experienced TA looking over your shoulder, but the approach scales way better.
• Every student opens a Github account.  Since we need to protect homeworks against plagiarism, students need private repos for those.  The Github free account doesn’t allow that, but we’ve been working with Github to offer a free time-limited Micro account to the online students (many of whom are not traditional full-time students and so don’t qualify for Github’s free student micro plan).  Shout out to Chris Wanstrath and Kami Lott for making that happen!

Whew!  That’s the list so far.  In another post I’ll write up what we did in the textbook both to embrace the coming ebook tsunami and to make the book more interactive and connected to other online course materials, even if you get the dead-tree version.

I had already been thinking of autograding technology even for the on-campus course, since enrollment has grown from a low of about 35 when I first took over the course to a high of 108 this semester, and we continue to have chronic TA shortages.  For 50,000+ students, there’s no choice but to automate everything.

Given the apprenticeship aspect of learning to write good code, can we really give automated feedback on programming assignments in a meaningful way, beyond just testing whether the code runs and computes the right answer?

Coursera already had autograding for multiple-choice and short-answer (match a regexp) questions, but we wanted to be able to autograde programming assignments too.  To that end, Coursera created an API that allows an autograder to run arbitrary code to evaluate student homework submissions.  We’ve used that ability to create a tool that runs RSpec in a sandbox, so that we can do an automatic but detailed evaluation of student code.  For example, if an assignment specifically calls for them to use map() rather than writing a loop to express an operation on a collection, RSpec allows writing fine-grained tests that would check this.

Our goal is to extend the tool to also run reek, which looks for code smells, flog, which estimates cyclomatic complexity, and flay, which looks for duplicated code across many , so that we can provide automated feedback on coding style and best practices as well as code correctness.  We also expect to hook up SimpleCov or cover_me to give students feedback on their code coverage.  We’ll release this tool on Github as soon as it’s been field-tested.

Since we teach BDD, I’m also looking for a way to automatically run integration tests against students’ homeworks, which they deploy on Heroku.  So my next adventure is to get in touch with someone at Saucelabs and see what we can work out.

Anyone want to introduce me?

As with previous times I’ve taught the class, students will learn fundamental SW engineering techniques in the context of using agile methods to develop SaaS using Rails.  (High buzzword quotient for that sentence.)  This is the fifth time we’ve done a SaaS-oriented class and the second time that the official CS 169 offering has been taught this way, and we’ve been happy with the results.

But some major things are new this time around.  First, Dave and I are writing a textbook that we hope will allow others to teach this course.  The alpha edition, which is missing some chapters, is available now in print and Kindle formats (I’ll blog later about our experience self-publishing it).  Writing the book has also helped better organize the lecture material.

Second, Github and Google join the list of companies who are being super-cool in connecting the students in the course with great services/products at their own expense, and/or providing great guest speakers to come talk about life in the real SW world.  (Current list already included Pivotal Labs and Heroku.)

Third and scariest, Andrew Ng at Stanford persudaded us to offer the first 5 weeks of the course using the online-learning infrastructure he and Daphne Koller pioneered last year.  Anyone can sign up for free at saas-class.org —and so far over 53,000 people have!  Holy s**t.  Even with 90% attrition, 5,300 people would be more than I’ve cumulatively taught in my whole life.

Basically, we’ve refactored our 70-minute on-campus lectures into 6-7 ten-minute chunks each, which Andrew recommended as a near-optimal quantum.  In their system, each video chunk is supposed to include one or two self-assessment questions (multiple choice) that the student must answer correctly before proceeding, kind of like when you go to traffic school online to get a ticket wiped from your record.  I had already been getting good results using peer instruction sporadically in lecture, so we just systematized it to include a peer instruction question at the end of every chunk of lecture material; these same questions become the in-video self-assessment questions.

At this scale, everything has to be optimized for automatic grading and evaluation.  (Although the on-campus enrollment for CS 169 right now is 108, which is about triple its typical size, so I had already been thinking about auto-grading and design-for-gradability even before the online course happened.)  I’ll write more about that shortly.  But meanwhile, if you’re off-campus and interested, go ahead and sign up—hey, what’s one more?

We’ve always intended to deliver it primarily as an e-book, and while we have a very promising prototype of a highly interactive tablet version, we recognize that in the near term the mass market for e-books will be the Kindle format.

We‘ve been assuming many (most?) people will use the free Kindle Cloud Reader since it displays color and can take advantage of a larger screen than the Kindle hardware devices.  So we were excited when the Kindle Fire was announced—a color e-book reader that doubles as an Android tablet at an attractive price point ($199) might be something that gets real student uptake.  We ordered a couple to evaluate internally.

First, people are invariably going to compare it to the iPad.  I’d say there’s no comparison.  Besides Fire’s smaller size, its user experience is downright clunky compared to iOS.  (I’m new to Android, but if this is representative of the gap between iOS and Android, they’re two different products.)  Using the tablet is far from intuitive.  Not all of my books appear under the Books tab; some appear only under the Home tab.  And the Home tab is different from the Apps tab.  In trying to review the apps I have, there’s a bunch on there I don’t even understand (hidden apps that came with the device but aren’t visible on the Apps screen), and I can’t imagine how a nontechnical user could figure out what to do with them.  Frequently, dialogs will pop up that require typing, yet the on-screen blind-up keyboard will cover some of the fields, so that it’s impossible to see what you’re typing except by repeatedly hiding and re-showing the keyboard.  There are ever-present buttons along the bottom of the screen for Back, Home, and Search, but these buttons don’t work the same way (or at all) in every app.  For example, I downloaded a YouTube viewer app and was puzzled when the Search button didn’t present a search bar. Turns out the app had its own Search function that doesn’t use the Android-standard everpresent search button.  Big lose.

I didn’t try the media-consumption features (watching TV/movies, downloading songs, etc.), but they’ve gotten only lukewarm reviews from others.

The touch interface is flawed.  The multitouch hardware is fine, and the build quality of the device feels solid, but the sensitivity to actual gestures is badly off.  It’s nearly impossible to open a book from the “carousel view” without flipping past it.  Page turns are oversensitive, and you can’t hold the device in just one hand because every part of the screen responds to touch and you end up turning pages without meaning to.  (The original Kindle had this problem with button placement.  It’s amazing they didn’t learn from that experience.)  The page turning animation is jerky rather than smooth, and if you balk partway through the page-turning gesture, it’s quite hard to detect whether the page was actually turned or not except by actually re-parsing the words—big lose.

So it’s not an iPad—is it a better Kindle?  Maybe.  I have 3 other black-and-white Kindles (1st gen, DX, and graphite-colored 3rd gen), and my position has always been that those devices are one-trick ponies, but it’s a really good trick.  The 3G in particular is ultralight, can be read in bright sunlight, can be easily held in one hand, is no strain on the eyes, has a battery that lasts for days or weeks, etc.  In contrast, the Fire is too bright to read in a dark bedroom, even at the lowest brightness setting; eyestrain sets in quickly.  You can’t hold it in one hand, not only because of the misdesigned UI that results in gratuitous page turns, but because it’s just too heavy to hold comfortably despite its small size.  Its battery life is a few hours—by the end of my playing around with it on the first day for a couple of hours, it was down to 60% charge.  It seems to combine the disadvantages of the iPad with the disadvantages of a smaller screen (about 7”, or only slightly larger than the 6” screen on the 3G).

What about the rendering of actual Kindle books on the Fire?  It seems to be identical to the rendering in Kindle Cloud Reader, which isn’t great news since the layout possibilities are extremely limited.  (They would be on a small screen anyway, even with more powerful layout facilities.)  The Kindle Format 8 is supposed to improve on this, but I’ve been on their mailinglist since the preannounce and nothing has been announced to developers on what new formatting is possible in KF8.  For the moment, though, the device I’ll always throw in my bag is still the less-expensive black & white Kindle 3G.

Having said all that, the Fire will probably sell well.  Android phones have sold well even though the user experience is clunky compared to the iPhone.  Windows killed Apple on the desktop even though its user experience is a far cry from Mac OS X (or for that matter Mac OS 1 through 9).  Both sold because they were cheaper and because they looked good enough from afar, until you try to really get intimate with them.  As some technology columnist recently wrote, the Fire doesn’t have to be better than the iPad—it just has to be good enough.

I miss Steve Jobs.

Various large libraries, including the New York Public Library and my local library the SFPL, are starting to allow loans of Kindle (and ePub) ebooks.

I used to think ebooks were the future, but actually, a better statement is that (most) print books are the soon-to-be-past.  I still appreciate the artisanal craft represented in some print books, though most exhibit little to none; but in terms of getting to the content, ebooks are the way to go.

I’m sufficiently convinced of this that the software engineering textbook being created by Dave Patterson and me will be available primarily as an ebook, with versions optimized for iOS, Android, and Kindle.  There will be a print version, but the e-versions will be enriched with reader-friendly features you can’t do in a print format.  And our goal is to have the ebook version rationally priced relative to the print version, and both versions much less expensive than a typical textbook.

Stay tuned.

So I wanted to buy a specific title for a friend (The Soul of a New Machine, in fact) who would be in town in just 2 days.  Not wanting to pay 2-day Amazon shipping, I went online to Borders and “reserved” the book at a local Borders outlet.

When I went to pickup the book, there was nobody at the customer service desk.  While waiting for someone to show up there, I went over to the Search terminal to see if they had a copy of the DVD Triumph of the Nerds.  Searching for ‘triumph of the nerds’ returned over 13,000 hits (even when narrowed to Movies & TV).  Searching for the whole phrase returned 0 hits.  When the customer service person finally arrived (I had to go to the cashier and ask to have someone sent there), and told me that the book I had reserved would be at the cashier station, I asked him to help me with the search.  He tried without success and asked me whether I was sure the item was still available for sale.  (It is.)  I then asked if he could direct me to the section where I might find other books on the history of computers and technology.  He needed an example title to answer the question, so I suggested Insanely Great.  He did some unsuccessful searches and asked me whether I was sure the item was still available for sale.  (It’s in its second printing.)

I went home and ordered all the items from Amazon.  I’ll have to wait a couple days to get them, but (a) the search function found every item as a top hit on the first search attempt, and (b) I am paying less, even without considering sales tax.

Borders is fcuked, and probably so are the other big box stores. From now on it’s my neighborhood independent bookstore when possible, and Amazon otherwise.

I think this is a great idea and another innovation where California should lead the way, in addition to being a cost saving measure (and potentially an ecologically friendly one). But I was confused when NPR interviewed Sheila Jordan, Alameda County Superintendent of Education, on this topic. Superintendent Jordan spent much of the interview expressing concern about how a lot of students don’t have reliable access to computers and the Internet, and how teachers certainly don’t have the resources to print copies of textbooks in class for such students.

Now, I agree wholeheartedly with Dr. Jordan’s concerns, but when I think of “digital textbooks” I don’t think of people reading off a Web browser at home, or taking home pieces of paper printed off a Web browser at school. I think of downloading content onto an ebook reader—no paper waste, easier to read than a screen, can take it with you, etc. Granted, most students don’t have those either, but they are less expensive than computers (especially if purchased in bulk or otherwise subsidized, e.g. by textbook publishers), don’t require Internet connectivity (the Kindle relies on the 3G data network for downloads, or files can be sideloaded via USB), and are ecologically friendly.

I know that current ebook readers are targeted at people buying books online, but I’ve downloaded plenty of public domain content onto my Kindle–academic papers, public domain and no-license-fee books from various sources (textbooks and otherwise), and more. Devices like the large-screen Kindle DX and the iRex iLiad Book Edition are no-brainers for textbooks, at least where black & white is sufficient, and Amazon’s even announced pilot programs where they are seeding colleges with Kindle DX’s.

So one of us—either myself or Supt. Jordan—may have misunderstood what the Gov meant by “digital textbooks”. I hope my interpretation of his words is correct.

More worrisome is the politics of state-level textbook review in California and the sham process that seems to make the review system fundamentally broken (well, at least we don’t have stickers warning students about evolution on biology books!). This may be why the Gov proposed e-textbooks for math and science first, which are least susceptible to these issues, though I agree with Supt. Jordan that because of the way those books are used (students flip pages back and forth as they study) they might not be the best candidates for early adoption. At any rate, one may hope that the profileration of open source textbooks may change the dynamics of the state textbook review situation—authors of open-source aren’t producing them for profit and so are less likely to feel constrained by such guidelines, and the quality of the free books may be just too good to pass up.