Thursday, May 22, 2014

The transitional form

Every new medium needs a transitional form exemplar—something that demonstrates the new technique or technology, but just replicates the previous techniques and doesn't use any of the unique affordances of the new medium.

The first movie was essentially the result of pointing a stationary motion picture camera at a stage where live actors performed a play.

The Lear-Siegler ADM-3, the first CRT terminal (predecessor to the ADM-3A), was no smarter than its Teletype printer forbears: there was no cursor control and only uppercase characters.  (Its predecessor was arguably Don Lancaster's TV Typewriter, construction plans for which are described in his TV Typewriter Cookbook.)

Show Boat, while acknowledged as the first true musical where the book (story) rather than the songs was the most important element, nonetheless borrowed heavily from its vaudeville traditions, with many "production numbers" unrelated to the story to show off the chorus and dancers.

That's where we are with MOOCs in 2014.  With a few exceptions, we have yet to take advantage of the new medium's affordances.

But without the ADM-3 we wouldn't have had graphics terminals, without the movie camera we wouldn't have Pixar, and without Show Boat we wouldn't have had West Side Story.  What will we have in a few years that we wouldn't have had without "MOOCs 1.0"?

Wednesday, May 21, 2014

Dealing with exploding demand for CS

UW CS chair Ed Lazowska makes the case nicely that interest in CS is booming, and that it's not likely to be just a flash in the pan, and how are we going to meet this demand and deal with larger course sizes?

In terms of offering a high-quality course experience to ever-larger numbers of students, my view (doubtless shared by others) is that we cannot just do the same thing we've been doing but more of it; we need to find new ways to do things.  Neither can we plop the students in front of a MOOC with minimal instructor guidance, though clearly those technologies have a role to play.

In the interest of trying to figure some of this out, I tried collecting some thoughts about this based on the Berkeley experience.  I presented this as a position at the recent Dagstuhl seminar about MOOCs, but I'm also on an internal committee in Berkeley CS charged with figuring out how to expand access to CS curriculum while keeping high quality, so these thoughts are percolating there as well.  Comments welcome.

Separating Scalable from Non-Scalable Elements to Refactor Residential Course Delivery


In Fall 2013, enrollment in UC Berkeley’s introductory CS course exceeded 1,000 for the first time, reflecting a trend in top CS departments for exploding demand for CS.


Most residential courses are offered in a “one size fits all” model: a single lecture, a single set of assignments or labs that everyone does, multiple recitation sections that generally cover the same material as one another, a lab session where everyone works on the same lab with TA’s on hand to help, and perhaps small-group tutoring sessions or office hours as the only place where students with differing needs get more customized attention.  The assumption is that this combination of elements serves the “mass of the distribution” of students, stragglers can squeak by with the additional support of office hours or tutoring, and superstars can use their spare time to do undergraduate research.

Yet as enrollments grow, we observe that (a) not all aspects of offering the course scale equally well in terms of instructor resources, and (b) the “outliers” (superstars and stragglers) become more and more pronounced, with the former representing underutilized talent and the latter an  inordinate drain on instructional staff time.

I argue that MOOC technology, in the format of a SPOC (Small Private Online Course), can provide the leverage needed to refactor course resources to improve how we serve these exploding populations by separating the “scalable” from “non-scalable” elements of the course and by carefully thinking about the role MOOCs can play in each part.

Things that scale well


Certain MOOCs have demonstrated that some elements of a course scale inexpensively and well:





  • Sophisticated automatic grading, such as used in many CS MOOCs, allow nontrivial assignments to be automatically graded nearly instantly (vs. handed back a week later) and with finer-grained feedback than TA’s or readers could provide.




  • In some domains, automatic problem generation builds on autograding to support mastery learning.  Both autograding and problem generation can take advantage of inexpensive public cloud computing rather than requiring extensive on-campus infrastructure.




  • Free video distribution (YouTube) makes lectures cheap to distribute.




  • Well-structured Q&A forums such as StackOverflow or Piazza allow students to help each other, with occasional intervention from teaching staff.




Things that scale poorly, and how to address them


More obvious variance across student cohorts.  Especially for courses with “broad but shallow” prerequisites, students’ levels of preparation may vary, and during the course, different cohorts of students may need help with different topics.   Mitigation: “Just-in-time” flexible deployment of teaching staff.


High-end outliers (superstars): Superstar students are often an underused resource.  As well, an instructor whose efforts are all expended on simply managing a large course is unable to identify these superstars (and sometimes they’re not obvious) and cultivate them further (invite as research assistants, etc.)


Mitigation: Identify ways to formally recognize and train these students to be effective in helping their peers.


Low-end outliers (stragglers): Stragglers can take a disproportionate amount of staff time, leading to an Amdahl’s Law-like effect limiting course scaling.  Mitigation: combine JIT deployment of teaching staff with SPOC resources that enable mastery learning.


Learning activities that are interaction-intensive: Especially in engineering courses, most real synthesis learning occurs in design projects, but these are grading-intensive and interaction-intensive.  Mitigation: Refactor the course into multiple courses, each of which concentrates either on “high scale” or “high touch” but not both, and resource the courses differently.


The main argument is that we must examine both creating new staff roles and amplifying the productivity and leverage of those roles using MOOC technology, thereby increasing overall teaching productivity.



Rethinking Teaching Staff: New Roles & Flexible Deployment


New teaching roles are already being created, but ad hoc/post hoc.  We need to formalize these roles, resource them, and train them.  One possible factoring of roles (with some overlap among them, e.g. some community stewards may also be contributors) might be as follows:





  1. Authors/Creators make an initial set of editorial decisions that result in a narrative through a body of material,  analogous to textbook authors.  We have argued that the combination of SPOCs and e-books is a promising formula for packaging such content.




  2. Core Contributors create additional material such as assignments, assessments, tutorials and other scaffolding  within the author-provided framework, which may be used and adapted by many instructors downstream.   They might, for example, participate in direct teaching during the school year and spend the summer doing course development or analyzing the previous semester’s learning outcomes data.  The SPOC delivery model and the software supporting it make it more convenient than ever to use SPOCs for “curricular technology transfer.”




  3. Community Stewards become experts on the materials and help other instructors (including TA’s and other teaching staff) work effectively with the much larger range of materials available in a SPOC (compared to traditional textbooks).




  4. Course Managers  keep courses running  smoothly by keeping tabs on student cohorts to understand who’s having difficulty where, marshalling and deploying instructional staff to respond to those needs, responding to escalations from instructional staff, and so on.  The course manager must have domain expertise comparable to a very strong student, and may also be responsible for handling violations of academic integrity in the course.




  5. Discussion Leaders facilitate small-group discussions (analogous to today’s recitation sections) using a combination of their own and provided materials.




  6. Tutors work with small groups of students on specific material with which they need help.




  7. Students/learners also help each other in person (e.g. “guerrilla lab sections” ) and virtually (e.g. discussion forums, hangouts).




These new positions will have to be recognized and trained.

Recognition:  Residential campuses currently recognize relatively few “official” teaching roles, such as lecturer, TA/head TA, lab TA/lab assistant, reader/grader.  New roles should be recognized with a combination of academic credit and stipends.  For example, graduate TA’s receive a stipend and tuition waiver, but are also required to complete certain teaching activities as part of their PhD preparation.  At some schools, undergraduates can also be either regular or lab TA’s.  At Berkeley, a third mechanism allows undergraduates to receive credit for “Teaching in EECS” even if they can only commit 2-4 hours a week (vs. the 10 hour minimum for regular TA’s.)

Training (teaching skills): Many campuses’ current orientations and courses for teaching assistants focus on training “full” TA’s who will teach sections, create materials, grade assessments, conduct review sessions, and more.  These courses therefore cover more than is necessary for some of the other roles.  For example, “dealing with disruptive students in class” is not a topic that (e.g.) tutors would need much experience with.  Some basic training for less-than-full-TA’s might cover:


  • how to help students stuck on problems without giving away the answer




  • how to “coach” students to effectively use techniques such as pair programming or peer grading, both to evaluate each others’ work and learn from the process of doing so,




Training (Orientation to the material): Both Code.org  and the NSF CS10K project aim to train high school teachers to deliver computer science courses.  Not only will the courses themselves be delivered as SPOCs, but they are creating “teacher training SPOCs” that will be combined with live and remote interaction (Google Hangouts) to train instructors on the use of materials.


Example 1: UC Berkeley/edX CS169x, Software Engineering

This MOOC was developed based on a campus course whose enrollment had also been growing, and it features many of the elements that “scale well,” including rigorous but automatically graded programming assignments.  Over 100,000 MOOC students have attempted the course and over 10,000 have earned certificates over five offerings of it.  The course now has a facilitator who is a faculty member at another university who became excited about the material after taking the course as a MOOC student.  He marshals the volunteer community TA’s drawn from alumni of previous offerings, but is also a contributor who has created his own materials.  He also stewards a community of classroom instructors using the material in their classrooms in a SPOC model.  A SPOC is a Small Private Online Course usually based on MOOC materials, but with heavy involvement of the instructor “on the ground” in customizing and facilitating the course.  I have argued elsewhere for the potential of this model and reported on successful initial trials using Berkeley MOOC materials in a SPOC setting at half a dozen universities, finding that different instructors use different subsets of the resources, some add their own, and some either don’t use our videos  or use them to increase their own understanding of the material before presenting to their own students.  In the meantime, the course’s original authors continue to improve the foundational materials and textbook, relying on the “network” of support to efficiently disseminate those changes and create new materials around them.



Example 2: UC Berkeley CS61A, Great Ideas in Software Development


This campus-based course, which has no corresponding MOOC as of this writing, is a rigorous introduction to the main paradigms of programming—procedural abstraction, data abstraction, functional, and logic.  It is based on a transliteration into Python of Abelson & Sussman’s renowned Structure & Interpretation of Computer Programs.  For the record-breaking 1100-student offering of the course in Fall 2013, the lecturer  would huddle with TA’s on a weekly basis to understand which topics students were having trouble with, and would then deploy a subset of teaching staff to create and run “guerrilla sections” specifically covering difficult topics.  Combined with making his lectures available online in advance of the live lecture, this meant that most students don’t attend lecture and most students don’t attend the same sections.  He also recruited star alumni from the previous semester to serve as tutors or lab assistants who committed only 2-4 hours per week in exchange for academic credit designated as “Teaching in EECS”, a teaching role that most CS courses have yet to exploit.  These helpers knew the material since they were alumni of the course, but received informal training on how to address common questions on homeworks/lab exercises without giving away the answers.



Summary




  1. The current “one size fits all” model of residential course delivery is a poor fit for exploding enrollments as well as for faculty productivity in an era of tightening budgets.




  2. Separating the scalable from the non-scalable parts of a course allows the two to be resourced separately.  The scalable parts can serve the mass of the distribution while the non-scalable parts can be resourced in a way more tailored to utilizing the outliers, both the superstars and the stragglers.




MOOC technology in the form of curated SPOCs, with appropriate new teaching roles supporting the course, can play a role in both the scalable and non-scalable elements.

Acknowledgments


These ideas come from conversations with the UC Berkeley Taskforce on Computer Science Curriculum (CS-TFOC), which includes David Culler, Dan Garcia, Björn Hartmann, David Wagner; John DeNero, Dave Patterson, and Andrew Huang (Berkeley); Mehran Sahami (Stanford); Saman Amarasinghe (MIT); Sam Joseph (Hawaii Pacific University, and lead facilitator for CS 169x on edX); and many others.

Thursday, May 8, 2014

Gallup poll: how your college experience affects fulfillment & well-being at work

This morning at my semi-monthly meeting with her, Cathy Koshland, our Vice Provost for Teaching, Learning, Academic Facilities & Programs, called my attention to the recently-released Gallup/Purdue report on what factors in alums' college experiences are predictive of whether they will be "engaged" at work—not just financially stable, but emotionally and intellectually connected to what they do in a way that makes it rewarding. Gallup’s previous extensive studies of workplace engagement have found that it correlates with important economic metrics such as productivity and employee healthcare costs, so even if you don’t care about some touchy-feely “people should feel good about their work,” there’s good reason to care about engagement.

So if a major goal of going to college is to “get a good job”—leaving aside whether that is the proper goal for college—and if "good" means something more than just financially remunerative, it's worth asking whether colleges are succeeding in doing this.

The timing of this report is fortuitous: my involvement in the world of MOOCs has made clear that while there seems to be some bland agreement that "there's more to college than taking classes" (and hence a college degree is worth more than the courses it comprises, and therefore MOOCs by and large won't replace that experience), it's been slippery to articulate what that "more" is beyond using vague words like "socialization", "professional networking", "mentoring", and so on.

In contrast, this report identifies specific elements of the college experience that are predictive of "workplace engagement."  Which factors (perhaps surprisingly) do not influence it?  Public vs. private college, race or ethnicity, whether you’re first in your family to go to college, the “selectivity” of your school (i.e. whether it’s in the USN&WR “top 100”).

Which factors do influence it? The most important ones—factors that double the odds of whether you’ll be engaged at work—are as follows, together with the fraction of respondents who agreed with the statement:

  • Whether you had a professor who you believed cared about you as a person (27%)

  • Whether you had a professor who made you excited about learning (63%)

  • Whether you had a mentor who encouraged you to pursue your dreams (22%)


The report’s unclear on whether these have to be three different people, but it is striking that these are all elements that residential colleges are much better positioned to provide than MOOCs, yet most students don’t experience them (and only 14% agreed with all three). The above three criteria are clustered as “[My college] is passionate about my long-term success.” Interestingly, while selectivity of the college didn’t influence engagement, graduates of for-profit private colleges (e.g. University of Phoenix) were less likely to be engaged at work (29% of respondents) than those of not-for-profit public (38%) or private (40%) colleges.

The other general area that doubles the odds of good workplace engagement is “College prepared me well for life after college.” This is broken down into subquestions having to do with internships, extracurriculars, and multi-semester projects such as research or volunteering, but it’s provocative to ask (I think) about the social aspect of “preparation” as well. A few of us CS faculty recently discussed Harry Lewis's provocative book Excellence Without a Soul, which charges that colleges in fact are doing a terrible job at social preparation (among other duties): they are infantilizing students by sheltering them from learning from their own mistakes, rather than using mistakes as character-building teachable moments where personal growth can occur; they allow students to self-segregate by class or ethnicity or whatever; and they make themselves appealing to students by shallowly providing what students myopically say they want—thereby creating conditions in which the students expect to "blame the system" when something happens to them (personal conflicts, lower-than-expected grade, etc.) and as a result are no better socialized when they graduate than when they began.

The other interesting observation has to do with when you graduated. “Engagement” is measured in terms of 5 subcategories in each of which you are said to be Thriving, Struggling, or Suffering. When graduates are grouped by decade, graduates in the 50s and 60s reported “thriving” in all 5 categories at double the rate of graduates in the 70s-80s, and 3 to 7 times the rate of graduates since the 90s. The report states that this “highlights the important role that age plays in determining the relative influence of experiences on one’s well-being,” but an alternative hypothesis is that this reflects the university’s abdication of moral standing and the development of students’ character, as Lewis charges.

So the good news is universities can indeed offer important experiences beyond course-taking that have a profound effect on graduates’ well-being when they enter the workforce; the bad news is for the most part they’re not doing a great job of it. Lewis’s book continues this thought nicely by providing specific observations on why, and even linking the observations to other problems like academic integrity violations, grade-point-grubbing, and the rancorous debates about the role of college athletics.