Wednesday, April 25, 2012

Needed: ebook authoring tools

I’ve been writing a self-published textbook that is sold in hardcopy, Kindle format, and an iOS app.  I firmly believe that ebooks are the future of textbooks, though unfortunately today’s e-textbooks are inferior versions of their print counterparts.  To that end, my co-author Dave Patterson and I took many steps to make the ebook version the preferred version of our book—not by crippling the print version, but by exploiting ebook features that you can’t do in print.

Our original vision was to make the richest version an iOS/Android version, in which we could use extensive CSS and JavaScript to make the book experience interactive in ways that the Kindle ebook format doesn’t support.  We originally tried using PhoneGap for this work, but it wasn’t stable because apparently it wasn’t designed to deal with such large assets (hundreds of KB per HTML “page”, plus many MB of images, plus GB of embedded screencast videos).  When Apple released iBooks Author, we realized we’d have to switch to that, since doing a “native” Android or iOS app was beyond our resources.  We ended up with much of what we wanted—better navigation both within and across chapters, embedded screencasts, and even interactive “check your understanding” question widgets that don’t reveal the answer until you try to answer the question.  But it came at great cost and pain: as we describe below, iBooks Author doesn’t import any useful markup format, even though it generates ePub.

Our experience with doing our own layout and publishing for the print and Kindle editions suggests that the majority of self-publishers and ebook authors may well want to work with a publisher.  The self-publishing process even even for simple hardcopy books is not trivial, but a process that can produce multiple formats with radically different output requirements from a single set of source files is definitely not for the faint of heart.  I described ina n earlier post how we generally did it for our book, concluding that  LaTeX is the worst way to prepare an ebook, except for all the others.

So what do we need?  A better unified authoring environment is a must.  What would my ideal authoring environment support?



  • Device-independent table specification: LaTeX’s table support is painful, but the real problem is that the aesthetics of table layout vary greatly per-platform.  A better platform-independent way of specifying table layout constraints is needed.

  • Tie-in to online resources:  we used Pastebin for code examples and Vimeo for screencasts to augment the content.  The RESTful APIs of these services allow high automation in cross-referencing the content—for example, since we agreed to put each code excerpt into its own file, I made a script that posts each code excerpt to Pastebin, notes the Pastebin URI, and modifies the book source (.tex files) in place to include the link.  I did the same with screencasts.  This could be automated in an authoring environment.

  • It seems likely most ebooks will use an XML-based format of some kind; exporting and importing such a format is essential.  Please, please, please don’t make me use a WYSIWYG editor.  When writing, I want to focus on the logicalstructure of the book and arrange the input the way it makes sense to me.  LaTeX is great for this. In fact, modulo table support, LaTeX isn’t a bad choice as a language, especially since it lets you use macros to define your own book elements, as we did with sidebars, chapter heads, sidebar graphics, pitfalls/fallacies, and soon.  (Though the way it interprets markup is highly user-unfriendly when an error occurs.)  It’d be great, of course, to have WYSIWYGpreviewers for the various output targets, and some people might even like a GUI front-end, but don’t make that the only way to edit!



If you’re thinking of self-publishing, be aware that you’ll have to take care of a lot of “little” typographic things that the publisher usually does, including but not limited to: getting “straight quotes” and “smart quotes”’ right; getting em-dashes and en-dashes rather than ’—’ and ‘–’; inserting © and ™ rather than typing ‘(C)’ or ‘(tm)’; dealing with ligatures (TeX does this for you, but most word processors don’t).  But the biggest one is going to be the hardest for Word users: getting accustomed to automating everything using macros so that changes are easy to make.  If you are used to manually styling your text (changing font size, applying indentation, etc.) rather than rigorously using styles, you’re sunk before you start, and so is your publisher (well, they’re not sunk; they’ll just charge you a lot of money for doing a labor-intensive manual job).

Here’s a list of our original “blue sky” desiderata for a software engineering ebook, and how we did on each one.



  • social networking for group annotations: not done, and we should wait to see how reader platforms’ feature wars about this feature shake out.

  • Code should be “live” — clicking it has effect of (eg) starting the app and dropping into a debugger on this function.  This is hard to do in general, since no current ebook platforms have the programmability to also act as a developer sandbox.  But our code snippets are all linked to Pastebin, allowing 1-click copy and paste into a terminal window or editor if you’re reading the ebook on a computer, and can even be pasted into and run on an Amazon EC2 instance.

  • Markup for code examples should be obvious.  Both the PDF and ebook outputs treat code specially.  The ebook version links each excerpt to 1-click-copyable source code on Pastebin, which has nice  syntax highlighting.

  • Figures should be  animatable (short movies): we currently have no animated figures, but we do have several screencasts sprinkled throughout; they’re hosted on Vimeo and ebook links are live, and in the iBooks Edition the content is bundled so they’re just embedded directly in the book.

  • “Where am I” (Web-style nav indentation) along side column or bottom of page:  iBooks Author does a nice job of this; Kindle doesn’t, even though we create the .ncx file that lists the names and location numbers of all the interesting “navigation points” in the book.

  • Margin notes (sidebars) in print  book are popups/hidden until tapped in ebook: Kindle format doesn’t support this, at least not yet.

  • “Pinning” figures or popups so they don’t go away while you flip  text underneath: Not done yet, but on the drawing board



I’ve been writing a self-published textbook that is sold in hardcopy, Kindle format, and an iOS app.  I firmly believe that ebooks are the future of textbooks, though unfortunately today’s e-textbooks are inferior versions of their print counterparts.  To that end, my co-author Dave Patterson and I took many steps to make the ebook version the preferred version of our book—not by crippling the print version, but by exploiting ebook features that you can’t do in print.If you’re thinking of self-publishing, be aware that you’ll have to take care of a lot of “little” typographic things that the publisher usually does, including but not limited to: getting “straight quotes” and “smart quotes”’ right; getting em-dashes and en-dashes rather than ’—’ and ‘–’; inserting © and ™ rather than typing ‘(C)’ or ‘(tm)’; dealing with ligatures (TeX does this for you, but most word processors don’t).  But the biggest one is going to be the hardest for Word users: getting accustomed to automating everything using macros so that changes are easy to make.  If you are used to manually styling your text (changing font size, applying indentation, etc.) rather than rigorously using styles, you’re sunk before you start, and so is your publisher (well, they’re not sunk; they’ll just charge you a lot of money for doing a labor-intensive manual job).Here’s a list of our original “blue sky” desiderata for a software engineering ebook, and how we did on each one.

  • social networking for group annotations: not done, and we should wait to see how reader platforms’ feature wars about this feature shake out.

  • Code should be “live” — clicking it has effect of (eg) starting the app and dropping into a debugger on this function.  This is hard to do in general, since no current ebook platforms have the programmability to also act as a developer sandbox.  But our code snippets are all linked to Pastebin, allowing 1-click copy and paste into a terminal window or editor if you’re reading the ebook on a computer, and can even be pasted into and run on an Amazon EC2 instance.

  • Markup for code examples should be obvious.  Both the PDF and ebook outputs treat code specially.  The ebook version links each excerpt to 1-click-copyable source code on Pastebin, which has nice  syntax highlighting.

  • Figures should be  animatable (short movies): we currently have no animated figures, but we do have several screencasts sprinkled throughout; they’re hosted on Vimeo and ebook links are live, and in the iBooks Edition the content is bundled so they’re just embedded directly in the book.

  • “Where am I” (Web-style nav indentation) along side column or bottom of page:  iBooks Author does a nice job of this; Kindle doesn’t, even though we create the .ncx file that lists the names and location numbers of all the interesting “navigation points” in the book.

  • Margin notes (sidebars) in print  book are popups/hidden until tapped in ebook: Kindle format doesn’t support this, at least not yet.

  • “Pinning” figures or popups so they don’t go away while you flip  text underneath: Not done yet, but on the drawing board

Monday, April 23, 2012

Crossing the software education chasm



Dave Patterson and I wrote an extended op-ed piece that appears in this month’s Communications of the ACM talking about how and why we reinvented UC Berkeley’s undergraduate software engineering course to bring it more in line with modern development methodologies.


Although at the time of writing we didn’t even know we were going to offer this content in an online course, as it turns out, the same reasons we believe the course worked well at Berkeley also allowed us to offer it as a MOOC.


What are your thoughts?

 Are you doing something similar at your institution?  Have suggestions for us?  Leave comments here or on the CACM article!




Tuesday, April 10, 2012

The UI on Amazon’s Kindle Fire is an embarrassing abject failure

I don’t get it.

Amazon’s mission is to be the world’s most customer-focused company, or something like that.

Did they even try to test out the Kindle Fire UI on any of their customers before inflicting it on us?

I’m really, really trying to like this device (our lab bought a couple to test out).  I’m a committed ebook fan and Kindle owner: I already own three Kindle hardware devices, not counting the Kindle Reader apps I have on my Macbook Air, iPad, and iPhone.  So I am predisposed to want to like this device.

If I download (buy) a book from the Kindle store, it shows up under Books.  If I sideload or download a PDF book, it shows up under Docs.  If I sideload or download a true ebook (.mobi file) notpurchased on Kindle store (e.g., purchased through Pragmatic Programmers, SitePoint, or any other of a number of technical publishers that distribute their ebooks directly off their site), it shows up in Docs.

When I go to Apps, my choices are “device” or “cloud”.  OK, I want to search for a new type of app, so I tap Cloud, and then Search.  No results.  Oh, I see, I have to go to the “store” to search for new apps.  That’s different from “cloud”.  In fact, the Search button seems to do something different and inconsistent in every context in which it’s available.

I have not found a decent free app for Google Maps, Google Calendar, Google News, Google Reader, or Gmail.  Weird given that Google is the company behind Android, the Fire’s OS.  (I know it’s an “Amazon adaptation” of Android, but it’s almost like they went out of their way to make it even moreclunky and unusable on the Fire than on other Android tablets.)  Yes, I can use the Web versions of those apps, but they don’t work particularly well on the Fire’s smaller screen.

The popup keyboard is unusable.  It’s too hard to describe why; try using it and you’ll see.  Auto-correction doesn’t work.  Different softkeys pop up at different times, so the softkey that was a left-parenthesis a second ago is now some random other thing, because you mistyped a word.  The key that means “shift” sometimes turns into the key that means “make the keyboard immediately disappear with no obvious way to bring it back.”  I have an iPhone and use the autocorrect all the time, and it’s graceful enough to give up and not correct your word if you “mistype” it twice in a row.  Fire’s autocorrect will correct your “wrong” word the same way again, and again, and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again,and again, until you just want to punch the device’s GorillaGlass screen.

The reading experience isn’t good.  I talked about this already in an earlier post.

The one and only thing this device has going for it—and the only reason I still try to use it at all: when reading in bed, its backlight means I don’t have to turn on a reading lamp.  This is important for my spouse, who doesn’t read in bed.  If it were up to me, I’d use my black & white Kindle 3G or Kindle DX exclusively.

I never thought I’d say this, but someone has come up with a mobile device experience that is worse than any version of Windows Mobile I’ve used, and I’ve used Windows Mobile since back when they were competing against the original PalmOS.

Apparently the Kindle Fire UI was designed by the same team that designed the unusable “Manage Your Kindle” page on Amazon.  In an era where Google and others have shown how tasteful use of JavaScript can make for a responsive and compelling UI, Amazon has figured out a way to make JavaScript so obtrusive that it makes the UI worse than if only HTML and Web 1.0 technniques were involved.  Don’t take my word for it; try it yourself.  I have about 140 items in my Kindle library, and “managing” them on Amazon’s site is just a joke.  It’s unusable.  There are bugs that cause random popup div’s with raw JSON to appear; the page is slow, and you can’t click on anything until it finishes loading because everything is re-rendering out from under you; no matter how many titles you own, you can never view more than 10 at a time, cannot filter the view, and cannot select multiple titles to perform a batch action (say, batch delete); you cannot manage “collections” and have them sync to your Kindle devices, as you can do with playlists in iTunes; I could go on.  If this UI was a class project at Berkeley it would get a C.

Thank heavens for Calibre.  It’s not perfect, but it’s by far the best overall ebook management app out there now.  And it’s free.  I couldn’t help myself, I sent a few tens of dollars as a donation to the author.  You should too, if only to get him on Amazon’s radar.  Amazon: buy this person, throw away all of your horrendous ebook “management” pages, and replace it with Calibre somehow.  Heck, rebrand and redistribute it and make the author rich.  He deserves it.

I know it’s possible to buy 2 Kindle Fires for the price of one iPad, but it’s also possible to buy two cans of dog food for the price of one sandwich—I’ll still take the sandwich.  Holy crap, does this device suck.  Too bad, because it would be good for customers if the iPad had some serious competition.  This device could have been that, but isn’t.