I’ve had a few comments and questions from people who have seen both the print edition and ebook edition of ELLS about what toolchains we used to do all this.

One of these days I’ll extract the toolchain from the content and post it somewhere, but in the meantime here is a brief description.  (UPDATE: I’ve done this.)

LaTeX

Like most EECS academics, I use LaTeX.  I like the fact that it separates logical structure from visual and physical formatting, has extensive support for cross-referencing, and all the rest.  Very early on I knew this was the way to go, since our goal was to have a single set of source materials from which we could automatically generate multiple very different formats.

The LaTeX macros I came up with are nothing unusual for advanced LaTeX  users; essentially, each type of book element—sidebar, chapter teaser, elaboration, fallacy/pitfall, etc., gets its own macro.  In the PDF version, the macros expand into sometimes-nontrivial low-level TeX formatting.  Those of you who’ve worked with complex macros know that it is a minefield of \protect, \expandafter, \relax and all kinds of other incantations to make up for the fact that TeX kind of looks like a programming language, but really isn’t one.  The PDF output is produced by fairly conventional (if complex) macros and the usual latex-bibtex-latex-makeindex-latex flow that you’re already used to.

tex4ht (aka htlatex)

The two most popular ebook formats right now are .mobi (used by Amazon Kindle) and .epub (used by most non-Amazon ebooks); both are based on markup, with .mobi using a lobotomized-and-then-extended version of HTML 3.2 and .epub being essentially XHTML.  The ebook editions are produced by running the same input through tex4ht, which bolts a back-end capable of emitting XML or HTML onto the TeX engine.

Tex4ht is remarkably difficult to learn to configure and use: there’s a lot of moving parts, documentation is sparse (the single best source is The LaTeX Web Companion, but I still need to scour TeX user groups and newsgroups when I run into problems), and information about how to expand different kinds of TeX constructs is split across different files.  The tex4ht macros are, for example, the place where sidebars get typeset as special div’s with gray backgrounds, where fallacies and pitfalls get a small .gif icon inserted, and so on.  In other words, for the most part, the tex4ht macros expand into HTML or XML markup that do approximately the right thing for preparation to ebook conversion.

The output of tex4ht has to undergo considerable postprocessing.  In some cases it’s because ebook formats have additional proprietary tags that need to be added, such as <mbp:pagebreak/> to force a new page in the Mobi format.  In other cases it’s because the default behavior of tex4ht is to insert markup that makes something look ugly, and I wasn’t able to find a way to change the behavior or turn it off.  In yet other cases, it’s because bugs in tex4ht cause weird LaTeX escape sequences to “leak into” the HTML code.  In any case, there is a long-ish Ruby script that uses the Nokogiri XML parser (a Ruby front-end to libxml2) and lots of XPath-fu to do very serious chainsaw surgery to the output of tex4ht.  The main reason to use tex4ht is that it can digest LaTeX and preserve things like cross-references, which is very valuable.

Pictures, Screencasts, and Code

We prepared our figures using OmniGraffle, and the versions in the book are PDF-encapsulated resolution-independent vector graphics.  However, since ebooks essentially follow HTML-like conventions, the figures need to be converted to formats such as GIF or PNG for ebook inclusion.  As well, they need to be layer-flattened, color-quantized, alpha channels removed, etc., or they will choke the ebook converters.  Most of this is taken care of by good old Makefile technology and free programs such as convert (based on RMagick and ImageMagick).

Each code example in the book is kept in its own tiny file, with LaTeX wrapper macros like \codefile used to include it.  Besides allowing code to be dealt with separately for PDF vs. ebook, this also enables another piece of automation, a Ruby script I wrote that uses Pastebin’s RESTful API to automagically keep the URL’s to each code example in the book up-to-date as the examples change.  (And in the ebook version, the URL’s are actually live links to Pastebin.)  Pastebin has proven so useful for this that I post “demo code” that I use during lecture, so that students can get access to the code examples after lecture.  Similar automation keeps the in-book descriptions and links to the Vimeo screencasts up-to-date.

Ebook Conversion

The final step in creating an ebook is performed by KindleGen, a Mobi converter that Amazon modified and released as a free download to encourage Kindle authors.  KindleGen takes a single ginormous HTML file, a collection of referenced assets (images mostly), and some other arcane files, and creates a .mobi file suitable for uploading to Amazon’s Kindle Store.  One of the other arcane files it needs is the .opf (Open Packaging Format) file, which specifies book metadata such as the authors and ISBN number and BISAC codes; the KindleGen documentation includes an example of this.  The more challenging one is the .ncx file, which is used for navigation between chapters and providing a “page map” for the Kindle book. I wrote another Ruby script that generates this file automatically by parsing the output of LaTeX’s intermediate files created when it processes macros like \chapter and \section that generate TOC entries.

Design

The book’s icons and covers were designed by our talented alumnus Arthur Klepchukov, who’s been working with us on a contract basis to do this work and also single-handedly put together the iBooks edition.  For better or worse, though, the overall layout of the book elements in the print versions is my responsibility.  Dave and I heavily borrowed great ideas from his previous successful textbooks (though we didn’t actually copy the layout or typography, so no need to sue us, publishers!).

Make

All of the above steps are performed automatically by a fairly hairy Makefile.  The only manual adjustment needed is to ensure that sidebars and other elements don’t overflow the gutters or margins of the print book, since LaTeX often can’t ensure this on its own.  The most troublesome edition so far has been the iBooks edition, since iBooks Author does not support automatic importing of ePub; we’re hoping future versions of iBooks Author fix this.

So that’s how we start with a set of LaTeX sources, code snippets, PDF vector images, and MP4 movies, and produce multiple versions of a book.

And budding authors, before you ask: writing in LaTeX is more like programming than writing.  This definitely isn’t a “user friendly” solution for authors unless they happen to be math or engineering geeks already very comfortable with LaTeX and the Unix/makefile way of doing things.  There’s no point-and-click here, unless you count “pointing” at the screen in rage while “clicking” the ice cubes in your bourbon because some trivial error caused LaTeX to spew 400 lines of incomprehensible error messages.  So people who offer formatting services for self-publishers will have some job security for a while.