For Students‎ > ‎

Writing (and Reading) Papers


Some other good sources besides the hints I've collected below:



Prof. John Ousterhout's Hints for Reviewing Papers

How and How Not to Write a Good Systems Paper, by SOSP-9 chairs Roy Levin and Dave Redell

How to write a good OOPSLA paper.  The comments apply equally well to other conferences, IMHO.

Related: Prof. Dave Patterson's How to Have a Bad Career In Research/Academia


General pitfalls of early-career grad students, and how to avoid them



Writing is much harder than you think. Advice: If you don’t have a solid draft 1-2 weeks before the conference deadline, you’re starting with 2 strikes.

80% or more of submitted papers are rejected. Advice: You need feedback from colleagues and outsiders to improve your paper. A poor way to get feedback is to submit the paper, wait 3-6 months, and get a rejection with cryptic reviews. A better way is left as an exercise to the reader. (Thanks to Mike Franklin for this particular way of looking at the “get feedback” issue.) The reviewer has 20 other papers waiting to be reviewed and is looking for a reason to set yours aside and move on. Advice: your job is to ensure no such opening is provided—whether by unsupported statements, poor writing, rambling style, etc.

When you write up your work, remember that nobody cares what you did but only why it advances the state of the art. Advice: edit an outline and paragraph map before you start writing. It’s much easier to rearrange/eliminate at this level than at the prose level.

Your advisor can bring her/his experience to bear and give you advice (hence “advisor”) on how to maximize the likelihood of this, but don’t mislead yourself into thinking that your goal should be to please your advisor.  If the community is pleased with your work, as represented by the (usually anonymous) reviewers who will be evaluating your paper, your advisor will be too.  Advice: Get lots of feedback on a paper from people other than your advisor—i.e., people representative of the reviewers who’ll evaluate it—before submitting it.

Reading a Paper

Here’s what I think about when reading a paper (and what you should think about when writing one). The answer to each question tells you something about the technical content of the paper, whereas the ease of extracting the answer to each question tells you something about the quality of the writing. For example, a paper may have a really great main contribution that is so poorly expressed that it takes you a couple of passes just to figure out what the paper is “really” about.

  1. Is this a vision/position/direction paper, or a measurement/implementation paper?
    1. Did you learn anything new?
    2. Does it propose a clear hypothesis/research question and evidence to support/contradict it?
  2. If I know the area well, can I mentally slot this paper somewhere in the taxonomy? (“Differs from X as follows; has the following in common with Y;” etc.)  If the paper is radically brilliant, new, or iconoclastic work, this question may not apply.
  3. Can I summarize the single most important contribution in one or two sentences?
  4. What is the single biggest flaw/weakness in the contribution?
  5. If you’re reviewing the paper for a conference: will it likely generate discussion at the conference? Is it relevant to a broader community?
Vision Papers describe your grand scheme of the world and why it is good. You need some data to back up your statements, but this is not a detailed measurements paper. The goal of this paper is to convince the reader that your scheme is interesting, different, better than other schemes that have addressed similar problems, raises legitimate research questions, and is therefore worth spending the time to pursue research on.
Make no statement that cannot be backed up by citation, quantitative data, or at least a very good first-cut experiment (“preliminary results suggest….”)

An Implementation paper, by contrast, gives detailed measurements of a system that was perhaps described in a previous vision paper. The goal here is to demonstrate what you learned from actually building the system: Did it validate your research hypothesis? What came out differently than you expected, and why? How much better, quantitatively, is your design than others’? If you’re writing a measurement-and-results paper, first determine which graphs will convey the results you think are important. Given those, the paper will practically write itself.

Before You Write…

If possible, present your work in a short 5-10 minute talk. This helps identify strengths and weaknesses and will give you an idea of what other people see as the important contributions. Surprisingly often, they will spot a significant contribution that you totally overlooked, or suggest a novel application of your ideas that dramatically increases the relevance and impact of your paper.

Don’t cram.  Recall the old saw about how nine pregnant women cannot produce a baby in one month.  You can’t throw all your time into a paper at the last minute and expect a good result: you will become saturated, lose perspective because you are too close to the material, and ultimately be spinning your wheels, changing stuff back and forth without a really good feeling for why you’re doing it.  Exceptions to this occur, but they’re rare.

Know when to say when.  Even if you have written the paper with plenty of time and had a lot of outside review, after a certain point you will not be able to add much value without taking a break for a while (maybe a week or two).  When that point comes, further work on the paper is just thrashing and not likely to improve it much, though it will leave you feeling dissatisfied.

Actually Writing

Write from an outline. Let me say that again, because it’s really important: write from an outline. I know no one who can reel off any coherent technical writing more than one page long without some kind of top-down strategy. At least sketch out the major sections of the paper, and what points you want to make in each, from 10000 feet. If you write any complete sentences during this phase, you’re getting mired in detail already. Bullets are what you want.

Don’t even try to write the title or abstract until after the whole rest of the paper is written. Then, and only then, will you actually know what the hell it is you want to say.

Unless you’re writing a PhD thesis, your paper will make only a small number of discrete points–say 2 to 4. Each important point should appear 3 times: once in the abstract/introduction, once in the body of the paper (where it is explained in detail), and once in the conclusions (where you derive some implications of this point for the future of systems research, or whatever).  Bulleted conclusions can help. Remember that conference referees are at least as busy as you and they have to read several of these. Make sure they remember yours.

Paragraph map. For each subsection, write no more than two lines describing the purpose/goal of that subsection. This text will NOT be part of the paper – it is only there to remind you what you are trying to accomplish. It is ESSENTIAL that you be able to capture the purpose of a subsection in one or two lines. If you cannot do this, then you probably don’t understand what the subsection is really about, and when you try to write the text, it will be jumbled.

Then, for each subsection, map out specific paragraphs: for each paragraph, write one sentence that explains the topic or main goal of just that paragraph. Again, this sentence probably will NOT make it into the actual text. It’s important to keep it to one sentence. (As every style manual will tell you, including Strunk & White, virtually every well-formed paragraph does indeed have one sentence that explains the point of the paragraph, with the other sentences supporting or expanding on the point of the topic sentence.)  If you cannot fit the point of the paragraph into 1 sentence, the paragraph is probably making >1 point, so should be split into multiple paragraphs.

Read through everything you have written and see if it has a logical flow, ie if you believe it represents your work adequately.

Give what you have written to a technical colleague completely unfamiliar with your work (but able to understand the computer science part), have them read it, then have them tell you (without looking at it) what s/he thinks the main point and contributions are.

If all goes well, now replace the topic sentences with complete paragraphs.This way of writing will not yield a shakespearean work of literature, but it is consistent and will result in readable, logically organized prose by construction.

Revising/Editing

Your section organization will change. Sometimes it will be shuffled dramatically. This is fine; it means you’re understanding what presentation order works best. If you don’t go through at least three or four major revisions (where you move around or chop entire sections), it’s probably lousy.

After doing some edits on each draft, give it a full top-to-bottom reading to evaluate its coherence and flow of ideas. Then, take a couple of hours and do something else; once you get close enough to your paper, you start missing the forest for the trees.

Even early drafts are valuable for getting your colleagues’ comments. Get comments from people who you think may be skeptical of your approach. Get comments from people who will really rip your writing style apart. Remember, at least they are your friends; the conference referees probably are not.

Cite, cite, cite! Ask your colleagues for suggestions and pointers. You never want to be asked: “What about the work done by xxx, which obviously has something in common with your own?” (or worse: “…which refutes your own?”) Give due credit to those whose efforts you build on, as well as pointing out how your approach is different from (and better than) previous ones.

Don’t artificially formalize your writing style. Technical writing must be clear and concise. Overblown writing rarely fools anyone and it makes the paper boring to read.
  • Bad: “Problem X is clearly a critical area that impacts our research agenda and hypothesis.  Our ideas about problem X are embryonic and still evolving, and doubtless our ongoing work in this area will quickly yield fruitful results.”
  • Better: “We recognize that problem X is central to our agenda, but we have only begun to investigate it.”
If you haven’t read Strunk and White’s The Elements of Style, read it now. If you have, read it again. If you can’t organize a paragraph, you won’t have much luck organizing a chapter.

Omit needless words. Don’t be surprised if this turns out to be 30-40% of the words you originally wrote. Your first effort rarely captures the most vigorous or concise way to say something. Spend time tersifying.

Run your paper by someone who is anal retentive about grammar to catch common errors: misuse of which and that, non-words and non-phrases such as for all intensive purposes or irregardless, lack of parallel sentence structure…

Final Checks

Pat Hanrahan says: “The Future Work section must be earned”.  If you haven’t made us care about your contribution thus far, we won’t care to read Future Work either.

Armando says: There’s 3 kinds of statements in a systems paper: statements supported by citations, statements supported by experiments (yours or others’), and opinions.  Avoid opinions.

Remember that this will be read by people who (a) have never heard of you and the review is anonymous anyway, (b) have never heard of your project, (c) are reading about 15-20 papers apiece, all in different subject areas. They will spend the first 5 minutes deciding if your paper is actually good enough to be worth a fully detailed read; they will then spedn an hour or so reading it in detail, trying to figure out (a) what your contribution is, (b) if the contribution is substantial enough to be worth publishing, (c) if the contribution is “feasible” (ie it is implementable and therefore would be useful to someone).

  • Does the paper make clear precisely what your new contributions are, and how they are different/better than existing approaches to this or similar problems?
  • Does the outline of the paper (sections, subsections, etc.) cohere regardless of the granularity at which you view it? (The Outline mode of MS Word is a valuable feature for this check.  I also wrote a simple Perl script that does this for LaTeX files.)
  • Have you observed the following invariant: Before telling me what you did, tell me why I should care.
  • Have you had it read by at least one person familiar with each of the areas the paper impinges on? (Think of them as consultants in that area. There is a risk that you will get some of the details wrong in talking about an area that is tangential to the paper but that you’re not very familiar with, and if a reviewer happens to be versed in that area, it decreases your credibility. Such references are easy to get right, so there is no excuse.)
  • Have you searched carefully for any related work, and properly acknowledged it? The availability of papers and search indices on the Web makes it worse than ever to overlook significant related work.
  • Can you read only the abstract and conclusions and be able to give someone else a 30-second digest of what the paper claims it says?
Good reviewers are overloaded.  Remember, Butler Lampson is looking for an excuse to stop reading your paper.  Don’t give him one.

Comments