Politically-correct program committeeing

Sariel and Suresh joins the discussion on the ethical behavior of conference program committees, picking up on Lance's (and his vistors') comments on Oded's suggestions.

Sariel mentions our colleague's shock at the suggestion that PC members should get outside information. I suspect her shock is related to the question of blind submissions. As a systems researcher, she is used to conferences where submissions omit the authors' names or affiliations, to avoid bias for or against certain authors or institutions. The fact that algorithms papers are not submitted blind is already shocking to many non-theoretical computer scientists.* Allowing a PC member to contact the author further erodes the illusion that papers are accepted or rejected solely on the basis of their content, not who wrote them.

But that is just an illusion. I agree with Paul Beame's comment: “To believe that extra information about a paper can be ruled out is to fool oneself about the reviewing process.” In fact, there are several back-channels:

• Theory people like to talk about their new results; active theory people give lots of talks. So naturally, active theory people tend to advertise their new results via talks long before they appear at conferences, often even before their paper is submitted. By the time a paper is submitted, several PC members may have already heard the talk and/or discussed the result with the author.

• PC members farm out papers to sub-referees, who may be more expert in the paper's particular topic than anyone on the committee. It's precisely those experts who are most likely to have extra information about the result: missing technical details, competing results, half-baked rumors, etc. IMHO, the PC must at least try to get these narrow experts' opinions; otherwise, good-looking garbage can (and does!) get accepted. (I know of at least one STOC/FOCS paper was incorrectly accepted for exactly that reason; it might even be the same one that Sariel mentioned. No, I won't tell you more, either.)

• Program committees for different conferences overlap. Sometimes the overlap is deliberate—for example, every STOC/FOCS program committee has at least one member of the previous FOCS/STOC committee—but more often it's caused by more active people being on more PCs. If a paper is submitted to a conference, rejected, and then submitted to another conference, someone on the new committee (or perhaps their expert subreferree) is likely to recognize the paper. And if the previous committee's comments weren't taken into account, the new submission is likely to be rejected with extreme prejudice. Sadly, it's fairly common for people to send exacly the same paper to one conference after another, until they find a committee who doesn't recognize it and who doesn't have the extertise to see the paper's flaws. (See my earlier example.)

• Occasionally the authors of two different papers on the same topic are asked to merge their results into a single paper, sometimes as a requirement for publication. (The committee may think the topic is interesting, but not interesting enough to publish twice.) This request gives each group of authors the oppportunity to criticially review the other paper, which they may see as competition.

• Many papers are submitted with appendices containing proofs of technical lemmas. Most of the time, these are simply ignored, since they're not part of the official submission, but not always.

• Authors with good reputations are more often given the benefit of the doubt. A hard discrete geometry proof is much more believable coming from a member of Miicha's Army than from an unknown student. (This is the usual justificaction for anonymous submissions.)
  

These extra sources of information often play crucial roles in deciding whether to accept or reject a paper. None of them are “fair”. They all favor people who are well-connected in the community. So given those back-channels, I have no problem adding another: PC members contacting authors for clarification. The extra information can be very helpful to the committee, but as Oded suggests, it does not always help the paper. Pointed questions are excellent for uncovering subtle bugs. If a PC member needs to contact the author for clarification, it should be done only with the committee's knowledge and consent (as Ken Clarkson suggests) and it should certainly count as a strike against the paper. But forbidding PC-author contact altogether is overkill.

---

*Symmetrically, theory people find it shocking that systems conferences can accept papers written by members of the program committee.

Comments

Why did you hyphenate "politically correct"? Isn't politically an adverb?

Isn't Sariel Har-Peled a he?

[Jeff Erickson]

(1) Temporary-hyphenitis. Won't happen-again.

(2) Yes, but-our systems-colleague is a she.

[Herve Bronnimann]

Olivier Devillers once told me, and I think it's an excellent suggestion, that PCs should judge papers only as submitted, and explicitly rule out corrections submitted to a PC member after the deadline, etc. I'm speaking mostly of the case where a paper presents a very good result but the proof is flawed, as pointed out by a PC member (sometimes determined by interaction with the authors during the evaluation process). It happened that a fix was found relatively easily and was "promised" in the final version. In that case, Olivier's position was that the paper should be rejected and resubmitted with the fix to another conference. Not all on the PC agreed, however, whence the polemics. But I agree with Olivier on this point (even though it was my paper that was hanging in the balance)!

[Herve Bronnimann]

Sorry, shouldn't have used the word "very good result". In truth the result wasn't that good :-) but it's not hard to generalize the discussion to other "very good" cases, and even then I would stand by Olivier's opinion.

[Herve Bronnimann]

Especially since a "very good" paper would get admitted to other conferences easily anyway.

Better stop those comments-to-comments-as-an-afterthought now.