The risk of inbreeding, some CPAIOR statistics

Following on the previous post of Jean-Charles.

One of our reader suggested to have a look at the the workshop on organising worshops  and conference in CS: http://static.usenix.org/events/wowcs08/tech/

So I had a look and found a very interesting presentation and article "Best Practices for the Care and Feeding of a Program Committee, and Other Thoughts on Conference Organization" (Fred Douglis, IBM).

On PC composition he says in his presentation:

Avoid inbreeding, or the appearance of it

- Overlap from year to year

- Institutional overlap

Here is the extract on this topic from his article:

A number of conferences have a tendency to become rather inbred: they have a certain number of effectively permanent PC members, and only rotate a small fraction of their PC members from year to year. This is a bad idea. I believe that the core USENIX conferences, such as the annual conference and OSDI/NSDI1 are pretty good in this regard, as are some other conferences like SOSP. Some other systems conferences retain a much higher fraction of PC members, which I think results in a bit of tunnel vision, focusing on the same topics each year with much the same perceptions of what are good ideas and what are not.

Another possible aspect of inbreeding is the number of PC members from a particular organization or with a particular background. One USENIX security conference included a few people from one organization, and then the chair joined the same organization as the CFP came out, making it seem like he had selected 1/3 of the PC from his own organization. This looked bad to some, and while no one faults the chair for changing organizations, there would not have been an issue of the other people didn’t overlap so much. I can think of two other USENIX conferences that included over half the PC members with ties to the same department as the chair. I’m sure these PCs contained very talented people and I am not accusing them of bias; I am only suggesting that conferences need to avoid the appearance of being cliquish.

I think that conference organizers (such as USENIX) should establish guidelines for the number of PC members that can overlap in these respects, and then do a sanity check on PC lists prior to publishing the CFP. Some overlap with previous years is important, but too much overlap is terrible; finding that sweet spot would be a good topic for discussion at WOWCS. (I would recommend 20-30%.) Some conferences such as USENIX ATC have an informal policy of ensuring that a program chair serves on the PC the years before and after they chair it, which offers very strong continuity and should be adopted by all conferences.

One way to bring in new blood is to look at authors who have not previously served on the PC. When I chaired ATC’98, I took a USENIX bibliography to identify all authors of ATC or OSDI papers in the previous few years, then count their papers. I found a couple of people in my own department at AT&T who had published pretty much every year but never been on the PC ... and sure enough they both turned me down, despite my pleas for the need for authors to play their part as reviewers.

A corollary to my point about identifying people who have published but not served is that I think it is, in general, a tragedy to appoint someone to a PC who has never published at a conference, if the conference has been around for at least a couple of iterations. Are there people who could serve on a PC for conference X based on their experience at conferences Y and Z? Sure. But if they haven’t published at X, they either haven’t been submitting there (meaning they may not be that interested in the conference and also that they may not be well calibrated to the material normally published there) or they’ve been having submissions rejected. There are generally enough published authors from previous conferences that these authors should be tapped.

I just wanted to check if one of my favourite conferences, CPAIOR has a risk of inbreeding according to the indicators we should look at from the paper of Fred Douglis.

Over the past 7 conference 2007->2013:

18% of the PC has been the same for the past 8 years (counting on average 40 PC members)

2008: 54% of the PC common with 2007

2009: 57%% of the PC common with 2008

2010: 79% of the PC common with 2009

2011: 46% of the PC common with 2010

2012: 50% of the PC common with 2011

2013: 32% of the PC common with 2012 (with 22% of persons coming from the same institution!, I've already posted a message on this)

Another stat: 
50% of the union of PC's of CPAIOR have been in at least 4/7 last PC.
42% in at least 5/7
32% in at least 6/7

I’m afraid Fred Douglis would say CPAIOR has a risk of inbreeding …