A new reviewing process

I totally agree with the remarks of Pierre.

I would like to present a new reviewing process that I will propose for CPAIOR-14

CP conferences are very special:

• publishing a paper in a conference is as hard (harder than?) as in a CP journal.
• a lot of members of the program committee (PC) publish papers in the conference

The first point gives too much importance to the conference publication. Reviewers are fighting each others. Some of them are killing papers because they thing that it can increase their chance to have their own paper accepted.
The second point is not really good and causes some problems. In my University, my CV has been reviewed by a Professor who is not in computer science. He/she write in the review: "we cannot count papers that are published in the conference in which the author is in the program committee". Therefore, he/she kills a huge number of my papers! I was upset! But is he/she wrong? I am not sure... When a best paper is given to a paper with an author who is in PC, I think we make a big mistake: the conference will not be well considered outside.

Since publishing in the conference is hard (in order to have low acceptation rate as we like so much), people are complaining a lot about the reviews they receive. The problem is that there are right most of the time. We cannot have ultra selective conferences, if the chairmans are not doing their job very well, for instance by reading all the reviews. Unfortunately, this is not possible because there are too many submitted papers.
Thus, a lot of things have been invented like:

• the authors may answer to the reviewer
• a senior PC is created

I don't really understand the first idea, because the reviewer has not to answer to the answer and it does not mean that a reviewer cannot write whatever he wants. It could just give more work to the chairmans because there are more things to read.

The second point is more interesting but it does not consider a big flaw in the general process:

Any Professor at the university knows that two correctors will give different evaluations for the same homework. He knows that it is really difficult to have several correctors without having this issue and that we need to define very precisely how the homeworks have to be evaluated if we expect the same final score.
For conferences, we have a lot a reviewers and, as if by magic, there is no problem to order, to compare and to discuss about a slight difference of scores. We like so much to use scores even if we know that it is so much difficult to evaluate in the same way the same work by two different persons! In order to help the chairman I introduced at CPAIOR-04 the notion of "confidence in the review", by thinking that it could help the chairman to classify the paper. However, it does not change the problem.

In addition, it is very dangerous to use the scores today, because people use them to kill some other papers and because the chairmen do not read all the reviews.

For instance, I recently received a review saying (I cite): "the proposed method is simple and elegant", then the reviewer says that the experiments are not sufficient to convince him, because he would like to see the use of the method on super computers. Roughtly, this is the review. Then it gave me :"strong reject". This is not normal. The chairman should complain about the reviewer. This is not possible to find a new method "simple and elegant", consistent for the classical benchmarks and complain about the absence hypothetical benchmarking and give a "strong reject". I have no problem with the rejection of the paper. I have a problem with the strong rejection. Since we use score this is important.

So, even by creating senior committee we have not the guarantee that the members will read all the papers and the reviews. In addition we still have the issue of the quality of the evaluation.
Thus, wan can we do?

I propose to apply the two following rules:

1. We accept a paper when at least one reviewer wants to accept it. In this case, the reviewer has to select a box "I want to see this paper accepted". We have just to ensure that the paper is not a republication of another work. The author will have to integrate the negative comments for improving his paper, but there are not part of the decision process. Honestly, if at least one person is interesting, this is positive!
2. When a reviewer reject a paper, he has to destroy it. This rule comes from Alain Colmerauer. If you keep a paper in your office it means that it is interesting, so you should accept it.

People are going to tell me: "your first rule is not reasonnable because friends are going to accept papers of other friends". Maybe, but I have three answers: a) it already happens. I can tell you that some papers are almost always reviewed by the same group of people working on the same topic.
b) some people are also trying to group together as reviewers in order to kill some papers. With my method, this part will be almost useless.
At least, if need we can propose another thing:
the name of the reviewers having accepted a paper is made public.

With this rule I guess that the selected papers by reviewer will change...