An insider report on the peer-review process for a top Software Engineering conference

Lionel Briand and André van der Hoek (PC Chairs of ICSE 2014, for those working on other research areas, I think it’s safe to say that ICSE is the most well-known research conference on Software Engineering) have published his analysis of the peer-review process for ICSE 2014 on the following report:

Insights and Lessons Learned from Analyzing ICSE 2014 Survey and Review Data

The first paragraphs reads as follows: “This document reports on survey and review data collected during and after the ICSE 2014 review process. Its goal is to provide full insights and lessons learned about the reviewing and decision making process that led to the selection of 99 scientific articles (out of 495) for the technical research program.”

Mouthwatering right ? If you’re a researcher (even if in a different area) I’d say this is a unique (?), let me know if you know other similar reports!, opportunity to get a better perspective of how PCs make their decisions.

Posted in evaluating research | Leave a comment

CyberChair vs EasyChair – Any reason to choose Cyberchair?

cyberchair vs easychairI’ve used EasyChair in all possible roles (author, reviewer, chair, proceedings manager,…) and Cyberchair as author and PC member. Based on this, I have no doubt, I’d choose EasyChair every time. From the login process (single account for all your conferences in easychair, a different account and login/pwd combination every time in Cyberchair) to the bidding, reviewing and discussion phase is much more intuitive (to me) in EasyChair. Moreover EasyChair is completely free (CyberChair not so much).

Still, many excellent conferences (MoDELS, ASE, ICSE, CAiSE,…) in my field use CyberChair so I may well be missing something. So, dear lazy web, my honest question to you is: “Can you give some reasons to choose CyberChair over EasyChair?”. Appreciated.

Posted in publishing, tools | Tagged , | 1 Comment

Cut the reviewing period for your conference – 80% of the reviews uploaded the last week

I took the time to count how many new reviews for the ECMFA’14 conference were uploaded each day to the easychair account for the conference.

The results, displayed below, are exactly what I was expecting (it’s also my own behaviour :-) ). Even if the reviewers had one full month to complete the reviews, 80% of them came in during the last week (day 30 in the graphic was the deadline, as part of the last week I count late reviews arrived days 31 and 32).

distribution of conference reviews

Even if this data is taken from a single conference (hey, this a rant blog post, not a scientific paper!) I’m sure you have the same feeling: no matter how much time you give to the reviewers, most of them they will always do the reviews last minute. If so, then, why do we need to give so much time to review (conference) papers? We could have a quicker turnaround (which should be one of the main benefits of sending a paper to a conference) if we just drastically cut the reviewing period and give just two weeks.

Based on our collective behaviour (there are always so many “urgents” things to do that we don´t plan, we just react so until we start getting the warning about upcoming deadlines we don´t put that on top of our to-do list) I don’t think the quality of the reviews would be worse than what we have now and authors would get their notifications earlier.

What do you think?

Posted in evaluating research, publishing | 10 Comments

Publishing my teaching book – My frustrations 18 months later

(cross-posted from the modeling languages portal)

Our Model-Driven Software Engineering in Practice book was published in September 2012. 18 months later I´d like to share with you some personal reflections on the publication (and marketing) process for the book. I belive that Marco and Manuel share most of these opinions but I´m just talking on my own behalf here.

I´ll skip the obivous (no, don’t even think about writing a book to make money, and yes, it takes much more time than what you ever thought) but focus on some details regarding the writing, publication and marketing process for the book that I found surprising (and to be honest, disappointing). I´ll try to be syntectic (if somebody wants more info on any specific topic just let me know). In fact, the main conclusion is simple. Going with a established publisher is just not worth it. Not surprised by this, we already clarified that we decided against self-publishing the book mainly because an “official” book would look better in our reserach CV (a longer explanation here). Clearly, now that I have already one, if I write another one ever again, I´ll self-publish it.

So, what I learnt in the process (of course, I talk based on the experience of writing one single book and interacting with one single publisher but I don’t believe things would be very different with others):

  • Established publishers won’t take any risk with your book. Unless they know you’re going to be a bets-seller, they will not take the pay for shelf-space for your book. Your book will only be sold online and following a print-on-demand model.
  • You can write in LaTeX. I would put this one in the win-win category. This is good for the publisher because they give you the template you should use and they basically forget about editing the book but, hey, we love LaTeX so this was also good for us. I do belive they should have been a little bit more helpful (e.g. we asked them to create an index of terms and they replied saying that if WE created it they would be happy to sell the book with it).
  • We don’t have the exact number of copies of the book we have sold (my approximation is around one per day since the day we published it). I’ve been using novelrank to follow at least somehow the sales on Amazon but until the first royalties check the editorial did not provide any sales information. And still now, I´m not sure I have the right number since the editorial not only sell the book separately but also as part of a collection and at that point things get fuzzy.
  • You get paid little and very late. One disadvantage of official publishers (not sure if I’ve found any advantage so far) is that they take 85% of the sales and you the remaining 15% (5% in my case since we are three authors). But even worse, in fact they 100% for one year and only after that period they pay you back your 15%. Just last month I got my first royalties check for the sales during 2012! If you’re curious the check was for 300 USD (after paying 30% of takes) which translates in a little more of 200 euros. So to be clear, one year after writing a book I’ve only made so far 200 euros!.
  • Understand that you and the publisher may have a different business model. You may think that you and your publisher are perfectly aligned, right? I mean you both want to sell as many copies of the book as possible. Well, this is not exaclty true. You want to sell your book but your publisher wants to sell its collection of books, which is a different story. Selling the full collection access to an institution, they make more money than selling individual books but you’ll make much less. They will put their marketing efforst in selling the collection not to help you sell your book. In fact, we had to put pressure on them to release a kindle version of the book. And they did it but they were not so keen on that for the simple reason that this is not a format on which they make a lot of money (price is fixed by Amazon) even if for the author this can make a significant difference
  • Forget about marketing. This was probably the biggest disappointment. I belived that by partnering with them we would have access to an audience that we could not reach directly (i.e. readers of this portal, fellow researchers and their institutions,…) which would be very good for the book since it’s especially targeted to people that are not already MDE believers. I know they did some basic mailing and printed some brochures they brought to some conferences (along with those of many other books) but the reality was that I’m convinced that more than 90% of the sales have come from our own efforst (the book web page, our own contacts, Marco’s participation at OMG meetings,…)

Of course, not everything is bad. We´re extremely excited that more than 50 organizations are using our book to introduce MDE to their students/employees but there is a bittersweet taste in my mouth. I do belive that with a little bit more of help we could have managed to have a greater impact for the book and for MDE.

Posted in Uncategorized | Leave a comment

Science vs Pseudoscience – Aiming at the former, doing the latter?

If I ask you whether you do “science” or “pseudoscience”, I’m sure you will answer that “science”. But after carefully looking at this comparison between science and pseudoscience (found online, can’t really point to a source though there is a blurred URL on the footer taken from here) I’m not so sure anymore.

I like to believe that I’m open to change my point of view if there is evidence against it and for sure all my work is peer reviewed, but not “ruthlessly” (we all know the typical quality of the paper reviews, specially for conference papers). And about the rest, well, be honest and decide by yourself.

(pseudo)science

If you liked this post, you should also follow my thoughts on twitter about software development or web design, WordPress and social media .

Be sociable, share!

Posted in Uncategorized | 2 Comments

Too much validation will kill you

Empirical validation is becoming a MUST if you want a publish a research paper on software engineering topics. In principle, this is a good thing, right? I mean, for sure, in a field like ours validating the usefulness/applicability/.. of our new methods/techniques/… can only be good.

The problem is that we are bringing this to the extreme and we know since Aristotle that a virtue is a point between a deficiency and an excess of a trait. Do NOT request validation for workshop papers (yes, I’ve seen this). In fact, do NOT request validation for many conferences. At this point we should still be in the proposal / maturity ideas phase so rejecting papers based on the lack of validation just kills creativity (if you don’t know how / have the resources to validate the idea and know your paper has no chance without it you may decide just to skip the topic) or, even worse, results in garbage validation.

Validation is only good for science if reviewers not only check whether some validation was done but also evaluate whether the validation was adequate (and this requires some knowledge on empirical validation that many don’t have). Accepting as validation experiments with students or with such a small groups of professionals that more than validation what we have is an anecdotical evidence is even worse than no validation at all.

When I started doing research, I experienced a similar phenomenon. Then, the “fever” (in my research niche) was the need to show some kind of prototype tool for the ideas of the paper. If you didn’t have an implementation section you had very little chances to get your paper accepted. Do you think this “pressure” contributed to make sure researchers started developing good tools that could benefit the community?. In case you wonder, the answer is a clear no. This only made all of us to mock up some shitty tools to capture a screenshot to put in the paper.

Validation yes but well done and when needed!

(and from a different perpsective and better reasoning, I recommend you to read Experiments as Research Validation – Have We Gone too Far? by Jeffrey D. Ullman )

If you liked this post, you should also follow my thoughts on twitter about software development or web design, WordPress and social media .

Be sociable, share!

Posted in evaluating research, publishing | 3 Comments

RR: The letter of a PhD student that decided to quit the thesis disappointed with the world of research

I found very interesting to read the list of reasons given by a PhD Student in his/her resignation letter (after four years working towards getting a PhD at EPFL).

The full letter can be read here. Though I don’t agree with all the points s/he makes, I do agree with a few of them. In fact, I dare to say they are well-known, my surprise is that s/he was able to realize this is how academia works so early. It took me quite a few more years to reach the same level of “enlightenment” (and start this “research rants” blog). This doesn’t mean that Academia completely sucks, IMHO the problem is that students enter the PhD with too high expecations so it’s also our task as advisors to make them understand that researchers don’t live in an ideal world but they are part of the real world which basically means that we, as any other “company”, need money to operate (and to get money you need to justify the quality of your work which end up producing some of the problems the student mentions in the letter).

Like it or not this is how life (including research) works. I just wish the best of luck to this student but I hope s/he doesn’t switch to a private company expecting a better working environment. Believe me I’ve been there and the problems (with co-workers, with “competitors”,…) are similar but the level of nastiness tends to be higher.

If you liked this post, you should also follow my thoughts on twitter about software development or web design, WordPress and social media .

Be sociable, share!

Posted in industry relations, job | 5 Comments