Peer reviewing is something I think the students in my rather large lab learn how to do really well. The learning happens mainly due to how the process is set up.
- Professor/post-doc/senior grad student gets asked to review a paper (or usually, it's a few papers at once for a particular journal/conference). They place the papers along with the review form in a commonly accessible place, making a directory for each individual paper. It is made very clear that the papers are not for redistribution.
- An email is sent out to the relevant students about the location of the papers. If there are many to review, the students have to report back with the list of the papers in the order of preference (a day or so is usually given for this). This gives the students a chance to pick the papers that are most relevant to their work or are interesting to them otherwise.
- Once the students have indicated their preference, 4-5 students are assigned to each paper. Depending on how many papers there are overall, this can mean a different number of papers per student. Care is taken to put at least one senior grad student on each paper.
- Student reviewers are given a deadline by which they need to review the papers assigned to them. By the deadline, they submit a text file of a review into the corresponding paper's directory (this is a way to make sure everyone reads the papers and forms their own opinions by this time).
- The entire group (including professor(s)) gathers to do the reviews in person. Each person assigned to a particular paper gives their opinion of the paper. If you liked the paper, you have to defend why. If you didn't, you have to play the other side. It is a journal club of sorts. At the end, everybody gets a chance to rewrite their reviews within a day or so. A senior grad student volunteers or is assigned from this small subgroup to compile all the rewritten reviews into one coherent review.
- The coherent review is looked over by the person the committee assigned the review in the first place (this is sometimes the same senior grad student as in #5, but usually a professor is involved) and then gets submitted. Primary reviewer is noted, along with all the secondary reviewers that helped along the way.
- The submitted review is made available to all the secondary reviewers for learning purposes (one can see which of their comments were included, how the whole thing ended up being worded, etc.).
Of course, some of the things one needs to be a good reviewer is a broad and in-depth knowledge of the subject matter. But, it turns out, it is actually feasible to educate oneself reasonably well on a tangential subject in a day or so, so it is possible to be a reasonably good reviewer early on in your graduate school career. And things get much, much easier with practice and as one learns more about their field.
9 comments:
Great post! Thanks for following up on my concerns.
That sounds like an effective system, both for training and reviewing.
We don't have peer review for conference papers (because we don't have conference papers) so it's generally just a paper here and there. Is there ever any complaint that the primary reviewer has shared the manuscript with others? My general impression is that the process should be confidential. Perhaps this is field-specific? Or an attribute of lab culture?
The process remains confidential - the students are not allowed to distribute the papers further. In a way, this is a more open process where it is not the select few that decide the fate of all publications. I have not heard a complaint about this system, although this very well may be field-specific because the field is very fast-paced and we review many papers while given only 2-3 weeks to complete the review cycle.
This sounds like a really good process.
Almost sounds like a grant panel! What a great system for students.
I like that system! Especially the part about students and the PI working together as a group. In my PhD lab, papers were assigned to individual students, and you just learned by trial and error--looking at the final review the PI sent and seeing what the PI liked and didn't like about your comments. You figure it out eventually, but the first few times are scary!
Wow, this is such a great idea! Much better than the learning-on-the-job most of us have to do. :)
What a great system! My experience was more like MadHatter's. I remember the first time a revised paper was published with an extra experiment in it that I'd suggested - it was before any of my own papers were published, and I was so thrilled!
Yeah, I agree with everyone else-- this sounds awesome! I had very much the same experience as Mad Hatter and Cath, but I would have learned much faster (and more!) if I was part of a bigger process.
Jennie - it totally is a good process, although it does take a lot of time.
Albatross - I think this is similar to the end stages of a grant panel, which just might be where the system came from :)
MadHatter - with so many hierarchical levels, it is not uncommon to see a review the senior grad student has put together be pretty close to what actually gets submitted. A more gradual way to learn for the students and less work for the profs - it really is a win-win scheme!
Jane - this sounds like a rough one to learn on-the-job; I am glad I am getting the experience that will make it smoother later on!
Cath - it is amazing how much work one can put into reviews! Going all the way to suggesting experiments, etc. And it is very cool when the authors (who are likely to know the subject best) agree that those are relevant!
UnReaction - the bigger system certainly lets one learn by watching others and making it easier to speak up when in doubt since there are plenty of checks and balances to make sure only the right stuff makes it into the review.
Post a Comment