a) Did reviewers' scores reflect real variations in methods and presentation?

b) Did extraneous factors (biases) affect reviewers' scores?

The null hypothesis is that reviewers' scores reflect real differences in quality in design/methods and presentation, and not extraneous factors (biases), such as the lead author’s sex or the country where the research was conducted.

The relevance of the study to the AIDS research/prevention community is that the test abstracts address current issues in AIDS prevention and treatment, and members of this community should gain from awareness of how peer review affects their work.

Reviewers were informed, some 3-4 months previously, that a maximum of two abstracts they receive would be part of a research project designed to evaluate the reliability of the grading process. However, they were not told which abstracts were fictitious, and it is likely that they were not conscious of the study while reviewing.

There were four fictitious abstracts. Each one dealt with a believable, current issue in AIDS research, applicable to at least one of the four speciality tracks of the Conference. Reviewers received one of several versions of a fictitious abstract, each version corresponding to a different combination of abstract characteristics:

• methods (good vs. poor)
• presentation (good English vs. awkward English)
• Results (positive vs. negative, or politically-correct versus other).
• Sex of lead author
• Professional degree of lead author (RN vs. MD/PhD).
• Country (France versus USA, and Russian Federation vs. USA).

Analysis of scores will involve abstract characteristics as well as reviewer characteristics (age, sex, previous AIDS reviewing, and time spent per review).

The study is funded by the Foundation Stiftung for AIDS research in Switzerland. The principal investigator is Dr. Bernard Hirschel. The randomized allocation of the fictitious abstracts to reviewers was made possible through the collaboration of Congrex, the professional conference organizer in Stockholm.

If you would like to see the original funding proposal, you can download proposal.doc. If you would like to see the detailed study proposal, you can download protocol.doc. If you would like to see all versions of the ficitictious abstracts, you can download test_abs.doc (72 pages).