Validation of the COSMIN checklist
Content validity of the COSMIN checklist was assured in the Delphi study by consensus study among a large panel of experts from all over the world.
Reliability of the COSMIN checklist has been assessed in an international inter-rater reliability study. In this study 88 researchers (raters) participated. Each rater evaluated the quality of 3 articles on measurement properties. Each article (out of a total of 75 articles) was evaluated by 2-6 raters to assess inter-rater reliability. Overall, the percentage agreement was appropriate (68% was above 80% agreement), but the Kappa coefficients were low (61% below 0.40 and only 6% aboven 0.75). Reasons for low agreement were need for subjective judgement, and accustom to different standards, terminology, and definitions. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. The instructions inthe COSMIN manual have been improved.
This study was published:
Mokkink LB, Terwee CB, Gibbons E, Stratford PW, Alonso J, Patrick DL, Knol DL, Bouter LM, de Vet HCW. Inter-rater reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist. BMC Medical Research Methodology 2010;10:82. (open access)