Systematic reviews of measurement properties

VU medisch Centrum

E&B

EMGO institute for health and care research

Systematic reviews of measurement properties


A systematic review of measurement properties is a systematic review in which the content and measurement properties of measurement instruments are critically appraised and compared. Systematic reviews of measurement properties are useful tools for selecting a measurement instrument for a certain purpose. In addition, it are useful tools to identify measurement instruments that need further validation.

Different types of systematic reviews of measurement properties
Several types of systematic reviews of measurement properties exist. Examples are:


Published systematic reviews of measurement properties of health status measurement instruments.
We systematically collect all systematic reviews of measurement properties of all available measurement instruments that intend to measure (aspects of) health status or (health-related) quality of life. To this aim, regular searches are performed in Pubmed and Embase. The number of such systematic reviews has increased from none or one review per year in the beginning of the 1990s up to 85 in 2013. 


The reviews are now available in a searchable database. Click here to enter the database. 


The first 148 published systematic reviews have been described and their methodological quality has been evaluated. This study was published in Quality of Life Research in 2009 (open access). It was concluded that systematic reviews of measurement properties often are of low methodological quality. The most important methodological weaknesses of the reviews were a poor search strategy, a lack of adeqaute reporting of the methods used to perform the systematic review, and a lack of use of standards and criteria of adequacy to assess the methodological quality of the included studies. The poor quality of these reviews hampers evidence-based selection of instruments.

 

In 2014 a similar study was performed on the methodological quality of systematic reviews published in 2013. On average, the quality of systematic reviews has been improved but there is still a lot of room for improvement with regard to the search strategy, and especially the quality assessment of the included studies and the included instruments, and the data synthesis. This study was published in Quality of Life Research in 2015 (open access).

 

Performing a systematic review of measurement properties
In general, a systematic review of measurement properties is performed according to ten steps. Some steps are briefly described below. More more information, we refer to our book "Measurement in Medicine". A protocol is also available. A manuscript is currently being prepared about the methodology of systematic reviews of outcome measurement instruments.

Objective
We recommend to clearly define the type of review that one intends to perform. In addition, it is important to define the construct of interest and the population of interest. A comprehensive research aim should include four key elements: 1) the construct or the name(s) of the outcome measurement instrument(s) of interest; 2) the target population; 3) the type of measurement instrument of interest (e.g., questionnaires, clinical rating scales, performance-based tests, imaging tests, laboratory tests, etcetera), and 4) the measurement properties on which the review focuses (e.g., reliability, validity, responsiveness or all measurement properties).


Inclusion and exclusion criteria
The focus of the study should be the development or evaluation of the measurement properties of a measurement instrument. Often all studies are included that report on at least one or more measurement properties. Studies in which another measurement instrument is validated against the measurement instrument of interest, are often excluded because these studies only offer indirect evidence for validity of the measurement instrument of interest.

Literature search 

A comprehensive literature search is of great importance for finding all available evidence. MEDLINE (e.g. through PubMed or OVID) and EMBASE (e.g. through www.embase.com or OVID) are considered to be the minimum databases to be used when searching for evidence. In addition, we recommend to search in other specific databases as well, depending on the target population and construct of interest, for example in CINAHL, or PsycINFO.

 

A comprehensive search strategy consists of a broad collection of search terms for three of the four key elements of the research aim: 1) construct of interest; 2) target population, and 4) measurement properties. In principle, we recommend not using search terms for key element 3) ‘type of measurement instrument’ because often a wide variety of terminology is being used and the type of instrument is not always reported in the abstract which can lead to a high risk of missing relevant studies.

 

Search filters for finding studies on measurement properties

Studies on measurement properties are sometimes difficult to find in Pubmed or other databases due to poor indexing, large variation in terminology, and poor reporting of measurement properties. Therefore we developed two search filters for finding studies on measurement properties in Pubmed: 

 

The search filters were published: Terwee et al. Terwee CB, Jansma EP, Riphagen II, de Vet HCW. Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments. Qual Life Res 2009:18:115-1123. (open access)

 

A translation of this filter was made for EMBASE, and can be downloaded here.

 

Two translations were made for CINAHL. Because we don't know which one is the most sensitive, we leave it up to the users to decide which filter they want to use. Both filters can be downloaded here

 

Another translation was made for MEDLINE using OVID. This filter can be downloaded here.

 

It should be noted that none of these translated filters have been validated.

No translations for other databases are currenlty available. If researchers make a new translation and are willing to share their translation, please let us know.

 

Search filter for finding Patient-Reported Outcome Measures

If the review concerns Patient-Reported Outcome Measures, a PROM filter developed by the University of Oxford can be used. This filter can be downloaded here.

 

Search filters for many other kind of studies can be found at the website of the Centre for Reviews and Dissemination from the University of York and the website of the Health Information Research Unit from McMaster University.

 

Screening abstracts and full-text articles

We recommend to perform the selection of abstracts and full-text articles by two persons, independently from each other. Consensus should be reached afterwards, if necessary with a third independent person. This strategy is also recommended for systematic reviews of other studies, e.g. clinical trials and diagnostic studies.

Data extraction
It is also recommended to perform the data extraction by two persons, independently from each other. Consensus should be reached afterwards, if necessary with a third independent person.

Assessment of the methodological quality of the included studies
In a systematic review it is important to take the methodological quality of the included studies into account. If the results of high quality studies differ from the results of low quality studies, this can be an indication of bias. The COSMIN checklist can be used to evaluate the methodological quality of studies on measurement properties of health status measurement instruments.

Assessment of the results of the included studies
In this phase of a review criteria for what constitutes good measurement properties should be applied to the results of a study on measurement properties, to assess the quality of the included measurement instruments. An example of a checklist providing such criteria was published by our group:
 

Terwee et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007; 60:34-42.

 

An updated version of these criteria can be found here.

 

Conclusions
It is often useful to present a sort of ‘consumer’ table of the ratings of the measurement properties of all included measurement instruments. Examples of such tables can be found in these example reviews:

 

Schellingerhout JM, Heymans MW, Verhagen AP, de Vet HCW, Koes BW, Terwee CB. Measurement properties of translated versions of neck-specific questionnaires: a systematic review. BMC Med Res Methodol 2011;11:87.

 

Schellingerhout JM, Verhagen AP, Heymans MW, Koes BW, de Vet HCW, Terwee CB. Measurement properties of disease-specific questionnaires in patients with neck pain: a systematic review. Qual Life Res 2011, July 7 [epub ahead of print]. (open access)

 


Reporting
Clear reporting of the methods and results of the systematic review are important. We recommend to use the PRISMA guidelines for reporting systematic reviews. In published systematic reviews it is sometimes unclear if things were not done (e.g. data extraction performed by at least two independent reviewers) or if they were not reported. This makes it difficult to judge whether the review had been adequately performed. Recommendations for reporting systematic reviews of measurement properties can be found in our book and in this article:

 

 

Mokkink LB et al. Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Qual Life Res 2009;18:313-333. (open access)