Systematic Reviews

On this page you will find a guideline which outlines a systematic approach to the reveiw and selection of outcome measures for use in trials and studies.  Below the guideline you will find links to other information pertaining to the systematic review of outcome measures.

Outline Measures Review and Selection Guidelines

Aim 

The purpose of these guidelines is to suggest how one might select outcome measures (OMs) for all types of studies, not just clinical trials, in neuromuscular diseases (NMD).  The motivation for producing these guidelines is to ensure that there is an agreed process for choosing OMs that is more likely to result in correct choices being made from all the potentially available OMs in a way that is acceptable to the majority.

Principles

The precise methodology required for choosing an OM will differ according to the nature of the OM however some core principles can be applied to this exercise regardless of the precise methodology.

We suggest that the principles should be aimed at:

  1. Ensuring that the choices made will be widely accepted and lead to harmonization of OMs used.
    Harmonisation of OMs will reduce the effort of training and implementation especially for multi-centre studies and will allow direct comparison of studies whether by formal meta-analysis or other means.
  2. Avoiding duplication of effort at a number of levels.
    a.  The reviewing and recording each of the OMs when this may have been done by other groups.
    b.  Devising a new OM when there is an existing one that is either fit for the purpose or could be made so with less effort.
  3. Dissemination of knowledge about OM choices already made and about the real life performance of OMs in the NMD field.

Achieving these principles may be helped by:

  1. A team approach with agreement upon a team of people representing the interested parties. The team composition should reflect multiple disciplines eg doctors, metrologists, patient representation etc. The team should have representation from more than one institution (and more than one perspective).
  2. There needs to be an acknowledgment of conflicts of interest that any team member may have which may include their role in devising a particular OM or their long standing use and familiarity with an OM that they would prefer chosen.
  3. A clear statement of the choice being made and its purpose; what is being measured and why?
  4. Agreement upon a protocol that defines the purpose and agreed criteria for the choice of OMs in advance of doing the review. This will reduce post hoc decision making which tends to be biased.

What OM Choices?

The range of OMs chosen for a study usually covers some or all of the consequences of illnesses as published by the WHO International Classification of Impairments, Disabilities, and handicaps, (World Health Organization, 1980) and updated in 2001 as the International Classification of Functioning, Disability, and Health (ICF) (World Health Organization, 2001) as shown in Table 1. Thus OMs may measure pathology, impairment, disability or handicap and they may be direct or surrogate measures of these categories. It is therefore appropriate to compare OMs with ones in the same category. Thus the questions to be answered may be of the type:

  1. Which biomarker?
  2. Which assessment of muscle strength?
  3. Which functional assessment?

The selection of OMs can be specific for a given NMD, a range of NMDs, or specific to a particular study for a NMD. OM selections for specific studies could be different for each study in that the study may impose additional criteria upon the choice of OMs. For example a genotype phenotype correlation study may wish to use an OM to assess the severity of the phenotype. A well resourced and funded study of this type may be able to use a evaluator assessed OM such as manual muscle testing. A less well resourced study may decide to use a patient completed functional scale.

Some studies especially clinical trials may require a range of OMs and may therefore require OM choices to be made at several levels:

  1. The categories of OM required for the trial ie pathology, impairment, functional, QoL etc?
  2. Which OM for each required category?
  3. Which will be the primary and the secondary OMs?
Primary verses Secondary OMs

Clinical trials usually have several OMs one of which is the primary OM while the rest are designated secondary OMs. A trial is usually regarded as a positive or negative trial based upon the performance of the primary OM during the trial. The choice of primary OM for a trial is mainly dependant on the purpose of the trial and is therefore trial specific. Phase 2 or proof of concept trials will tend towards pathology measures or biomarkers while phase 3 trials will tend towards impairment or functional measures as their primary OM. For example:

  1. A phase 2 trial of anti-oxidants for SMA may choose reduction in the level of a biomarker or oxidative stress as the primary OM. After all if the treatment does not actually reduce oxidative stress then it can hardly be expected to improve other OMs downstream (Unless of course, the anticipated mechanism whereby a putative therapy might work, is misunderstood.)
  2. A genetic therapy for Duchenne MD may use expression of dystrophin in muscle biopsy samples as the primary OM.

The first and third level choices for a given trial are more properly dealt with in the context of the planning for a specific trial. However it is possible, helpful, and indeed essential to work towards making the second level choices that best suit particular NMDs so as to speed up the process of choosing trial specific OMs and ensure some uniformity in the use of OMs that can be beneficial in several ways.

In making such choices the following points should be considered:

  1. Some choices are based upon intrinsic OM characteristics with factors such as the validity, reliability and sensitivity of the OM being important. However some choices are based upon extrinsic study requirements. The choice may require a balance between these intrinsic and extrinsic factors.
  2. There is no one best outcome measure in a given category that will be suitable for all studies even if they are for the same disease. The most suitable outcome measure for a given category may vary with:
    a.The precise NMD being studied.
    b.The severity of the disease being studied; what is appropriate for ambulatory Duchenne muscular dystrophy may not be appropriate for non-ambulatory Duchenne muscular dystrophy.
    c.The age range of the study population; what is appropriate for babies with SMA 1 may not be appropriate for adults with SMA 3.
    d.The expected resources available.
  3. In some cases there may be no best outcome measure available and the choice will be:
    a.To compromise and select the best amongst those which are available.
    b.OMs that are not entirely suitable might be worthwhile including as secondary OMs since their use within the well controlled setting of an RCT alongside other OMs may give an opportunity for validating the OM allowing it to be a primary OM in a subsequent study.
    c.Invest some work in modifying or validating an existing OM, or creating a new outcome measure. If this choice is made it is important not to underestimate the work involved in producing an acceptable and validated outcome measure. In some cases the extra work involved might be less, for example in producing validated translation of an existing outcome measure suitable for a multinational study.

Methodology

There is no agreed methodology for choosing OMs and no single method that is applicable for all types of OM. The suggested methodology that follows is particularly applicable to OM scales and would need modification for other types of OM.

These steps involved include:

  1. Agree the team
  2. Agree the task
  3. Agree the protocol
The Protocol

The protocol should:

  1. define the disease/condition/circumstances for which the OMs are being used.
  2. define the OMs in terms of its purpose and its category.
  3. Establish the extent to which the potential OMs must fulfill agreed quality standards (intrinsic OM characteristics). These may include Validity, Reliability and/or Responsiveness, such as:
    a. Face validity
    b. Content validity
    c. Construct validity, including:
        1. Convergent validity
        2. Discriminant validity
        3. Divergent validity
    d. Criterion-related validity, including:
        1. Concurrent validity
        2. Predictive validity
    e. Internal consistency
    f. Observer reliability, including:
        1.Inter-observer reliability
        2.Intra-observer reliability
    g. Test-retest reliability
    h. Statistical responsiveness
    i. Heuristic techniques
  4. Define the external criteria that the OMs need to fulfill. These might include:
    a. Age range of the study participants
    b. Languages required; if the OM is a patient reported OM then a validated linguistic translation will be required. Translation of investigator applied OMs (eg manuals, instruction etc) may be less of a problem
    c. Resources required
        1. Self filled v investigator filled
        2. Equipment required, available
        3. Implications for training and rater reliability testing
        4. OM costs if these are payable
    d. Reliability and sensitivity data c/w achievable recruitment targets and duration of trial
  5. 5.Construct a table grid giving the OMs in column 1 with the agreed intrinsic and extrinsic properties required for each OMs in subsequent columns.
  6. Agree the search strategy for finding outcome measures. These may include;
    a. The database that will be used for any search which could include;
        1. Medline
        2. EMBASE
        3. Psychlit
        4. Cochrane Database of RCTs
        5. Cochrane Database of Systematic Reviews of RCTs
        6. TREAT-NMD Registry of Outcome Measures (ROM) www.researchROM.com
        7. Controlled Clinical Trials database
        8. others
    b.Agree a date range for the search
    c.Agree search terms for the search; these may be related to the disease, OM, or intervention. You may need to expertise of a librarian to agree a suitable search strategy. Such search strategy may include MESH terms and filters that an expert librarian can utilise to optimise the search.
    d.Additional papers may be obtained by looking at the bibliography of each of the papers selected by the database search, as well as discussion with the authors of outcome measures and other study investigators.
  7. The number of potentially suitable papers should be recorded. Each of the papers should be reviewed by two or more investigators, excluding those papers which do not provide information on the outcome measure leaving a final shortlist.
  8. The number and the reasons for exclusion of any papers should be recorded.
  9. Written copies of the actual outcome measures should be obtained together with all information on the individual criteria that have been previously set as is being required for the outcome measure.
  10. The basic information on each OM should be recorded.
  11. With this information it should be possible to complete the agreed table of required criteria. Based on this completed table an informed and agreed choice of OM can then be made.

The Role of the TREAT-NMD Registry of Outcome Measures (ROM)

The process of choosing OMs can seem a daunting one. The TREAT-NMD Registry of Outcome Measures (ROM) has been conceived as a way to reduce this workload. We aim to embody many of the above principles such as avoiding duplication of effort, supporting team-working, promoting access, and enabling wide participation in the choosing of OMs.

ROM has a number of helpful features currently available and we have plans to expand the tools. We are keen to engage the NMD clinician and research communities to use the existing resources of ROM and contribute to the expansion of these resources. ROM is not going to do the work for us but rather it is going to enable us to do the work for ourselves.

The resources of ROM are available on line thus facilitating team-working; www.researchrom.com

Currently ROM has the following resources:

  1. a registry of OMs giving essential details for each OM with information on where to access the OM and its relevant literature. To date there are over 100 OMs recorded with differing level of detail.
  2. a review and selection tree for specific neuromuscular diseases, listing OMs under consideration for that particular disease by category.
  3. a comparison report that allows one to compare up to 4 OMs of interest by chosen criteria, in tabular format.

Future planned enhancements to ROM include:

  1. the incorporation of quality assessment data for each OM (i.e. details of validation, reliability and sensitivity)

The expectation is that as teams work to obtain the required information on OMs under their consideration their data will be added to ROM by them. In this way the information will be made available to other members of their team as well as to other teams who may also be considering the same or similar OMs for their NMD or study.

Other Information

BMJ: Routinely administered questionnaires for depression and anxiety: systematic review

BJPsych: Outcomes research in mental health: Systematic review

CMAJ: Screening and case-finding instruments for depression: a meta-analysis