130501P - META-ANALYSIS

Presentation at a Training Program on Biostatistics for physician managers working in Public Health Administration, Qassim Province on May 1, 2013 by Professor Omar Hasan Kasule Sr MB ChB (MUK), MPH (Harvard), DrPH (Harvard) EM: omarkasule@yahoo.com

1.0 INTRODUCTION TO META ANALYSIS

1.1 DEFINITION

Meta analysis refers to quantitative statistical methods used to combine data from several independent studies in order to produce a quantitative summary statistic. It is a form of integrative systematic review of several research papers to derive a general conclusion on which policy or public health decisions can be based. It is used for studies that are similar. It is meaningless for studies that have much diversity.

1.2 HISTORICAL BACKGROUND

Review articles written by discipline leaders were the most popular method of combining findings from various studies. It was realized that they gave poor and unsatisfactory results. This is because of three serious draw-backs: (a) they were subjective and therefore prone to error and bias and (b) the reviewer was free to make decisions on what data and conclusions to emphasize. (c) Secondly reviews dealt with summarization of conclusions without looking at the data on which the conclusions were based.

A systematic meta-analysis review leaves little room for the author to impose his views or biases because of its rigorous methodology that derives final conclusions only from the data available.

Meta-analyis was first developed for randomized clinical trials. Its use in observational studies is more problematic mainly because of the difficulty of knowing and adjusting for confounders. Randomized clinical trials by their very nature have confounders balanced by the randomization process and have a lot of high quality information that can be used to adjust for any residual confounders.

1.3 ADVANTAGES OF META ANALYSIS

Meta-analysis has become popular with the proliferation of epidemiological studies on particular subjects. Writers of review articles and practicing epidemiologists would like to have some form of consensus or summary of the findings of various studies. Meta analysis enables computation of an effect estimate for a larger number of study subjects thus enabling picking up statistical significance that would be missed if analysis was based on small individual studies. Many clinical trials especially with invasive intervention cannot recruit enough patients in one center to reach statistical significance. Meta analysis provides more power than would be found in the individual studies.

Meta analysis can provide results that are more precise than the individual primary studies. Meta analysis can also help reach consensus in situations in which the primary studies have contradictory results. Meta analysis also enables study of variation across several population subgroups since it involves several individual studies carried out in various countries and populations. Meta analysis makes the process of reviewing several studies with view to reaching a general conclusion very transparent because it is based on quantitative assessments.

1.4 DIFFICULTIES OF META ANALYSIS

Meta analysis is methodologically complex because of different study designs, study analyses, and even different data quality. The major problems are: over-conclusion and bias (publication bias and selection bias) and use of wrong methods. Over-conclusion arises when a conclusion is artifactual and is not supported by the aggregated data. The results of meta-analysis based on published sources may not reflect the true situation because of existence of publication bias. Positive findings are more likely to be published than negative ones. Studies carried out in academic or government institutions are thought to be more credible and are therefore more likely to be published whereas studies by pharmaceutical firms have lower publication rates. Inadequate search for reports may lead to bias just as multiple publications of the same study data leads to bias. Bias, conscious or unconscious, may occur in the selection of studies for analysis. In some cases the methods used for meta-analysis are wrong or inappropriate

2.0 SELECTING STUDIES FOR META-ANALYSIS

2.1 PROBLEM IDENTIFICATION AND DEFINITION

Meta-analysis starts with identifying a specific health or clinical problem that requires a solution. It cannot be undertaken as a fishing trip in the hope of finding something worthwhile. The problem should be of practical importance that several researchers have undertaken research on it. Stating the problem clearly and unambiguously helps focus the meta-analytic review process. The framing of the question should include the type of process studied, the study population, the outcome(s), and the study design. The framing of the question is so important because it provides guidelines for the rest of the review process for example the search for research reports, data collection, and the analytic methods depend on the type of question.  It is preferable that the question be narrowly focused. It is more efficient to carry out several focused valid analyses that to carry out one large and broad analysis. The problem can be refined or even modified as the search proceeds when the reviewer becomes aware of issues or information that was not available initially.

2.2 SEARCHING FOR APPROPRIATE STUDIES

A meta-analysis protocol has to be written in advance to avoid having to make adhoc decisions that may end up biasing the study. The protocol should set out in detail criteria for selecting studies to be analyzed and the statistical methods. Details about the studies should be provided to guide the selection of what research to include in the meta-analysis. These details should cover the study design, the source population, the type of data collected, the methodology of analysis, and the outcome measures used.  These details help identify a fairly homogenous list of studies that will enable a successful meta-analysis exercise.

The search methods for research reports must be thorough and unbiased. These must be defined explicitly in the protocol. The search should be wide ranging because some research reports are found in some data bases and not others. Many research reports are not yet published or will never be published.

The search for research reports goes through various stages. A search strategy / methodology should be defined. Then the search should be carried out systematically according to the protocol. Some research reports that are retrieved are not appropriate. It is therefore necessary to revisit the list of retrieved papers using pre-fixed inclusion criteria. The process of selecting what studies to use may involve writing to authors for additional information.  In some cases authors of unpublished papers have to be identified and to obtain their research results.

Electronic data bases are generally used as sources of research reports. MEDLINE (Index Medicus accessed for free as pubmed) is the most popular data base but misses many important research reports. Other data bases that can be used are: EMBASE (Excerpta Medica), Scisearch, CENTRAL, the Chinese Biomedical Literature Data Base, the Latin American Caribbean Health Sciences Literature, the Japan Information Center of Science and Technology File on Science, Technology and Medicine (JIFCS-E), The Cumulative Index of Nursing and Allied Health, and the Cochrane Collaboration.

Hand searching is manual searching of a whole journal for relevant material. It may pick up relevant studies not indexed by the electronic data bases. Hand searching is also useful for conference proceedings that are not usually included in the electronic data bases. Useful research reports can also be identified by going through reference lists of articles. Previous reviews of the same problem may yield research reports otherwise unobtainable.

Unpublished studies can provide useful information. Usually negative studies do not get published. There is no formal way of identifying unpublished studies. Attempts can also be made to identify on-going studies.

In selecting key words and formulating search and article retrieval queries we have to balance two opposing requirements: comprehensiveness and precision. A comprehensive search will pick all what is needed (high sensitivity) but it will also pick up much that is not relevant (low specificity). A more precise search will miss some relevant material (low sensitivity) but will not pick up much that is irrelevant (high specificity). More weight is given to sensitivity than to specificity.

The start of the search should be developing key words to use. The key words can be combined using Boolean logic for more precise searches. The search should be fully documented: title of data base, date and time of search, period of years searched, the key words, and queries used.

Selection of articles starts with looking at the title and the abstract to enable preliminary elimination of what is clearly irrelevant. Reading the full text may reveal that the article is not relevant even it seemed relevant from the title and the abstract.

2.3 ASSESSING THE QUALITY OF THE STUDIES

Not all papers that fulfill the inclusion criteria fulfill the quality requirements of a particular meta-analytic exercise. The quality must therefore be reviewed systematically using pre-fixed quality criteria and papers found wanting should be eliminated from the final review.

Each research report should be assessed for validity (i.e. absence of systematic confounding, selection, misclassification, detection, and follow up bias). Check lists of validity quality criteria can be used to make objective and systematic assessment of validity. Some of these scales have a total score with a cut-off score for describing studies as valid. Where check lists are used it is advised to run a pilot test.

2.4 DATA ABSTRACTION:

A data collection form is used to abstract data from the selected studies. The data form can be paper or electronic. Paper forms enable rechecking of data entered for data entry errors. Electronic forms remove the need for data entry but their errors are difficult to detect later. The name of the person abstracting the information should be recorded. Each study abstracted must be given a specific identifying number. The data collection form must be designed to allow writing notes and updates. There must also have an indication whether the study has selection eligibility and its quality was checked. Data is entered on methods of the study, description of the participants, the interventions or treatments, and outcome measures. Coding instructions should be comprehensive and clear. To avoid mistakes the form should be piloted before use.

If abstraction of the data is carried out by more than one person a system must be developed for resolving differences and reaching a consensus. The abstractors may be blinded to several aspects of the study to prevent bias. In some cases it may be necessary to contact the investigators to obtain data that is not available in the published report.

3.0 DATA ANALYSIS

Meta analysis proceeds in several stages. The first stage described is to determine for each study the effect estimate with its standard error. Then the summary statistic is computed as the weighted sum of the effect measures thus weighted average = ∑ E_i W_i / ∑ W_i  where E_i = Effect estimate and W_i  = weight given to the study. The Mantel-Haenszel method uses its own weights derived from knowledge of cell numbers that are often not available in published papers. In the inverse variance methods, the weight given to each study is the inverse of the variance of its effect estimate in other words W_i = 1/ S_i.. 

The analysis may assume a random effect or a fixed effect. A random effect analysis assumes that the different studies are estimating different effects that are distributed all over the studies. The fixed effect analysis assumes that the different studies are estimating the same effect.

The standard error of the pooled effect estimate can be computed using specific formulas and can be used to get a confidence interval around the effect estimate. Meta analysis also incorporates a test of heterogeneity across studies. If there is heterogeneity there is no need for a combined effect measure.

The next step is to assess heterogeneity. A simple way is to see whether confidence intervals of effect measures for the various studies overlap or not. Overlap indicates homogeneity and no overlap indicates heterogeneity. A chi square test can in addition be used to test for heterogeneity. If heterogeneity is found the data should be rechecked for accurate abstracting. Fixed effect analysis ignores heterogeneity while random effects analysis incorporates heterogeneity. Other measures of dealing with heterogeneity are: changing the effect measure, excluding some studies, carrying out subgroup analyses, or meta-regression (the effect measure is the dependent variable and various study characteristics are the independent variables).