Lecture
The problem of the systematic and regular generalization of new results and their connection with the already accumulated base of knowledge.
The classical approach to such systematization is the narrative (discursive) literature review. However, this approach has a number of limitations: the absence of a formalized approach to comparing results; a high level of subjectivity in the review, leading to systematic distortions.
An important step toward overcoming these shortcomings was the emergence of the method of data meta-analysis.
Meta-analytic studies (MAS) are concerned with assessing the non-randomness of relationships between variables and with comparing competing explanations of the dependencies being established.
By means of meta-analysis, all the accumulated data are weighted (taking into account their reliability, statistical decisions, etc.) and an index of the «aggregate» fact is constructed.
History.
In fact, the first meta-analytic study was carried out in 1904 by Pearson; it was devoted to studying the effectiveness of smallpox vaccination. Meta-analysis gained popularity toward the end of the 20th century.
The term was first used in 1976 by G. Glass.
1077 – Glass and Smith – published the results of 375 studies investigating the effectiveness of various types of psychotherapy. (On average, a client feels better than 75% of those who did not undergo psychotherapy.)
In the most general terms, meta-analysis can be defined as a method for assimilating, generalizing, and integrating the results of many studies. Meta-analysis is often discussed as a quantitative statistical method, but it is more adequately regarded as a method that involves the systematic study of the literature, careful formulation of hypotheses, development of criteria for including/excluding studies from the meta-analysis, restructuring and statistical synthesis of results and the corresponding effect sizes, the search for mediating variables, and the drawing of conclusions.
Meta-analysis establishes:
1) measures of the central tendency of the distribution of research results obtained across many works;
2) the variability of these results;
the possibility of explaining and predicting the variability of the empirical facts being established.
Examples: the actor-observer effect, established by Jones and Nisbett in 1971 (the actor has an external locus of control, the observer an internal one). In 2006, Malle's meta-analysis based on 173 studies: the actor-observer asymmetry with respect to situational and dispositional attributions is a greatly exaggerated phenomenon.
Sometimes meta-analysis leads to indirect but significant conclusions. Who is more easily influenced – men or women? Women. But the predominant part of researchers are men.
Thus, meta-analysis is a critical mode of psychological thinking that makes it possible to move toward more generalized generalizations in psychology and to overcome the centrism of one's own scientific position.
Stages of meta-analysis:
1. Selecting the research area, as well as the IV and DV whose connection is to be analyzed. Here the research hypothesis is formulated, the form of operationalization of the variables is chosen, and criteria are developed on the basis of which particular studies are included in the meta-analysis database.
2. Systematic search for studies and collection of the relevant information.
Searching in scientific journals, articles, books, dissertations, etc.
3. Coding the obtained data and the individual characteristics of the studies found. Choosing the effect size, converting effect sizes to a common metric, and evaluating the varying individual parameters. There are 3 types of such parameters: substantive (the type of experimental manipulation), methodological (the study design), and external (year of publication, etc.).
4. Statistical analysis (synthesis) of the effects obtained in the studies.
Computing measures of central tendency and the significance level of the obtained indices of central tendency.
5. Assessing the variability of the effect sizes.
The heterogeneity of effect sizes is assessed using the following criteria: chi-square and the homogeneity test. The standard deviation of the effect size makes it possible to estimate the probability that there are mediating variables that may give rise to this heterogeneity.
6. Discussion and evaluation of the effect size, interpretation of the results, and conclusions.
Writing the meta-analytic report.
There are two fundamental approaches to meta-analysis: the fixed-effects model and the random-effects model.
The former assumes that the unknown population effect is constant across all studies included in the meta-analysis; the latter model assumes that the obtained results vary, having been randomly sampled from a «superpopulation».
For the fixed-effects model, the standard error associated with the effect size includes only the intra-study variability, whereas the random-effects model also includes a component of variability that is a function of the differences between studies.
The choice of model is connected with the choice regarding the possible generalizations.
Using the random-effects model is preferable. It makes it possible to generalize conclusions beyond specific studies and specific samples.
In meta-analysis one should adhere to three principles: precision, simplicity, and clarity.
Two main methodological approaches to meta-analysis (rationales and instructions for carrying out special computations):
- the Hedges-Olkin and Rosenthal-Rubin approach
- the Hunter and Schmidt approach.
Advantages of meta-analysis:
- it makes it possible to take into account small and non-significant results when testing hypotheses about connections;
- it makes it possible to assess and predict inter-individual variability in research results.
Disadvantages:
- errors and biases in the results because of the «file-drawer» problem (published studies usually contain significant and positive results rather than non-significant ones);
- the difficulty of using certain data, e.g. regression coefficients obtained in a multiple regression analysis, when the coefficient obtained for one of the variables depends on the inclusion of other variables in the model;
- the problem of mixing «apples with oranges» - the difficulty of interpreting results obtained by combining information from studies that differ greatly from one another in the operationalization of the variables.
Comments