Lecture
Metascience (also known as meta-research) is the use of scientific methodology to study science itself . Metascience seeks to improve the quality of scientific research and increase its effectiveness. It is also known as «research on research» and «the science of science», since it uses research methods to study how research is conducted and to identify areas where improvements can be made. Metascience deals with all fields of research and is described as « a bird's-eye view of science ». As John Ioannidis put it, «Science is the best thing that has happened to humanity ... but we can do it better».
In 1966, one of the first meta-research studies analyzed the statistical methods of 295 articles published in ten leading medical journals. It found that «in almost 73% of the reports read ... conclusions were drawn when the justification for those conclusions was invalid». Meta-research over subsequent decades revealed numerous methodological flaws, inefficiencies, and poor practices in research across various scientific fields. Many scientific studies could not be reproduced, especially in medicine and the humanities . The term « reproducibility crisis » was coined in the early 2010s as part of growing awareness of this problem.
Measures have been taken to address the problems identified through metascience. These measures include the pre-registration of scientific studies and clinical trials, as well as the establishment of organizations such as CONSORT and the EQUATOR network, which issue guidelines on methodology and reporting. Efforts continue to combat the misuse of statistics, eliminate negative incentives in academic institutions, improve the peer-review process, systematically collect data on the scientific-publishing system, combat bias in the scientific literature, and enhance the overall quality and effectiveness of the scientific process. Thus, metascience is an important part of the methods underlying the open science movement.
In 1966, one of the first meta-research studies analyzed the statistical methods of 295 articles published in ten leading medical journals. It found that «in almost 73% of the reports read... conclusions were drawn when the justification for those conclusions was invalid». A 1976 article called for the funding of meta-research: «Because the very nature of research, especially if it is prospective, requires long periods of time, we recommend the establishment of independent, highly qualified groups with sufficient long-term support to conduct and sustain retrospective and prospective studies of the nature of scientific discovery». In 2005, John Ioannidis published an article titled « Why Most Published Research Findings Are False », which argued that most articles in the medical field contain incorrect conclusions. This article became the most downloaded article in the Public Library of Science and is considered foundational to the field of metascience. In a similar study with Jeremy Howick and Despina Koletsi, Ioannidis showed that only a minority of medical interventions are supported by «high-quality» evidence according to the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach. Later meta-research revealed widespread difficulties in reproducing results across many scientific fields, including psychology and medicine . This problem was named the « replication crisis ». Metascience emerged as a response to the replication crisis and to concerns about waste in research.
Major publishers have allocated resources to meta-research and quality-improvement initiatives. Leading journals, such as Science, The Lancet, and Nature, regularly cover meta-research and reproducibility issues. In 2012, PLOS ONE launched the Reproducibility Initiative. In 2015, BioMed Central introduced a minimum standards of reporting checklist for four publications.
The first international conference in the broad field of meta-research was held in Edinburgh in 2015 and was called Research Waste/EQUATOR; the first international conference on peer review was held in 1989 and was called the Peer Review Congress . In 2016, the journal Research Integrity and Peer Review was launched. The journal's editorial called for «research that will expand our understanding and offer potential solutions to problems related to peer review, the reporting of research findings, and research and publication ethics».
On July 8, 2025, Nature published an editorial announcing the creation of the Metascience Alliance — a coalition of more than 25 foundations, academic groups, companies, and other institutions engaged in metascience: the use of scientific methods to understand and improve science itself. More than 830 people from about 65 countries took part in the London conference. The editorial calls on metascientists not to limit themselves to research within academia, but to address broader social problems, communicate scientific uncertainty effectively, and restore public trust in science.
Metascience can be divided into five main areas of interest: methods, reporting, reproducibility, evaluation, and incentives. These correspond, respectively, to how research is conducted, reported, verified, evaluated, and rewarded.
Metascience seeks to identify unsound research methods, including research bias, poor study design, and misuse of statistics, and to find ways to reduce these practices. Meta-research has revealed numerous biases in the scientific literature, in particular the misuse of p-values and excessive reliance on significance testing .
Meta-research data science is the use of data science to analyze scientific articles. It includes both qualitative and quantitative methods. Research in meta-research data science includes fraud detection and citation-network analysis .
Journalology, also known as the science of scholarly publishing, is the scientific study of all aspects of the academic publishing process . This field seeks to improve the quality of scientific research by introducing evidence-based methods into academic publishing. The term «journalology» was coined by Stephen Lock, a former editor-in-chief of The BMJ. The first Peer Review Congress, held in 1989 in Chicago, Illinois, is considered a turning point in the establishment of journalology as a distinct field. The field of journalology has had a significant impact on advancing the pre-registration of research in science, especially in clinical trials . Registration of clinical trials is now expected in most countries.
Meta-research has revealed shortcomings in the presentation, explanation, dissemination, and popularization of research, especially in the social and medical sciences. Poor-quality reporting makes it difficult to accurately interpret the results of scientific research, to reproduce studies, and to identify bias and conflicts of interest among authors. Solutions include implementing reporting standards and increasing the transparency of scientific research (including stricter requirements for disclosing conflicts of interest). Attempts are being made to standardize the presentation of data and methodology through the creation of guidelines by organizations such as CONSORT and the larger EQUATOR network.
The reproducibility crisis is an ongoing methodological crisis in which it has become apparent that many scientific studies are difficult or impossible to reproduce . Although the crisis has roots in mid-to-late-20th-century meta-research, the term «reproducibility crisis» was coined only in the early 2010s as part of growing awareness of the problem. The reproducibility crisis has been extensively studied in psychology (especially social psychology) and medicine, , including cancer research. Replication is an integral part of the scientific process, and the widespread failures of replication call into question the reliability of the affected fields.
Moreover, replicating a study (or failing to replicate it) is considered less impactful than the original study, and its results are published less often in many fields. This hinders the presentation of research findings and even attempts to replicate them.
Metascience seeks to establish a scientific basis for peer review. Meta-research evaluates peer-review systems, including pre-publication review, post-publication review, and open review . It also seeks to develop better criteria for research funding.
Metascience seeks to improve the quality of research by refining incentive systems. This includes examining the accuracy, effectiveness, costs, and benefits of various approaches to ranking and evaluating research and those who conduct it. Critics argue that perverse incentives have created a «publish or perish» environment in academia that fosters pseudoscience, low-quality research, and false positives . According to Brian Nosek, «the problem we face is that the incentive system is almost entirely focused on publishing research, rather than on conducting research correctly». Reform advocates seek to structure the incentive system so that it favors higher-quality outcomes. For example, by assessing quality based on peer judgment («rather than [solely or primarily] on indices»), institutional evaluation criteria, and by ensuring transparency and professional standards.
Research has proposed machine-readable standards and a (taxonomy of) badges for scholarly publication management systems that focus on author contribution — who contributed what and how much to a research work — instead of using the traditional notion of mere authorship — who participated in some way in producing the publication. One study noted one of the problems associated with the continued disregard of the nuances of contribution — it found that «the number of publications has ceased to be a good indicator as a result of growing author lists, shortened articles, and a sharp rise in the number of publications».
Beyond the merits of the submitted work, reviewers' assessments can be significantly influenced by other factors. However, factors such as the use of data on a researcher's prior publication reliability and their relevance to the public interest can also be important. Nevertheless, evaluation systems, including peer-review systems, may largely lack mechanisms and criteria oriented, or effectively oriented, toward merit, real positive impact, progress, and public usefulness, rather than toward analytic metrics such as citation counts or altmetrics, even if these may serve as partial indicators of those goals. Rethinking the structure of academic reward «to provide more formal recognition of intermediate products, such as data» could have positive consequences and reduce data withholding.
One commentary noted that academic rankings do not take into account where (country and institution) the relevant researchers received their education.
Scientometrics is concerned with measuring bibliographic data in scientific publications. Key research questions include measuring the impact of scientific articles and academic journals, understanding scientific citations, and using such measurements in the context of policy and management. Studies show that «the metrics used to measure academic success, such as the number of publications, citation counts, and impact factor, have not changed for decades» and to some extent «have ceased» to be good indicators, , which leads to problems such as «overproduction, unnecessary fragmentation, oversale, predatory journals (pay-and-publish), sneaky plagiarism, and deliberate misrepresentation of scientific results for the purpose of sale and oversale».
New tools in this field include systems for quantitatively assessing how much a cited node influences a citing node. This can be used to convert unweighted citation networks into weighted ones, and then to assess importance, to obtain «influence metrics for various participating entities, such as publications, authors, etc.», and, among other tools, for search engines and recommendation systems .
Science funding and science management can also be studied and complemented by metascience.
Various measures, such as setting priorities, can be important. For example, the concept of differential technological development implies the deliberate development of technologies – for example, control, safety, and policy technologies, as opposed to risky biotechnologies – at different rates, in order to reduce risks, primarily global catastrophic risks, by influencing the sequence of technological development. Traditional legislative and incentive structures may be insufficient to ensure proper scientific governance, since they often respond too slowly or inadequately to emerging challenges.
Other incentives for governing science and related processes, including through reforms based on metascience, may include ensuring public accountability (for example, in terms of the availability of research, especially publicly funded research, or in terms of seriously considering various research topics of public interest), increasing the skilled and productive scientific workforce, improving the efficiency of science for better problem-solving in general, and helping to ensure that clear societal needs based on reliable scientific data – for example, in the field of human physiology – are adequately identified and met. Such measures, incentives, and intervention schemes can be the subject of metascience.
Scientific awards are one category of incentives for science. Metascience can examine existing and hypothetical systems of scientific awards. For example, it has been found that works honored with Nobel Prizes are concentrated in only a few scientific fields, with only 36 out of 71 having received at least one Nobel Prize. Of 114 out of 849 fields of science that can be divided into the DC2 and DC3 classification systems, five were shown to account for more than half of the Nobel Prizes awarded between 1995 and 2017 (particle physics [14%], cell biology [12.1%], atomic physics [10.9%], neuroscience [10.1%], molecular chemistry [5.3%]).
Research has shown that delegating responsibility to policymakers – a centralized top-down approach — for the production of knowledge and the corresponding funding of science, whereby science is then supposed to somehow deliver «reliable and useful knowledge to society», is overly simplistic.
Measurements show that the distribution of biomedical resources may be more strongly linked to prior distribution and research than to the burden of disease .
Research suggests that «if peer review is to be retained as the main arbitration mechanism in the competitive selection of research reports and funding, the scientific community must ensure that it is not arbitrary».
Studies show that there is a need to «reconsider how we measure success» (see #Factors of success and progress ) .
Funding information from grant databases and funding-acknowledgment sections can serve as a data source for scientometric research, for example for studying or assessing the influence of funding organizations on the development of science and technology.
Science draws its value as a global public good from two attributes: researchers must make their knowledge claims and supporting evidence open to scrutiny, and they must communicate results promptly and effectively. Metascientific research examines topics in science communication, such as the coverage of science in the media , science journalism , and the online communication of results by science educators and scientists. Studies show that scientists use social media primarily to amplify their own work. However, institutions should cultivate a culture that emphasizes genuine engagement with research rather than mere visibility. Science communication may also include conveying the needs, concerns and requests of society to scientists.
Alternative metric tools can be used not only to assist in evaluation (of effectiveness and impact) and accessibility, but also to aggregate the many public discussions of a scientific article on social media, such as Reddit , citations in Wikipedia, and reports on the research in news media, which can then be analyzed within metascience or provided to and used by relevant tools. From the standpoint of evaluation and accessibility, altmetrics assess the effectiveness or impact of publications based on the interactions they receive through social media or other online platforms, which, for example, can be used to rank recent research by measured impact, including before other research has begun to cite it. The specific procedures of existing altmetrics are not transparent , and the algorithms used cannot be configured or modified by the user, as can be done with open-source software. One study describes various limitations of altmetrics and identifies «avenues for further research and development». They are also limited in their use as a primary tool for researchers to obtain constructive feedback. (see above )
It has been suggested that science could benefit if «intellectual exchange — especially regarding the social implications and application of science and technology — were better valued and encouraged in the future».
Primary studies «without context, comparison or generalization ultimately have limited value», and various types of research synthesis and generalization integrate primary studies. Progress on key socio-ecological problems on the global environmental agenda «is hindered by a lack of integration and synthesis of existing scientific data», with a «rapidly growing volume of data», fragmented information, and generally unresolved problems of data synthesis. According to Khalil, researchers face the problem of too many articles — for example, in March 2014 more than 8,000 papers were submitted to arXiv — and in order «to keep up with the enormous volume of literature, researchers use reference-management software, compile summaries and notes, and rely on review articles to give an overview of a specific topic». He notes that review articles are usually devoted (only) to topics on which a great deal has already been written, and that they can quickly become outdated, and proposes «wiki review articles» that are continuously updated with new research on a topic, summarize the findings of many studies, and suggest directions for future research. Research suggests that if a scientific publication is cited in a Wikipedia article, this could potentially be regarded as an indicator of a certain influence of that publication, for example, since over time this could indicate that the reference contributed to a high level of generalization on the given topic.
Science journalists play an important role in the scientific ecosystem and in communicating science to the public, and must «know how to use relevant information when deciding whether to trust the results of a study, and whether and how to report on them», verifying the results that are conveyed to the public.
Some research is devoted to science education , for example the teaching of specific scientific controversies and the historical process of discovering major scientific findings , as well as common scientific misconceptions . Education can also be a more general topic, for example how to improve the quality of scientific output and reduce the time needed for scientific work, or how to expand and retain a diverse scientific workforce.
Many students hold mistaken notions about what science is and how it works. Anti-science views and beliefs are also a subject of research. Hotez suggests that anti-science «has become a dominant and highly deadly force threatening global security», and that there is a need for a «new infrastructure» to mitigate its effects.
Meta-analysis can examine how scientific processes evolve over time. Research has shown that team sizes are growing, «increasing by an average of 17% per decade». (see the workforce advantage below)
It has been established that common forms of non-open-access publication, and the prices charged by many traditional journals – even for publicly funded articles – are unjustified, unnecessary or suboptimal and constitute harmful barriers to scientific progress. Open access could save significant financial resources that could be used otherwise, and level the playing field for researchers in developing countries. There are significant costs for subscriptions, for gaining access to specific studies, and for article processing . «Paywall: The Business of Scholarship» is a documentary film devoted to these issues.
Another topic concerns the established styles of scientific communication (e.g. long text-based studies and reviews) and the practice of scientific publishing — there are concerns about the «glacial pace» of traditional publishing. The use of preprint servers for the early publication of research drafts is growing, and open peer review , new tools for selecting research, and improved matching of submitted manuscripts to reviewers are among the proposals for speeding up publication.
Metadata from scientific publications can be extracted, enriched, and made widely available using digital tools. OpenAlex is a free online index containing more than 200 million scholarly documents, which integrates and provides metadata such as sources, citations , author information , scientific fields, and research topics . Its open-source API and website can be used for metascience, scientometrics, and new tools that query this semantic network of articles . Another project under development, Scholia , uses metadata from scientific publications for various visualization and aggregation functions, such as providing a simple user interface that summarizes the literature on a particular feature of the SARS-CoV-2 virus using Wikidata's «main topic» property .
Beyond metadata explicitly assigned to research by humans, natural language processing and AI can be used to match scientific publications to topics — in one study examining the impact of scientific awards, this was used to match article text (not just keywords) to the linguistic content of Wikipedia's science-topic pages («pages that are created and updated by scientists and users through crowdsourcing »), producing meaningful and plausible classifications of high-precision scientific topics for further analysis or navigation.
Metascientific research examines the overall growth of science, using, for example, data on the number of publications in bibliographic databases . Research has shown that segments with different growth rates appear to be linked to phases of «economic (e.g. industrialization)» events — money being regarded as a necessary resource for the scientific system — «and/or political events (e.g. World War II)». It also confirmed the recent exponential growth in the volume of scientific literature and calculated an average doubling period of 17.3 years.
However, others point out that it is difficult to measure scientific progress in a meaningful way, in part because it is difficult to accurately assess how important a given scientific discovery is. Books and articles describe various perspectives on the trajectories of science's development as a whole (impact, number of major discoveries, etc.), including the view that science is becoming more difficult (per dollar or per hour), that if science «is slowing down today, it is because it is too focused on established fields», that scientific articles and patents are increasingly less «disruptive» in terms of their break from the past, as measured by the «CD index» , and that there is a broader stagnation – possibly as part of a wider trend – when, for example, «almost nothing has changed since the 1970s», if one excludes the computer and the Internet.
Understanding the potential slowdown in scientific productivity could open up opportunities for accelerating research and addressing humanity's most pressing problems. For example, the emphasis on citation counts in measuring scientific productivity, information overload, reliance on a narrower body of existing knowledge (which may include narrow specialization and related contemporary practices) based on three indicators of «use of prior knowledge» , and risk-averse funding structures, may lead «to the gradual development of science and a move away from research projects that are more likely to fail». The study that introduced the «CD index» suggests that the total number of articles has increased, while the total number of «highly disruptive» articles, as measured by the index, has not increased (notably, the 1998 discovery of the accelerating expansion of the universe has a CD index of 0). Their findings also suggest that scientists and inventors «may be struggling to keep pace with the expansion of knowledge».
To address the potential bias associated with a lack of novelty, researchers have proposed novelty metrics that measure whether a study creates new combinations of cited journals, while taking methodological complications into account. Approaches include text analysis and comparative bibliographic methods. Other approaches include actively funding high-risk projects. (see above )
Science maps can display the main interrelated topics within a given scientific field, their changes over time, and key participants (researchers, institutions, journals). They can help identify the factors driving the emergence of new scientific fields and the development of interdisciplinary areas, and may be relevant for science-policy purposes. (see above ) Theories of scientific change can guide «the study and interpretation of visualized intellectual structures and dynamic patterns». Maps can show the intellectual, social or conceptual structure of a field of research. Beyond visual maps, studies based on expert surveys and similar approaches can identify understudied or neglected socially significant areas, topic-level problems (such as stigma or dogma), or potential errors in prioritization. Examples of such research include studies of public-health policy and the social sciences of climate-change mitigation , where it is estimated that only 0.12% of all climate-related research funding is spent on this, despite the fact that the most pressing task at this stage is developing ways to mitigate the effects of climate change, while the natural sciences of climate change are already well developed.
There are also studies that describe a scientific field or topic, such as research into the use of research findings in policy and practice , partly through surveys .
Some studies examine scientific controversies and disputes and can reveal ongoing major debates (e.g. open questions) and disagreements among scientists or studies. One study suggests that the level of disagreement was highest in the social sciences and humanities (0.61%), followed by the biomedical and medical sciences (0.41%), the life and earth sciences (0.29%); the physical sciences and engineering (0.15%), and mathematics and computer science (0.06%). Such studies can also show where disagreements are located, especially if they cluster, including visually, for example by means of cluster diagrams.
Studies devoted to a specific research question or topic are often examined in the form of higher-level reviews, in which the results of various studies are integrated, compared, critically analyzed, and interpreted. Examples of such works are scientific reviews and meta-analyses. These and related methods face a variety of challenges and are themselves a subject of metascience.
Various problems with existing studies, such as heterogeneity of the methods used, can lead to erroneous meta-analytic conclusions.
Various problems require rapid integration of new and existing scientific knowledge. Shared frameworks or «public resources» are especially useful in settings where a large number of loosely connected projects and initiatives exist.
Evidence synthesis can be applied to important and, crucially, relatively urgent and well-defined global problems: «climate change, the energy transition, biodiversity loss, antimicrobial resistance, poverty eradication, and so on». It has been suggested that a more advanced system would allow research summaries to be kept up to date through «living» systematic reviews – for example, in the form of «living» documents. Although the volume of scientific articles and data (or information and online knowledge) has grown substantially, the number of published academic systematic reviews has increased from «about 6,000 in 2011 to more than 45,000 in 2021». An evidence-based approach is important for progress in science, policy, medicine, and other fields. For example, meta-analyses can quantify what is known and identify what remains unknown, as well as place «genuinely innovative and interdisciplinary ideas» in the context of established knowledge, which can amplify their impact. (see above)
It has been suggested that a deeper understanding of the factors underlying successful science could «improve the prospects of science as a whole in more effectively addressing social problems».
Two metascientists reported that «structures that foster breakthrough research and focus attention on new ideas» may be important, since in a growing scientific field citation flows are disproportionately consolidated around already well-cited articles, possibly slowing and hindering canonical progress. One study concluded that in order to amplify the impact of genuinely innovative and interdisciplinary new ideas, they should be placed in the context of established knowledge.
Other researchers have reported that the most successful individuals – in terms of «the likelihood of receiving an award, being elected to the National Academy of Sciences (NAS), or attaining superstar status» – were protégés who trained under mentors who published research for which they were later honored following the protégé's mentorship. Studying original topics rather than the research topics of these mentors also correlated positively with success. Highly productive partnerships are also a subject of research – for example, «super-ties» of frequent co-authorship between two people who may complement each other's skills, which is likely also the result of other factors such as mutual trust, conviction, commitment, and enjoyment.
The emergence or origin of ideas in successful scientists is also a subject of research, for example a review of existing ideas about how Mendel made his discoveries – or, more generally, the process by which scientists make discoveries. Scientific discovery is inherently collaborative, involving the appropriation, modification, and combination of existing ideas rather than isolated innovation. Few studies examine the cognitive processes, information practices, and digital workflows that characterize productive researchers
One study proposed the theory that in many disciplines, the higher scientific productivity or success of elite universities can be explained by the larger number of funded workers available. The study found that university prestige was associated with higher productivity only for faculty working in groups, not for faculty publishing independently or for the group members themselves. This is presented as evidence that the high productivity of elite researchers is not driven by more rigorous talent selection by leading universities, but by workforce advantages gained through greater access to funding and the appeal of prestige to graduate students and postdocs.
Success in science (as shown in dissertation evaluation processes) is often measured using metrics such as citations rather than in terms of possible or potential impact on life and society, as awards sometimes do (see above). Problems with such metrics are described in general terms elsewhere in this article and include the substitution of citations to primary research with citations to reviews. There are also proposals to change academic incentive systems in ways that would increase recognition of social impact in the research process.
The proposed field of «progress studies» could examine how scientists (or funders, or evaluators of scientists) should act, «design interventions», and study progress itself. This field was explicitly proposed in a 2019 essay and described as an applied science that prescribes action.
Research suggests that improving the methods of conducting scientific research could accelerate the pace of scientific discovery and its application, which could be useful for finding urgent solutions to humanity's problems, improving living conditions for humanity, and deepening understanding of nature. Metascience research can seek to identify aspects of science that need improvement and to develop ways of improving them. If science is recognized as a fundamental engine of economic growth and social progress, this may raise «the question of what we – as a society – can do to accelerate science and direct it toward solving society's most important problems». One author clarified that funding science requires diverse approaches rather than a single model. DARPA-style models, curiosity-driven research, mutual-support mechanisms, and other approaches each have their merits and applicability in different contexts. Nevertheless, evaluating them can help accumulate knowledge about what works or works best.
Metascience research identifying shortcomings in scientific practice has spurred reforms in science. These reforms aim to address problems in scientific practice that lead to low-quality or inefficient research.
A 2015 study listed «fragmented» efforts in metascience.
The practice of registering a scientific study before it is conducted is called preregistration. It arose as a means of addressing the reproducibility problem. Preregistration requires submission of a registered report, which is then accepted for publication or rejected by the journal based on its theoretical rationale, experimental design, and proposed statistical analysis. Preregistration of studies serves to prevent publication bias (e.g., non-publication of negative results), reduce data «cherry-picking», and improve reproducibility.
Studies revealing low consistency and quality in reporting demonstrated the need for reporting standards and guidelines in science, leading to the emergence of organizations that develop such standards, for example CONSORT (Consolidated Standards of Reporting Trials) and the EQUATOR Network.
The EQUATOR Network (Enhancing the QUAlity and Transparency Of health Research) is an international initiative that develops and promotes reporting guidelines for health research with the aim of improving the quality and reliability of medical literature. The EQUATOR Network was established to raise awareness of the importance of high-quality research reporting, to help develop, disseminate, and implement reporting guidelines for various study designs, to monitor the state of reporting quality of research papers in the health sciences literature, and to conduct research on issues affecting the quality of health research reporting. The network acts as an «umbrella» organization, bringing together developers of reporting guidelines, medical journal editors and reviewers, research funders, and other key stakeholders with a mutual interest in improving the quality of scientific publications and of research itself.
Metascience is used in creating and improving technical systems (ICT) and standards for evaluating, incentivizing, communicating, commissioning, funding, regulating, producing, managing, using, and publishing science. Such metascience can be called «applied metascience», and it may seek to explore ways of increasing the quantity, quality, and positive impact of research. One example is the development of alternative metrics.
Various websites or tools also identify unsuitable studies and/or allow feedback to be left, for example PubPeer, the Cochrane risk-of-bias assessment tool, and RetractionWatch. Medical and academic disputes are as old as antiquity, and research calls for the study of «constructive versus intrusive criticism» and for the development of policies that would «help turn social media into a lively forum for discussion rather than merely an arena for gladiatorial combat». Feedback on research can be found through altmetrics, which are often integrated into a study's website — most commonly as an embedded Altmetrics badge — but this can often be incomplete, for example showing only social media discussions that directly link to the study but not those that link to news reports about the study. (see above)
Tools may be developed as part of metaresearch, or used and studied within the framework of such research. Notable examples include:
According to one study, «because of reproducibility problems in science, a simple way is needed to check how often studies have been replicated and whether the original results are confirmed». One study proposes a tool for screening studies for early signs of research fraud.
Clinical research in medicine is often of low quality, and many studies cannot be replicated. An estimated 85% of research funding is wasted. In addition, the presence of bias affects the quality of research. The pharmaceutical industry has a substantial influence on the design and conduct of medical research. Conflicts of interest are common among the authors of medical literature and among the editors of medical journals. Although almost all medical journals require their authors to disclose conflicts of interest, editors are not required to do so. Financial conflicts of interest are associated with higher rates of positive study outcomes. In antidepressant trials, pharmaceutical company sponsorship is the strongest predictor of trial outcome.
Blinding is another subject of meta-research, since errors caused by poor blinding are a source of experimental bias. Blinding is not well described in the medical literature, and widespread misunderstanding of the topic has led to poor application of blinding in clinical trials. Moreover, instances of blinding failure are rarely measured or reported. Studies showing blinding failure in antidepressant trials have led some scientists to claim that antidepressants are no better than placebo. In light of meta-research showing blinding failure, CONSORT standards recommend that all clinical trials assess and report on the quality of blinding.
Studies have shown that systematic reviews of existing scientific evidence are used suboptimally when planning new studies or summarizing results. Cumulative meta-analyses of studies evaluating the effectiveness of medical interventions have shown that many clinical trials could have been avoided if a systematic review of existing evidence had been conducted before the new trial. For example, Lau et al. analyzed 33 clinical trials (involving 36,974 patients) evaluating the effectiveness of intravenous streptokinase for acute myocardial infarction. Their cumulative meta-analysis showed that 25 of the 33 trials could have been avoided if a systematic review had been conducted before the new trial. In other words, the randomization of 34,542 patients was potentially unnecessary. One study analyzed 1,523 clinical trials included in 227 meta-analyses and concluded that «fewer than a quarter of relevant prior studies» were cited. They also confirmed earlier findings that most clinical trial reports do not include a systematic review that justifies the study or summarizes prior results.
Many treatments used in modern medicine have proven ineffective or even harmful. John Ioannidis's 2007 study found that it took the medical community an average of ten years to stop referencing popular treatments after their effectiveness had been conclusively disproven.
Metascience has revealed significant problems in psychological research. The field suffers from high bias, low reproducibility, and widespread misuse of statistics. The reproducibility crisis affects psychology more severely than any other field; up to two-thirds of widely publicized findings may prove irreproducible. Meta-research shows that 80–95% of psychological studies confirm their original hypotheses, which strongly suggests the existence of publication bias.
The reproducibility crisis has led to renewed efforts to re-verify important results. In response to concerns about publication bias and p-hacking, more than 140 psychology journals have implemented results-blind review, in which studies are pre-registered and published without regard to their results. An analysis of these reforms showed that 61 percent of studies with results-blind review yield null results, in contrast to 5–20 percent in earlier studies. This analysis shows that results-blind review substantially reduces publication bias.
Psychologists often confuse statistical significance with practical importance, enthusiastically claiming great confidence in inconsequential facts. Some psychologists have responded to this by increasing their use of effect-size statistics rather than relying solely on p-values.
Richard Feynman noted that estimates of physical constants were closer to published values than would be expected by chance. This was thought to be the result of confirmation bias: results that agreed with the existing literature were more likely to be regarded as reliable and, consequently, to be published. Physicists now use the "blind analysis" method to prevent this kind of bias.
Web-measurement research is important for understanding the workings of the modern Internet, especially in the areas of security and privacy. However, such research often requires specially designed or modified crawling systems, which has led to the emergence of many analytical tools for similar tasks. In a paper by Nurullah Demir and colleagues, the authors analyzed 117 recent scholarly papers to identify best practices for web-measurement research and to establish criteria for reproducibility and repeatability. They found that experimental setups and other information important for reproducing and repeating results are often missing. In a large-scale web-measurement study of 4.5 million pages using 24 different measurement setups, the authors demonstrated the impact of minor differences in experimental setups on overall results, underscoring the need for precise and complete documentation.
There are several organizations and universities in the world engaged in meta-research, including the Berlin Institute for Meta-Research Innovation, the Stanford Meta-Research Innovation Center, the Meta-Research Center at Tilburg University, the Meta-Research and Evidence Synthesis Unit, the George Institute for Global Health in India, and the Center for Open Science. Organizations developing tools for metascience include OurResearch, the Center for Scientific Integrity, and companies working in altmetrics. An annual metascience conference is organized by the Association for Interdisciplinary Meta-research and Open Science (AIMOS), and a biennial conference is organized by the Center for Open Science.
Comments