HN Discussion

cross-posted from: https://beehaw.org/post/15160546 | ghost archive | Excerpts:

... findings with null or negative results — those that fail to find a relationship between variables or groups, or that go against the preconceived hypothesis — gather dust in favour of studies with positive or significant findings. A 2022 survey of scientists in France, for instance, found that 75% were willing to publish null results they had produced, but only 12.5% were able to do so2. Over time, this bias in publications distorts the scientific record, and a focus on significant results can encourage researchers to selectively report their data or exaggerate the statistical importance of their findings. It also wastes time and money, because researchers might duplicate studies that had already been conducted but not published. Some evidence suggests that the problem is getting worse, with fewer negative results seeing the light of day3 over time.

At the crux of both academic misconduct and publication bias is the same ‘publish or perish’ culture, perpetuated by academic institutions, research funders, scholarly journals and scientists themselves, that rewards researchers when they publish findings in prestigious venues, Scheel says.

But these academic gatekeepers have biases, say some critics, who argue that funders and top-tier journals often crave novelty and attention-grabbing findings. Journal editors worry that pages full of null results will attract fewer readers, says Simine Vazire, a psychologist at the University of Melbourne in Australia and editor of the journal Psychological Science.

One of the most significant changes to come out of the replication crisis is the expansion of preregistration (see ‘Registrations on the rise’), in which researchers must state their hypothesis and the outcomes they intend to measure in a public database at the outset of their study (this is already the norm in clinical trials). ... Preliminary data look promising: when Scheel and her colleagues compared the results of 71 registered reports with a random sample of 152 standard psychology manuscripts, they found that 44% of the registered reports had positive results, compared with 96% of the standard publications^7^ (see ‘Intent to publish’). And Nosek and his colleagues found that reviewers scored psychology and neuroscience registered reports higher on metrics of research rigour and quality compared with papers published under the standard model^8^.

ghost archive | Excerpts:

... findings with null or negative results — those that fail to find a relationship between variables or groups, or that go against the preconceived hypothesis — gather dust in favour of studies with positive or significant findings. A 2022 survey of scientists in France, for instance, found that 75% were willing to publish null results they had produced, but only 12.5% were able to do so2. Over time, this bias in publications distorts the scientific record, and a focus on significant results can encourage researchers to selectively report their data or exaggerate the statistical importance of their findings. It also wastes time and money, because researchers might duplicate studies that had already been conducted but not published. Some evidence suggests that the problem is getting worse, with fewer negative results seeing the light of day3 over time.

At the crux of both academic misconduct and publication bias is the same ‘publish or perish’ culture, perpetuated by academic institutions, research funders, scholarly journals and scientists themselves, that rewards researchers when they publish findings in prestigious venues, Scheel says.

But these academic gatekeepers have biases, say some critics, who argue that funders and top-tier journals often crave novelty and attention-grabbing findings. Journal editors worry that pages full of null results will attract fewer readers, says Simine Vazire, a psychologist at the University of Melbourne in Australia and editor of the journal Psychological Science.

One of the most significant changes to come out of the replication crisis is the expansion of preregistration (see ‘Registrations on the rise’), in which researchers must state their hypothesis and the outcomes they intend to measure in a public database at the outset of their study (this is already the norm in clinical trials). ... Preliminary data look promising: when Scheel and her colleagues compared the results of 71 registered reports with a random sample of 152 standard psychology manuscripts, they found that 44% of the registered reports had positive results, compared with 96% of the standard publications^7^ (see ‘Intent to publish’). And Nosek and his colleagues found that reviewers scored psychology and neuroscience registered reports higher on metrics of research rigour and quality compared with papers published under the standard model^8^.