this post was submitted on 07 Jul 2024
930 points (99.4% liked)
Science Memes
9984 readers
2052 users here now
Welcome to c/science_memes @ Mander.xyz!
A place for majestic STEMLORD peacocking, as well as memes about the realities of working in a lab.
Rules
- Don't throw mud. Behave like an intellectual and remember the human.
- Keep it rooted (on topic).
- No spam.
- Infographics welcome, get schooled.
Sister Communities
Science and Research
Biology and Life Sciences
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- !reptiles and [email protected]
Physical Sciences
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
Humanities and Social Sciences
Practical and Applied Sciences
- !exercise-and [email protected]
- [email protected]
- !self [email protected]
- [email protected]
- [email protected]
- [email protected]
Memes
Miscellaneous
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
See, replication isn't a problem if your entire field is vibes-based. A lot of economics papers I come across are like that (so much so that I am close to writing off the entire decipline as unscientific). The diff in the level of rigour you would see in e.g. particle physics versus in economics is baffling.
It used to be psychology as well but I am noticing they are more than aware of their replication crisis lately. Whereas economics feels pseudoscience with a maths clothing.
I see the same issues also in computer science especially when looking into recent trends such as AI or blockchain/NFTs before that. There are definitely areas that are more rigorous than others but the replication crisis is a problem in many many scientific fields. If your results are not completely outlandish and don't go against the vibe, no one will ever bother to check your results.
There are so many different areas of computer science though... Everything from pure mathematics (e.g 'we found a new algorithm that does X in O(logx)') to the absurdly specific ('when I run the load tests with this configuration it's faster'). The former would get published. The latter wouldn't. And the stuff in the middle ranges the gamut from 'here's my new GC algorithm that performs better in benchmarks on these sample sets' to 'looks like programmers have fewer bugs when you constrain them with these invariants'. All the way over on the other side, NFT/Blockchain/AI announcement crap usually doesn't even have a scientific statement to be expressed, so there's nothing to confirm or deny. There are issues with some areas, but I'm not sure that replication is really the big one for most of these. Only one it commonly applies to IMO are productivity or bug-frequency claims which are generally hella suss
A field that definitely has a problem with replication is Computer Human Interaction. There are a lot of user studies in that field and you basically never see a study done twice. The setup of the studies usually doesn't even allow it to be repeated as it hinges on some proprietary software written for that very study that is not released to the public.
Yeah that's a very good point. I was kinda thinking of HCI at the end there but I'm a software engineer so I was only talking about dev experience 😅. Definitely the same ballpark though and 100% agree with you