this post was submitted on 17 Sep 2023
150 points (100.0% liked)
Chat
7498 readers
3 users here now
Relaxed section for discussion and debate that doesn't fit anywhere else. Whether it's advice, how your week is going, a link that's at the back of your mind, or something like that, it can likely go here.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I’m not sure that’s possible with images being allowed. If Google, Facebook, Instagram, and YouTube all struggle with it I think it will be an issue anywhere images are allowed. Maybe there’s an opening for an AI to handle the task these days but any dataset for something like that could obviously be incredibly problematic
Yeah, the key problem here is that any open forum, of any considerable popularity, since the dawn of the Internet has had to deal with shit like CSAM. You don't see it elsewhere because of moderators. Doing the very job Op does. It's just now, Op, you're in the position. Some people can, and have decided to, deal with moderating the horrors. It may very well not be something you, Op, can do.
The thing is though, with traditional forums you get a LOT of controls for filtering out the kind of users who post such content. For instance, most forums won't even let you post until you complete an interactive tutorial first (reading the rules and replying to a bot indicating you've understood them etc).
And then, you can have various levels of restrictions, eg, someone with less than 100 posts, or an account less than a month old may not be able to post any links or images etc. Also, you can have a trust system on some forums, where a mod can mark your account as trusted or verified, granting you further rights. You can even make it so that a manual moderator approval is required, before image posting rights are granted. In this instance, a mod would review your posting history and ensure that your posts genuinely contributed to the community and you're unlikely to be a troll/karma farmer account etc.
So, short of accounts getting compromised/hacked, it's very difficult to have this sort of stuff happen on a traditional forum.
I used to be a mod on a couple of popular forums back in the day, and I even ran my own community for a few years (using Invision Power Board), and never once have I had to deal with such content.
The fact is Lemmy is woefully inadequate in it's current state to deal with such content, and there are definitely better options out there. My heart goes out to @Chris and the staff for having to deal with this stuff, and I really hope that this drives the Beehaw team to move away from Lemmy ASAP.
In the meantime, I reckon some drastic actions would need to be taken, such as disabling new user registrations and stopping all federation completely, until the new community is ready.
So this just got posted on lemmy.dbzer0. They've got an AI-based CSAM screen up and running with promising initial results. The model was trained using CLIP, which as far as I understand it means they used written descriptions of what CSAM is or is not.
Could something like this work for Beehaw?
I’m sure the mods saw that, and it’s really more of a question for them tbh, but if it works for other Lemmy instances I’m not sure why it wouldn’t work here.
Wonder whether in theory one could use a dataset of... everything else, have the AI exclude what it does not recognise, then run the exclusions against a dataset to see whether or not they contain children. There could be an additional layer of running the exclusions against a dataset of regular sexual content.
One issue is that admin of any site would still want to report any CSAM to authorities. That could be automated by an AI checker, but one would have to have a lot of faith that the AI was decently accurate and not generating many false reports. The workaround I described to avoid using datasets of abuse is unlikely to be particularly accurate - ok for the purposes of protecting admin, but leaves them in an odd spot when it comes to banning a user, especially where a user's livelihood could be impacted, or things like paid online courses. I guess specialist police departments probably would have to use highly relevant datasets, along with review by humans, but still - nobody wants to inadvertently clog up that system with false reports.