The robots.txt construct is completely voluntary and some bots use it to specially target content.
In my opinion, anyone relying on this to protect their content has no business publishing anything online.
this post was submitted on 29 May 2024
39 points (100.0% liked)
technology
23115 readers
258 users here now
On the road to fully automated luxury gay space communism.
Spreading Linux propaganda since 2020
- Ways to run Microsoft/Adobe and more on Linux
- The Ultimate FOSS Guide For Android
- Great libre software on Windows
- Hey you, the lib still using Chrome. Read this post!
Rules:
- 1. Obviously abide by the sitewide code of conduct. Bigotry will be met with an immediate ban
- 2. This community is about technology. Offtopic is permitted as long as it is kept in the comment sections
- 3. Although this is not /c/libre, FOSS related posting is tolerated, and even welcome in the case of effort posts
- 4. We believe technology should be liberating. As such, avoid promoting proprietary and/or bourgeois technology
- 5. Explanatory posts to correct the potential mistakes a comrade made in a post of their own are allowed, as long as they remain respectful
- 6. No crypto (Bitcoin, NFT, etc.) speculation, unless it is purely informative and not too cringe
- 7. Absolutely no tech bro shit. If you have a good opinion of Silicon Valley billionaires please manifest yourself so we can ban you.
founded 4 years ago
MODERATORS
We will sue their unauthorized use in the marketplace of ideas.
Of course it's voluntary, but if entities like OpenAI say they will respect it then presumably they really will.
Couple of things:
- Do you believe anything coming out of OpenAI when it's abundantly clear that they'll say anything to protect their bottom line.
- OpenAI are not the only people harvesting data and selling it to interested parties.
- There is no legal requirement to adhere to the standard and I'd be shocked if any court in the USA could understand the issue, let alone enforce a voluntary standard.
- The amount of automated data collection online is staggering. On my own services it accounts for 50% of the hits. Good luck with policing that.
I agree with your points 2-4 but I have observed on my own website that the crawlers who don't respect won't, and the crawlers who do respect will.
How did you find this information? I know how to check traffic for my website, but idk how to get from "list of IPs" to "these ones are crawlers"
apologies if this is a silly question
I used to sit and monitor my server access logs. You can tell by the access patterns. Many of the well-behaved bots announce themselves in their user agents, so you can see when they're on. I could see them crawl the main body of my website, but not go to a subdomain, which is clearly linked from the homepage but is disallowed from my robots.txt.
On the other hand, spammy bots that are trying to attack you will often instead have access patterns that try to probe your website for common configurations for common CMSes like WordPress. They don't tend to crawl.
Google also provides a tool to test robots.txt, for example.
Perhaps this will help your understanding of my first point.
https://gizmodo.com/former-openai-board-member-sam-altman-chatgpt-1851506252
presumably they really will
but why presume that the guys trying to scam people by claiming their algorithms are aRtIfIcIaL iNtElLiGeNcE aren't lying about that?
but if entities like OpenAI say they will respect it then presumably they really will.
Eh, will they really? It'd be pretty hard to prove they didn't respect it.
Can it work as a way to manifest unconsenting juridically?
It's not about relying on it, it's about changing the behaviour of web crawlers that respect 'em, which, as someone who has adminned a couple scarily popular sites over the years, is a surprisingly high percentage of them.
If someone wants to get around it, they obviously can, but this is true of basically all protective measures ever. Doesn't make them pointless.
I want the AI to internalize marxist shitposting though
I too am a fan of privacy, but in the other hand… I kind of like the idea that we’re training AIs to convince people to become communists and/or change their gender
Such a measure merely punishes entities that respect the rules. If the content can be accessed, it will be scraped and used to train AI.
Lets be honest, AI should be incorporating hexbear. If anything we should have a hundred site mirrors with a free-for-all robots.txt.
After all, this site's only praxis is posting, why not use it to fill the AI with 
I find this to be a very compelling argument, actually.
No, robots.txt doesn’t solve this problem. Scrapers just ignore it. The idea behind robots.txt was to be nice to the poor google web crawlers and direct them away from useless stuff that it was a waste to index.
They could still be fastidious and follow every link, they’d just be ignoring the “nothing to see here” signs.
You beat scrapers with recursive loops of links that start from 4pt black on black divs whose page content isn’t easily told apart from useful human created content.
Traps and poison, not asking nicely.