technology

23166 readers

166 users here now

On the road to fully automated luxury gay space communism.

Spreading Linux propaganda since 2020

Rules:

1. Obviously abide by the sitewide code of conduct. Bigotry will be met with an immediate ban
2. This community is about technology. Offtopic is permitted as long as it is kept in the comment sections
3. Although this is not /c/libre, FOSS related posting is tolerated, and even welcome in the case of effort posts
4. We believe technology should be liberating. As such, avoid promoting proprietary and/or bourgeois technology
5. Explanatory posts to correct the potential mistakes a comrade made in a post of their own are allowed, as long as they remain respectful
6. No crypto (Bitcoin, NFT, etc.) speculation, unless it is purely informative and not too cringe
7. Absolutely no tech bro shit. If you have a good opinion of Silicon Valley billionaires please manifest yourself so we can ban you.

founded 4 years ago

MODERATORS

[email protected]

Should Hexbear have a more robust robots.txt? (hexbear.net)

submitted 3 months ago by [email protected] to c/[email protected]

18 comments fedilink hide all child comments

Consider https://arstechnica.com/robots.txt or https://www.nytimes.com/robots.txt and how they block all the stupid AI models from being able to scrape for free.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 7 points 3 months ago (2 children)

I agree with your points 2-4 but I have observed on my own website that the crawlers who don't respect won't, and the crawlers who do respect will.

[–] [email protected] 6 points 3 months ago (1 children)

How did you find this information? I know how to check traffic for my website, but idk how to get from "list of IPs" to "these ones are crawlers"

apologies if this is a silly question

[–] [email protected] 2 points 3 months ago* (last edited 3 months ago)

I used to sit and monitor my server access logs. You can tell by the access patterns. Many of the well-behaved bots announce themselves in their user agents, so you can see when they're on. I could see them crawl the main body of my website, but not go to a subdomain, which is clearly linked from the homepage but is disallowed from my robots.txt.

On the other hand, spammy bots that are trying to attack you will often instead have access patterns that try to probe your website for common configurations for common CMSes like WordPress. They don't tend to crawl.

Google also provides a tool to test robots.txt, for example.

[–] [email protected] 1 points 3 months ago

Perhaps this will help your understanding of my first point.

https://gizmodo.com/former-openai-board-member-sam-altman-chatgpt-1851506252