this post was submitted on 22 Aug 2024
4 points (100.0% liked)

Pulse of Truth

400 readers
59 users here now

Cyber Security news and links to cyber security stories that could make you go hmmm. The content is exactly as it is consumed through RSS feeds and wont be edited (except for the occasional encoding errors).

This community is automagically fed by an instance of Dittybopper.

founded 10 months ago
MODERATORS
 

For a whille now, I have seen scans that contain the pattern "%%target%%" in the URL. For example, today this particular URL is popular:

top 4 comments
sorted by: hot top controversial new old
[–] [email protected] 1 points 2 weeks ago (1 children)

I have a robots.txt which does nothing but exclude "GPTBot"

I was coincidentally looking at my logs for unrelated reasons and caught it reading my robots.txt... and then promptly ignored it and scraped my whole site. Like yeah okay cool man

[–] [email protected] 1 points 2 weeks ago (2 children)

How does Reddit block users? Just by headers and IP addresses? Could the same be done once the GPTbot headers are known?

[–] [email protected] 4 points 2 weeks ago* (last edited 2 weeks ago)

Every bit of information being sent to your web server can be spoofed. There is nothing you can do about this unless you're willing to exclude an increasing percentage of real users.

My webserver is constantly barraged by crawlers and bots because I have zero defenses. I've considered intercepting the obvious ones, like the ones targeting wordpress plugins. I don't use wordpress. I could serve them a 200 instead of a 404 and hopefully waste a real humans time if they check the hits manually.

[–] [email protected] 1 points 2 weeks ago

Header spoofing is scraping 101