Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Who gets to decide what is a good bot?
 help



Presumably the site owner with Cloudflare providing enforcement.

Generally isn't a good bot one that respects robots.txt and is respectful of the site's resources by not being spammy?


by checking which follow robots.txt?

According to their plan, the good bots pay for scraping.

> Who gets to decide what is a good bot?

Paying Cloudflare's tolls => good bot.


Cloudflare, apparently.

no, their customers. Why do you assume that people who run websites are clueless and lack agency in this?

inb4 "we're updating our Terms of Service - all Cloudflare-based scrapers will now automatically be enrolled to scrape your cached websites"

and they'll explain how it's great because it's only reading the cached versions, not hitting upstream.


Me… obviously



Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: