What is Cloudflare’s new tool and how does it empower digital content creators?
Cloudflare has unveiled a new feature called “Pay‑per‑Crawl”, designed to give website owners the ability to monetize or block artificial intelligence bots that crawl their content.
This innovation addresses a growing issue in the digital ecosystem: AI companies such as OpenAI, Anthropic, and others send bots that scan vast amounts of content from across the internet to train their large language models (LLMs). However, these bots often consume data without generating any traffic, credit, or revenue for the original websites. According to Cloudflare, traffic from AI bots can be up to 1,500 times higher than traditional search engine crawlers, yet offers nothing in return.
With Pay‑per‑Crawl, site owners can either set a custom fee per access or outright block AI bots that don’t agree to pay. Cloudflare acts as the intermediary, negotiating and handling payments between AI companies and content providers.
This move has already attracted support from major publishers like Reddit, Condé Nast, The Associated Press, and Pinterest, all of whom are seeking fairer terms in the new AI-driven content economy.
By putting control back in the hands of content creators, Cloudflare’s tool aims to establish a more equitable, transparent, and sustainable internet, one where AI benefits don’t come at the cost of those who produce the content it learns from.
