Cloudflare announced the launch of an experimental Pay per Crawl marketplace, allowing website owners to control AI bot access to their content and earn payment for each “crawl”. Participants in the private beta can decide whether to allow individual AI companies to scan their site for a fixed micropayment, for free, or to completely block such access. Cloudflare’s tools enable users to see exactly what these scans are used for — for instance, for training models or for responses in AI search.
To participate in the marketplace, both website owners and AI companies must have Cloudflare accounts. They can set their own rates for “crawl”, and Cloudflare acts as an intermediary in transactions, distributing payments between parties. The company emphasizes that currently, Pay per Crawl does not use cryptocurrencies or stablecoins, but is considering the possibility of implementing such payment methods in the future.
Since June, Cloudflare automatically blocks all AI bots on new sites added to the service. Administrators can manually grant access to specific bots if they deem it necessary. Major publishers, including TIME, The Atlantic, ADWEEK, Fortune, Conde Nast, and The Associated Press, have already joined the initiative, supporting the approach of allowing scanning instead of automatic data collection.
According to Cloudflare, the ratio of scans to site visits from different bots varies significantly. For example, in June, the Google bot performed 14 scans per visit, OpenAI — 1700, and Anthropic — 73000. Meanwhile, new “agent” AI services can independently visit sites, gather information, and provide it to users without visiting the original source, impacting traditional content monetization models.
