Cloudflare's innovative "pay per crawl" lets publishers charge AI crawlers for content

Cloudflare has launched a new "pay per crawl" feature in private beta that allows content creators to monetize their work by charging AI crawlers for access, offering a middle ground between blocking AI entirely or allowing free access.

The system leverages HTTP response code 402 (Payment Required) and established web authentication mechanisms. When AI crawlers request content, they either present payment intent through request headers for successful access or receive a 402 response with pricing information. Publishers can set flat, per-request pricing across their entire site and choose to allow, charge, or block specific crawlers.

The feature addresses a common frustration among publishers and news organizations who want to allow AI access to their content while receiving compensation. Previously, this required individual negotiations with AI companies—a challenge for smaller publishers lacking scale and leverage.

Cloudflare acts as the merchant of record, handling billing and payment distribution. The system uses Ed25519 cryptographic signatures to prevent crawler spoofing and ensure authentic payments.

Looking ahead, Cloudflare envisions the technology enabling dynamic pricing models and supporting an "agentic world" where AI agents can programmatically negotiate and pay for content access with predefined budgets, reshaping how digital content is valued and accessed online.