AI advertising publishers' new compensation structure revealed by IAB Europe
In an effort to address the rising issue of declining publisher traffic revenues due to increased AI scraping, IAB Europe has published a new framework titled "Crawling for Compensation". The framework, authored by IAB Europe Data Analyst Dimitris Beis, was published in September 2025 and aims to establish fair compensation for content used in AI training and inference.
The framework addresses three major challenges: controlling AI content access, auction dynamics, and content valuation. It establishes three mechanisms for AI platform compensation to publishers: content access controls, discovery protocols, and monetization APIs including Cost-per-crawl and LLM ingest content APIs.
One of the key features of the framework is the introduction of Cost-per-crawl (CPCr) APIs. These APIs feature tiered pricing based on content type, bot classification, and access frequency.
Unauthorised scraping increased significantly from Q3 to Q4 2024, with robots.txt compliance declining significantly. The framework proposes blocking unauthorised AI scraping through robots.txt files and Web Application Firewall methods.
The news and media sectors experienced a significant growth of 770% traffic from AI platforms during the same period. Adobe research found that AI-referred visitors stayed 8% longer on sites, viewed 12% more pages, and showed 23% lower bounce rates.
The framework applies globally but emphasizes European implementation through IAB Europe's Artificial Intelligence Working Group. European publishers can contact Dimitris Beis at beis [at] iabeurope [dot] eu for participation information.
The framework was developed following industry discussions throughout 2025, including the July IAB Tech Lab summit and August working group launch. In June 2025, AI platform referrals increased 357% year-over-year, reaching 1.13 billion visits compared to 191 billion visits from organic Google search. According to Cloudflare CEO Matthew Prince, the ratio of pages crawled to visitors referred increased from 2:1 a decade ago to 18:1 in June 2025.
While the IAB Europe framework has been published, no specific AI platforms are explicitly named as having established connections with the lab to participate in implementing this framework. The framework addresses a paradigm of publisher remuneration for content ingestion and is a significant step towards ensuring fair compensation for publishers in the AI era.
Read also:
- Web3 gaming platform, Pixelverse, debuts on Base and Farcaster networks
- Goodyear in 2025: Advancement in Total Mobility through the Launch of Kmax Gen-3 by Goodyear
- Electric SUV Showdown: Vinfast VF6 or MG Windsor EV - Your Choice Revealed
- United States Secures $632 Million to Fuel Electric Vehicle Revolution
