New Guidance Helps Content Owners Evaluate Management Approaches for Bots and Crawlers To Support CoMP API Adoption With the Goal of Creating A Sustainable Content Marketplace; For Public Comment Until June 25, 2026

NEW YORK, May 27, 2026 /PRNewswire/ — Today, IAB Tech Lab, the global digital advertising technical standards-setting body, announced the release of new guidance on bot and crawler management strategies, now available for public comment until June 26, 2026. The guidance outlines different approaches content owners and publishers can take to inform access from non-human user agents, helping the ecosystem make more informed decisions alongside the adoption of the CoMP API V1.

“Giving content owners clear, practical guidance is key if we want adoption to move forward in a meaningful way,” said Anthony Katsur, CEO, IAB Tech Lab. “This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace.”

Following the release of the CoMP API V1, it became clear that many content owners and publishers have not yet developed formal strategies around their approach to bots and crawlers, which sit outside the scope of the API itself. To address this gap, the CoMP Working Group developed this guidance to outline the range of available approaches and the advantages and trade-offs of each strategy. This gives stakeholders a clearer understanding of how to manage access from all non-human traffic, including AI systems in a way that aligns with their business goals.

The guidance is designed to support the adoption of the CoMP API by helping content owners navigate an area that can be operationally complex and resource-intensive. It is aimed at non-Technical leadership to give them the questions they need to ask. Those questions will ultimately help them understand the value and hard costs associated with each lever they pull.  With clearer direction on these strategies, organizations can make more confident decisions, reduce risk, lower their operating cost, and operate more efficiently as they engage with AI systems. Establishing these strategies is also critical to enabling partnerships between AI systems and content owners, which are necessary to build a sustainable and scalable content marketplace.

Industry participants involved in the development of the guidance emphasized the importance of clarity and practical direction as the ecosystem evolves:

“Content owners are being asked to make important decisions quickly, often without clear frameworks,” said Shailley Singh, EVP & COO, IAB Tech Lab. “This guidance helps break down the options so they can choose an approach that aligns with their goals.”

“The AI revolution has caught many publishers with their front doors wide open, essentially leaving the lights on for bots and crawlers to pilfer intellectual property.” said Scott Messer, Principal & Founder, Messer Media, “This IAB Tech Lab guidance is the essential roadmap content owners need—a practical framework to navigate the complex maze of decisions, tech, and tools available to protect their business. By empowering publishers to identify who is really there, this guide helps them distinguish between a value-driving ‘ally’ and a resource-draining ‘extractor’. Managing AI bots is no longer just a technical background task; it is a critical business strategy for protecting what you’ve built and restoring a fair exchange of value for your content.”

“Non-human traffic now dominates most of the open web, and AI agents and crawlers are steepening the curve. Blanket bot-blocking is no longer a useful strategy. For publishers, advertisers, and ad-tech alike, distinguishing beneficial automation from invalid traffic through graduated, standards-based controls has become a baseline requirement,” said Dave Bellous, VP of Strategy, Metal Toad. “The Bot and Crawler Guidance from IAB is a much-needed tool in crafting a modern web strategy.”

“As AI agents increasingly serve as proxies for consumers, publishers and platforms need clearer frameworks for transparency and control,” said Tom Koch, Head of Sales, Ads & Marketing Ecosystem at TwelveLabs. “The crawler management framework is an important step toward giving content owners greater visibility into how their content is accessed and used. TwelveLabs is proud to contribute as this work is foundational to ensuring video, the richest form of data, is no longer overlooked in how AI engages with the web.”

The guidance was developed by the CoMP Working Group, with public comment providing an opportunity for industry feedback ahead of finalization. IAB Tech Lab will continue working with members to support implementation and encourage adoption of both the guidance and the CoMP API.

IAB Tech Lab invites the industry to participate in the public comment period until June 25, 2026. To review the guidance, https:///iabtechlab.com/botmanagement/

About IAB Technology Laboratory
Established in 2014, the IAB Technology Laboratory (Tech Lab) is a non-profit consortium that engages a member community globally to develop foundational technology and standards that enable growth and trust in the digital media ecosystem. Comprised of digital publishers, ad technology firms, agencies, brands, and other member companies, IAB Tech Lab focuses on solutions for brand safety and ad fraud; identity, data, and consumer privacy; cross-channel programmatic effectiveness and ad measurement; and the impact of LLMs and AI agents on advertising. Its work includes the OpenRTB real-time bidding protocol, ads.txt anti-fraud specification, Open Measurement SDK for viewability and verification, VAST video specification, Trusted Server, and the Agentic Roadmap initiative for agentic advertising. Board members and companies are listed at https://iabtechlab.com/about-the-iab-tech-lab/tech-lab-leadership/. For more information, please visit https://iabtechlab.com.

View original content to download multimedia:https://www.prnewswire.com/news-releases/iab-tech-lab-releases-guidance-on-ai-system-bot-management-strategies-for-public-comment-302782667.html

SOURCE IAB Tech Lab