Every week, we review the impacts of our journalism as a team- but our Google Analytics setup can’t see scrapers that don’t trigger JavaScript.
AI agents scrape our HTML, summarize it for their users, and send almost no traffic back. They don’t generate ads. They don’t donate. They mask their presence. Bay City News Labs is seeking pathways to address this challenge, and we could use partners in addressing this issue and then sharing our findings with the broader local news community to help all indies capture metrics and money.
Bot traffic has long been part of the Internet, and in the current age of AI bots are becoming a greater presence and a greater challenge when it comes to web traffic. This brings up a host of new questions and challenges for newsrooms. Should we block bots or give them access? Will agents one day subscribe to their favorite news sources? We believe collective action in particular will help us address the issue that scraping is not providing fair and direct compensation to all news providers.
Big media can cut deals. We can’t, threatening our sustainability.
Major Licensing Deals for Big Media:
News Corp + OpenAI
$250M over five years
News Corp + Meta
Up to $50M per year for three years
AP + OpenAI
Licensed text archive + tech access
Independent Reality:
No seat at the table
Indie operators are scattered, with no collective bargaining power.
Limited tech access
Dependent on platforms like Wordpress and Newspack for publishing infrastructure.
Blind to the threat
Analytics can’t see bot traffic — they don’t trigger JavaScript — so outlets can’t combat the problem
Bay City News Labs is seeking a solution.
We aim to focus our team on solutions that can help sustain us and our independent media peers. We need control of our own content in order to capitalize on our value in the marketplace. We have developers ready to test, product managers ready to create a work plan and framework, and a revenue team ready to build partnerships with others and find ways to monetize this work – with deals, a better understanding of our audience, and even consulting.
The results will benefit all independent local news providers — reusable open tools, playbooks, and tested partnerships that any small operator can adopt, leading to more control and revenue opportunities.
1. Gain Visibility
Move to cloud hosting level to see all bot traffic — current JavaScript-based analytics miss AI agents entirely.
2. Negotiate Access
Work with Newspack and Cloudflare to establish rules for which bots get through and which don’t.
3. Experiment
Find new ways to test tools while navigating ethical boundaries, maintaining trust with our communities.
1. Build & Share Tools
Create reusable tools for indie operators to benefit from AI superpowers to boost newsgathering activities.