AI bots are scraping journalism all over the internet and repurposing it for users without giving credit where it’s due. For a news outlet like Bay City News that shares its content widely with partners, chasing down what are arguably copyright violations can seem like plugging a leak with your finger.

The good news is there are tools that can clamp that leak shut — at least on the websites we control. Cloudflare, a service that acts like a gatekeeper on internet traffic, has toggles that let you shut down specific bots, like ChatGPT-User and Claude-User, which crawl live sites in response to user queries. On Nov. 5, we started using Cloudflare on LocalNewsMatters.org — one of several news websites we run — and targeted these bots for blocking on Nov. 10. The response was quick and effective.

When we used Cloudflare’s AI Crawl Control tools to target and block ChatGPT-User and Claude-User, these scrapes went to zero right away.

The not-so-great news is that our client websites are republishing stories we send them — as is their right — and they are getting scraped themselves. Unless all of us agree to block, manage or otherwise incentivize monetization, we don’t have a way of preventing chatbots from summarizing our content from those sites.

Ultimately the issue is that we humans are producing the news at great expense — people need to be paid to go places, take pictures, do interviews, crunch data and write stories. But the chatbots are gobbling up the product of that labor, making billions of dollars in revenue by summarizing it, and not sending money, clicks or often even citations back our way. It’s disintermediation of journalism on a massive scale.

If AI chatbots capture all of the economic value of journalism without passing it on, the news business will cease to exist.

I argued in a previous post that collective action is one path forward. If all news outlets block these AI bots, the chatbot experience will be so degraded it will force AI companies to the table to pay fairly for the fruit of journalistic labor.

In the last few weeks, we’ve discovered the most effective of these blocking tools is enforcement on the edge. But it took some learning to get there.


At first we tried the gentleman’s agreement method: Robots.txt.

In a nutshell, scrapers are supposed to look at this file and obey if they are called out by name and told they are ‘disallowed’ from scraping, either the whole site or certain parts of it. On Oct. 22, we disallowed several live search bots that have been scraping our content. You can see ChatGPT-User and Claude-User name-checked below.

Unfortunately that didn’t work at all.

Not only did the numbers not move, when we inspected these AI labs’ documentation, it was clear that many explicitly ignore Robots.txt. Google placed its controls for opting out of AI experiences in its Google Search Console, because it commingles its search scrapers with AI experiences.

OpenAI says in its developer documentation (https://developers.openai.com/api/docs/bots) that its ChatGPT-User may ignore robots.txt

So on Nov. 10, we brought down the hammer — selectively of course, blocking the same bots we named in Robots.txt using Cloudflare’s tools. We also used Google Search Console to exclude our content from appearing in generative AI experiences on Google.

It worked. The results were immediate.

On Nov. 10 we used Cloudflare to block ChatGPT-User and Claude-User, bots that scrape sites based on user queries. These scrapes went to zero, while AI referral clicks and Google Search clicks stayed steady.

What’s more, referral clicks — both from AI experiences and regular Google Search — stayed steady.

The above data is from Axioma, an AI agent experience platform working with Newspack, a Wordpress plugin company that helps manage our websites. Axioma would prefer it if, instead of blocking, we make our website more amenable to bots and develop new ways of making money in that ecosystem. Similar to Cloudflare, it sits on the edge of the internet, right before agents request data from our website.

In truth, we don’t really want to block bots. We’d love to have the world experience our journalism, including in chatbot experiences, but be compensated fairly for it. Blocking is just the threat — and collectively, using options like Cloudflare, every publisher has access to tools that enable the threat to be carried out. The leverage gets even stronger if we act together.

But what’s that experience worth? How much would people pay to have an AI-mediated journalism experience? Are there other ways to have journalism be fairly valued in this new agentic ecosystem?

We plan to explore how to answer these questions at Bay City News Labs, a division of Bay City News Foundation, over the next several months, thanks to support from the Maynard Institute’s Fire Up Entrepreneurship program.

Among the topics we’ll be looking at are:

  • What does a chatbot experience look like with and without access to local news? (Hint: the experience is live on our site right now! We’ll report back on the results later)
  • How can we apply a license to our content that traverses the web?

Stay tuned.

Interested in supporting our work? Contact us at [email protected]

Ryan Nakashima is secretary of the board at Bay City News Foundation, technology advisor and participant in the Fire Up program, and held product roles at Hearst Newspapers and MediaNews Group. Reach him on LinkedIn.

Ask about the Bay Area

An A/B test by Bay City News Labs. Rate your experience and help save the future of journalism.

Try:

Secretary of the Board, Bay City News Foundation