How AI Crawlers Actually Read the Web, and Why Publishers Are Demanding Model Destruction

Website administrators decide exactly which artificial intelligence bots can read their pages by editing a single text file, a choice they can update at any time. But for data scraped before those rules were written, the U.S. District Court for the Southern District of New York will now decide whether the resulting AI models must be dismantled entirely. On September 4, 2026, The Seattle Times and Newsday filed a federal lawsuit against OpenAI and Microsoft, escalating the ongoing conflict over digital copyright.[1][2]

The two publications accuse the technology companies of scraping their paywalled journalism without permission to train generative AI systems, including ChatGPT and Microsoft Copilot. Unlike previous lawsuits that primarily sought financial compensation, this filing demands the physical destruction of the training datasets and the AI models that incorporate their work.[1][2]

The legal confrontation exposes the underlying mechanics of how artificial intelligence systems actually acquire their knowledge. To build a large language model, developers deploy automated software programs known as web crawlers or bots. These programs systematically navigate the internet, following hyperlinks from one page to the next, downloading the text they encounter along the way.[3]

Web crawlers have long served as the foundation of digital discovery. Search engines like Google and Bing use them to index the web, allowing users to find relevant pages. When a search crawler visits a site, the implicit transaction is an exchange of access for visibility: the search engine reads the content, and in return, it directs human traffic back to the publisher.[3]

AI training crawlers operate under a different economic model. When a bot like OpenAI's GPTBot reads a news article, it is not indexing the page to send a reader there later. Instead, it is extracting the text to teach a neural network how to generate human-like language, predict the next word in a sequence, and answer user queries directly.[3]

The publishers argue this extraction fundamentally undermines their business. The lawsuit alleges that the AI products can reproduce passages from their reporting and closely paraphrase full articles, providing users with answers that reduce the need to visit the original website or purchase a subscription.[1][2]

To control these automated visitors, the internet relies on a voluntary protocol established in 1994: the robots.txt file. This simple text document sits at the root directory of a website and acts as a traffic controller, listing specific instructions for different bots.[3]

An administrator can add a line of code to their robots.txt file explicitly disallowing a specific user-agent, such as GPTBot. When a compliant crawler arrives at the site, it reads the file, sees the restriction, and leaves without downloading the content. When a site administrator updates this file, it typically takes roughly 24 hours for major AI systems to adjust their crawling behavior.[3]

However, the robots.txt system has significant limitations. It is entirely an honor system; malicious or poorly configured bots can simply ignore the instructions and scrape the site anyway. More importantly for the current litigation, a robots.txt directive only governs future behavior. It cannot retroactively delete data that a bot scraped before the rule was implemented, nor can it extract that data from a model that has already been trained.[3]

The distinction between different types of AI crawlers has also complicated the landscape for publishers. Major AI laboratories now operate multiple, independently governed bots. For example, OpenAI deploys GPTBot specifically to collect training data, but it uses a separate crawler, OAI-SearchBot, to index pages for its real-time search features.[3]

If a publisher blocks GPTBot, they prevent their future articles from being used to train the next generation of models. But if they accidentally block OAI-SearchBot in the process, they remove themselves entirely from ChatGPT's search answers, cutting off a growing source of referral traffic. Industry data cited in similar publisher lawsuits indicates that search referral traffic to midsize publishers fell by 47 percent year-over-year in December 2025.[3]

The Seattle Times and Newsday are not the first to challenge this ecosystem. The New York Times filed a similar lawsuit in December 2023, and a coalition of nearly 400 local newspapers followed suit in June 2026. The addition of two prominent regional papers underscores the specific vulnerability of local newsrooms, which rely heavily on subscription revenue to fund their reporting.[1][2]

Microsoft has publicly responded to the latest filing with a measured stance. A spokesperson for the company stated, "While we're surprised by the lawsuit, we appreciate the importance of local journalism and we're always happy to sit down and explore solutions to this type of dispute."[1][2]

The core legal question now rests on the doctrine of fair use. AI companies maintain that training models on publicly available data is a transformative act protected by copyright law, while publishers argue it is wholesale theft that directly competes with their original products. The court's eventual ruling will establish the boundaries of what automated systems are permitted to learn from the open web.[1][3]