The Chattanooga Times Free Press website still tells a visitor called Windows 95 to go away. It says the same to a handful of programs people once used to download a whole website over dial-up so they could read it offline. Alongside those, somebody more recently added Claude, GPTBot, DeepSeek, and about two hundred other things. All of it lives in a plain text file called robots.txt. Every website has one, or should. It is the sign on the door that tells automated visitors, the search engines and the AI crawlers and whatever else comes knocking, which rooms they are asked to stay out of. It is public. Anyone can read it. Almost nobody does, and fewer still would want the job of working out what it is and is not doing.
The Chattanooga file also mentions an Arkansas word game and a public salary database from the Arkansas Democrat-Gazette, because it came with the paper's former owner in Little Rock and nobody ever rewrote it. The Times Free Press was sold to a Chattanooga family on July 30. Fifty-four people lost their jobs that day. The old instructions stayed.
I spent thirteen years running the websites for a small newspaper group in Northeast Tennessee, through four content management systems. When McClatchy laid off more than 90 unionized journalists last Thursday, I read the coverage the way I read a server log. The memo tells you what management decided. The files on the server tell you what they had already done. This is a piece about that layer, the one underneath the newsroom, and about what a paper could do if it had control of it. I want these papers to survive. I should say that up front, because the criticism that follows comes from someone who would like them to earn their place in people's days again.
What happened, as documented
On September 10, McClatchy eliminated more than 90 unionized journalists and other union-represented workers across 17 of its publications, about 30 percent of the workers the NewsGuild represents at the company. The Guild's tally is the primary source; McClatchy has not published a paper-by-paper list, and the non-union count is not public. The Miami Herald and El Nuevo Herald lost 32 positions, including the entire El Nuevo Herald writing staff, the city hall reporter, the audience team, and the paper's only video producer. The Idaho Statesman lost ten journalists and three editors, roughly 60 percent of the room. The rest is in the chart.
The layoff notice, as reproduced by Poynter, said McClatchy was "reshaping its newsroom structure to more closely align resources with changing subscriber interests." The internal memo from Greg Farmer, the executive vice president of local news, gave the number behind that sentence: consumer revenue fell 41 percent over five years while local news spending stayed flat. "We cannot continue investing where subscriber interest does not support the investment."
Two things about that number. Nobody outside McClatchy can check it. The company has been private since a hedge fund, Chatham Asset Management, bought it out of bankruptcy in 2020, and "consumer revenue" is not defined anywhere a reader can find. Jon Schleuss, the NewsGuild president, said it "needs to be verified." I would have said the same thing about a number in a memo that justified my own layoff. And the closest study I could find points the other way. Boston University researchers analyzed a year of clickstream data from one large regional daily and estimated that ten more in-house stories a day would raise the subscription rate about 15 percent, while ten more wire stories a day would lower it about 11 percent. Those are estimates from the 2020 working paper; the final version was published in MIS Quarterly this month. One newspaper and a model, not McClatchy's books. But it is the only study I found that asked the question, and the answer was: cut the reporting, lose the subscribers you were trying to keep.
The work without a byline
Read the Guild's list of who was cut and a pattern shows up that the headline count hides. Miami lost its audience team, its only video producer, and its only female photographer in a round the company justified by "the ways audiences engage." Charlotte lost the one person who managed election data and built charts for the rest of the room. Poynter noted veteran copy editors on the list. The audience development editor for McClatchy's fifteen smallest newsrooms, one person covering the digital reach of half the chain, posted that she was in the same round.
This is not a McClatchy quirk. When Hearst bought the Dallas Morning News last year, the first cut was the 26-person multiplatform and copy desk, with print page production outsourced the same day it announced 18 newsroom hires. When the Fullers bought the Times Free Press, 30 of the 54 people let go were in print production and distribution. USA Today Co. dissolved its audience organization this month and replaced it with something called an Audience and Digital Production Team, in memos that blamed weaker search traffic.
I ran a digital department. I know how easily its work becomes invisible. A reporter's name is on the story. The person who makes that story load, display properly on a phone, and reach readers through Google rarely gets a byline. Their work shows up as an expense. Cut that support and the reporter may still be there, doing the work, with fewer people helping it reach an audience. That is a risk worth examining before treating weak engagement as proof that readers do not care.
Four owners, four robots files
This week I pulled the robots.txt file from seven newspaper sites: three McClatchy papers, one USA Today Co. paper, the Times Free Press, and the two Six Rivers Media dailies I used to maintain in Kingsport and Johnson City. A robots file is a request, not a lock; a well-behaved crawler honors it and a badly behaved one reads it as a map. It says nothing about paywalls or what Google decides to index. What it does do is record, in public, what the owner asked for. Four owners. Four different answers.
McClatchy uses one corporate template across Miami, Sacramento, and Lexington, identical down to the comment text. The distinctive finding is the shape of the block: the OpenAI, Anthropic, and Perplexity agents, Google-Extended, and three Internet Archive agents are disallowed from every story page but not from section pages, roughly a hundred more agents are disallowed site-wide, and one vendor's bot, Klangoo's Magnetbot, gets a global allow. The file does not mention the training crawlers for Meta, Apple, or Amazon. Whether that is a deal, an oversight, or a list nobody has opened since it was written, the file cannot tell you. I have maintained lists like that. Usually it is the third one.
USA Today Co., formerly Gannett, took the opposite approach at the Knoxville News Sentinel. Everything AI is disallowed at the root, including the Internet Archive and, for reasons I would enjoy hearing, the coding agent Devin. Then come the exceptions. GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, and PerplexityBot each get an allow for the sponsored-content folder plus two individually listed sponsor stories, one for a children's hospital and one for a nursing college. USA Today Co. asks AI crawlers to stay out of its journalism and points them at the advertising.
The Times Free Press is the strictest of the four and the only chain-built file that leaves the Internet Archive alone. More than 250 named agents are disallowed, including the Chinese models, Google's NotebookLM and Gemini Deep Research, and five coding assistants. It still carries the former owner's Arkansas paths. Six weeks after the sale, a Chattanooga newspaper is telling Googlebot to stay out of an Arkansas word game. Somebody will fix that eventually, probably the week after someone notices, which is how every robots file I ever touched got fixed.
Six Rivers is where I have to correct my own first draft, which said the Kingsport and Johnson City files were the same ones I left behind. They are not, and the Wayback Machine has the receipt: a capture from February 10, 2025, weeks before the platform move. That file, mostly the vendor's list with a few agents I added, disallowed thirty named agents at the root, including GPTBot, ChatGPT-User, and OAI-SearchBot; ClaudeBot, Claude-Web, and anthropic-ai; PerplexityBot; Google-Extended; Common Crawl; Bytespider; the Meta and Apple training crawlers; and both Cohere bots. Training and retrieval alike; the finer split I apply now on other sites, where a machine may read a page to answer a question but not train on it, came later. It had no Internet Archive rule, and it declared eighteen editorial sitemaps, one per year, 2007 through 2024. A declaration is not proof that every one of those years still resolved to live stories; migrations had carried roughly ten years forward each time, and some older entries may have been surviving declarations for pages already gone. But the file was telling Google where to look, year by year, back to 2007.
When the papers moved to mynews360 in 2025, the file went with the platform. It is hosted and vendor-managed, and what is on the server today is the vendor's default: ten administrative and asset paths, no AI rule, and no sitemap declared. The conventional address, /sitemap.xml, returned a 404 when I checked this week; whatever sitemap exists now is submitted some other way. The Internet Archive was unrestricted before and is unrestricted now. What changed is that the AI rules disappeared and the sitemap declarations disappeared with them. I did not write the new file, I cannot change it, and I was not consulted about replacing the old one. The point is not that the default is wrong. The point is that choices the people maintaining the local product had made went away with the migration. That is what losing local control looks like. It does not announce itself. It ships with the new platform.
The archive is the other thing on the table
The Internet Archive rules are the part of this story I keep coming back to. In January, Nieman Lab found that the New York Times, the Guardian, and USA Today Co. had started blocking the Archive's crawlers, on the theory that AI companies might scrape the Wayback Machine instead of the publishers' sites. By May, Nieman counted more than 340 local news sites doing the same, most owned by five of the seven largest chains, McClatchy included. The Atlantic's CEO said out loud what the others implied: blocking the Archive preserves leverage in licensing negotiations.
Leverage is a fine reason to lock a door. It is a poor reason to lock that particular one. For a small or mid-size paper, the Wayback Machine is often the most complete independent record of what its website published, held by someone other than the paper's own CMS vendor. I cannot speak to every paper's preservation arrangements. I can speak to ours. The papers I ran went through three platform migrations and a redesign in thirteen years. Each migration carried roughly the last ten years of stories forward and left the rest behind. Each one changed the URL structure, so every link anyone had ever shared to an older story broke. I argued against the loss every time, on the grounds that a 2004 city council story has value the 2004 traffic report does not capture. I lost every time, on the grounds that migrating 20 years costs more than migrating 10. Both arguments were correct. The archive got smaller anyway.
McClatchy's robots file shows something adjacent. Under the heading "Block indexing of specific sections/content at the market's request," the Miami Herald disallows crawlers from /news/special-reports/neglected-to-death and /news/special-reports/hurricane-andrew; Lexington disallows the Comair Flight 5191 crash report. These are the projects that won the awards, on paths from a section structure that no longer exists. A disallow does not mean the stories were deleted or are absent from search; answering that takes HTTP responses and Wayback captures. What the file establishes is narrower: those restrictions remain in the current file, alongside a rule for a URL extension from a CMS the company left around 2015, and two sections disallowed twice in identical blocks. I have written files like this. Everyone who has done the job has.
So here is the position local news is in. The best-known outside institution that archives news sites for free is being asked to stay out, to protect content from AI. The content is meanwhile being thinned by migration and orphaned by URL changes. Press Forward gave the Internet Archive a million dollars last year specifically to preserve local news that "often goes offline when a publication ceases to exist." The publications are blocking the grantee.
The distribution problem
The Farmer memo blames subscriber interest. Industry traffic data cannot explain McClatchy's five-year revenue number; it is a different population, period, and measure, and Chartbeat's own report notes total pageviews across its network fell only 6 percent, with some channels growing. What the data can describe is the door most readers came in through, and what happened to it. The door was Google. The door got smaller. Chartbeat found Google Search pageviews to publishers fell 34 percent and Google Discover 16 percent between December 2024 and December 2025. Facebook referrals are down 43 percent from their May 2023 level. Pew observed that when a Google result page carried an AI Overview, people clicked a traditional result 8 percent of the time versus 15 percent without one. All the chatbots combined send publishers less than one percent of their traffic.
The number that matters for a paper the size of the ones I ran is the split by publisher size: search referrals fell 60 percent at small publishers, 47 percent at mid-size, and 22 percent at large ones over two years, per Chartbeat via Axios. The bigger you are, the less you lost. USA Today Co.'s own memo this month blamed weaker search traffic and platforms keeping users instead of referring them. My reading: the reader left through the search results, and the company responded by disallowing the crawler, cutting the audience team, and calling the remainder subscriber interest.
One more thing in the record. McClatchy did not cite AI as the reason for the layoffs, and nothing shows automation replacing the eliminated jobs. But in early 2026 the company rolled out a Claude-based tool that turns reporters' stories into summaries, newsletter items, and video scripts; in March, 31 of the Sacramento Bee's 35 union journalists withheld their bylines from its output; in June the Centre Daily Times unionized over it; in July the Pacific Northwest Guild ratified a contract barring AI from bargaining-unit work except under a member's control, and six weeks later the company cut a third of that unit. That is the order things happened, not proof of cause. The same tools are arriving at the smallest papers, on platforms their own staff cannot configure.
Why it is one size fits all
If a paper wanted to offer readers more than one product at one price, why doesn't it? Because the subscription is not built where you think it is. Readers assume the website decides what it sells. It does not. The circulation system does, the same software that manages print routes and carrier payments, and at most small and mid-size papers it is the oldest thing in the building, and the building has a press in it.
A modern circulation system can run into six figures just to implement, before the monthly fees, before any custom work. No circulation vendor publishes a price. The only public figures I found are a custom-development firm's estimate of what it charges to build the equivalent: $90,000 to $200,000 for a subscriber and payment layer, $250,000 to $700,000 for a full replacement over one to two years. That is a build estimate from a firm selling the build, not a survey of vendor pricing. The order of magnitude matches what I saw. So papers buy the older version, or keep the one they have, and the older version has limits. It does not connect cleanly to a modern CMS. It cannot express a sports-only product or a web-only price, because those were not products when it was written. A day pass it can usually manage; most papers, including the ones I ran, already sell one that does not renew. That is the one option on the list the back office already knows how to do. I watched a CMS ship tools for dynamic subscriptions and watched the circulation system say no, and I watched it happen again on the next platform. The website was ready. The back office was not. That is why the papers I worked with sold one price. Not strategy. Plumbing.
The price itself has the same problem. When papers price the digital subscription, they price in the cost of print, which is fair because the bills are real. The reporting, editing, and photography cost the same whether the story is printed or not. But digital delivery avoids the press, the paper, the trucks, and the carriers, and a reader who never wanted those should be able to buy a version that does not include them. Twenty or thirty dollars a month for a website that was free five years ago is not a value proposition. It is a dare.
Advertising went the same way. When ad revenue fell, the fixes that paid were the interstitials, the full-screen ads that pay well and look terrible, and the programmatic networks where you do not know what ad is going to load next to your obituaries. That is the case for making local-only advertising the default on paid tiers, with an ad-free upgrade for readers who will pay for silence. There is one published number on the upgrade: in 2020, the Bay Area News Group offered an ad-free option for $4 more a month across 13 local sites, and about one in five new digital subscribers took it.
From inside the buildings, the smallest papers are about ten years behind on technology and the small-metro papers five or six, with some going backwards on purpose to save money. The vendors who sell into that market will not enjoy this paragraph, so I will be specific about what I am asking for: access rules the paper can edit, integrations between the CMS and the circulation system that work without a consultant, an archive and its sitemaps that move with the paper when the platform changes, and subscription products the paper can configure without a change order.
What I would try
I think the paywall era can work. At the papers I know, it has not worked because they could only sell one product at one price, and when readers did not want it, the decline got called subscriber interest. Here is what I would test. A few options, not twenty, and every one of them something a person could explain to a neighbor.
The line that matters is between public-service information and commodity news. An evacuation notice can take real reporting and should still be free, because the town needs it and because a locked public-safety story donates its traffic to whoever posts it open. What earns a subscription is the in-house work: the investigations, the columnists, the depth on local sports, the county commission meeting nobody else attended. The Boston University study says the same thing in a model. Lock what is scarce. Open what is not.
The sports package is not a theory, and the evidence comes from the company that just cut its sports desks. In August 2018 the Miami Herald built Sports Pass in two weeks, $30 the first year against a $129.99 full subscription. McClatchy rolled it to ten markets within a year and reported subscribers in the "thousands," about 80 percent of Miami's from outside South Florida. Nobody has published retention or cannibalization numbers since, and the Dallas Morning News later folded its own sports products back into the main subscription without saying why. Take it as proof of demand, not a business plan. But the demand was there, at $2.50 a month, from readers outside the market the full subscription was priced for.
The Washington Post tested the rest. Starting in September 2024 it tried week passes, then a $4 day pass, then $2 per article. Its own report says the day pass lifted conversion to paying access 35 percent and the article 83 percent (relative lifts, and paying access, not necessarily a recurring subscription), that 42 percent of buyers were people the paper had never identified before, that 91 percent bought once, and that a few percent upgraded to a subscription within six months. It also reported that its conventional paywall converts about one percent of the people who hit it. Kingsport is not Washington. But it is a large publisher testing the thing small ones say cannot be tested, and the direction is the one I would bet on.
Two qualifications, both from watching this go wrong. More options have to feel simpler to the reader, not more confusing; three choices a person understands beat a pricing page that needs a legend. And affordable has to be true at renewal, not just on the offer. I watched readers spend a little patience on every migration. A surprise renewal price spends it faster.
If I were starting tomorrow at a paper this size, the first step would be small. Ask occasional readers and former subscribers what stopped them from paying or staying; the answers are cheaper than any consultant. Put one affordable offer next to the existing subscription and the day pass most papers already have; a web-only price is the likeliest candidate the back office can express. Track four numbers for a quarter: new paying readers, repeat visits, cancellations, and net revenue after existing subscribers who move to the cheaper plan. Then decide whether to add the second option from those results, not from a pricing deck. That is the working goal. It fits on an index card, which is more than the last three vendor decks I sat through could say.
The other half is trust
Options do not fix trust by themselves. The industry took some of that damage from outside and earned some of it, and either way the road back does not run through the guilt appeal. Asking a community for support is legitimate. I have no problem with a paper saying it needs its readers. What I have a problem with is asking in place of the work.
The ask cannot stand in for prices and renewal terms people understand. It cannot stand in for cancellation that takes one click, or pages that load and logins that work, or corrections a reader can find, or a visible line between reporting, opinion, and advertising. It cannot stand in for asking residents what they need covered and then covering it.
On prices and cancellation, the industry is moving the wrong way. As of this summer, several large publishers disclose in renewal notices that an algorithm set your price, with one chain's subscribers seeing $130, $145, and $175 for a product posted at $100 and $200. The numbers on cancellation friction cut both ways: Newsday found 29 percent of people who started cancelling online never finished, and a 2026 field study at an unnamed regional paper found one-click cancellation barely moved cancellations and left the people who did cancel more engaged than the ones who had to fight their way out. A payment you kept by hiding the button is revenue. It is not loyalty.
Do the work and the ask gets easier. Skip it and the ask sounds like begging, and people can tell the difference.
What is being tried elsewhere
None of this is the only thing being tried. California's AB 2222 would pay newsrooms a refundable credit of $20,000 for each of their first five journalists and $15,000 for each one after that; it passed with supermajorities and sits on Gov. Newsom's desk with a September 30 deadline. Illinois ran a smaller version in 2025. Press Forward has committed more than $500 million in philanthropy. Nonprofit ownership and local ownership are both being tried, with mixed results; the Fullers bought the Times Free Press to keep it in Chattanooga and laid off 54 people the first morning. I am for most of it. None of it fixes the product. A tax credit for a reporter's salary does not redirect a broken URL, give a reader a price they can afford, or hand a paper back the settings its vendor took. That part is on the papers.
The lesson
Carl Hiaasen, the novelist and longtime Miami Herald columnist, worked at the paper for more than 40 years. The day after the cuts he posted on Facebook that they were "a death blow to the already skeletonized Miami Herald, once one of the country's top newspapers." The post ended: "If you own a newspaper company, don't sell it to a hedge fund unless you truly don't give a shit about your readers." That is the lesson from the newsroom side, and it is the right one.
From the server side the lesson is smaller and closer to home. The robots.txt files say what each owner decided to protect and from whom. The one the papers ran until early 2025 said: keep the AI crawlers out, and here is a map to seventeen years of our stories. Those rules are gone, and so is my ability to set them, not because anyone at the paper disagreed with them, but because a migration moved the file to a vendor and the decision went with it. I want the papers to have that control back. I want them to use that control to serve readers: keep the settings you can reach, and use them to give people a way in they can afford.
I spent thirteen years keeping these sites up. I would like them to still be here when I am done.
Sources
- NewsGuild-CWA, "McClatchy laid off 30% of its unionized workers," Sept 11, 2026. newsguild.org
- Associated Press, "For the US news industry, a tough summer draws to a close with McClatchy chain's cuts," Sept 12, 2026. apnews.com
- Poynter, "McClatchy cuts more than 90 staff in its most severe layoffs in years," Sept 10, 2026. poynter.org
- TheWrap, "McClatchy Guts Newsrooms Nationwide Months After Controversial Push to AI," Sept 10, 2026 (Farmer memo, Schleuss quote). thewrap.com
- Kentucky Public Radio / LPM, Sept 10, 2026 (Lexington count, layoff email wording, SPJ Louisville statement). lpm.org
- Boise State Public Radio, Sept 11, 2026 (Idaho count). boisestatepublicradio.org
- St. Louis Public Radio, Sept 11, 2026 (Belleville). stlpr.org
- Fresnoland, Sept 10, 2026; The Charlotte Ledger, Sept 10, 2026; SFGate, Sept 11, 2026; CalCoastNews, Sept 2026 (per-paper counts).
- Yang, Sahoo, and Simcoe, "Effects of In-house and Wire Content Mix on Online Newspaper Subscriptions," MIS Quarterly 50(3), Sept 2026; working paper June 2020. MIS Quarterly; 2020 working paper (SSRN); summary at inma.org
- Editor & Publisher, "Hearst to lay off 26 at the Dallas Morning News," Nov 19, 2025. editorandpublisher.com
- Chattanooga Times Free Press, "7 things to know after Fuller family buys Chattanooga Times Free Press," July 30, 2026; WDEF, July 31, 2026; Columbia Journalism Review, "Chaos in Chattanooga," Aug 7, 2026. timesfreepress.com
- Search Engine Land, "USA TODAY overhauls audience strategy, citing search pressure," Sept 4, 2026. searchengineland.com
- robots.txt files for miamiherald.com, sacbee.com, kentucky.com, knoxnews.com, timesfreepress.com, timesnews.net, and johnsoncitypress.com, captured Sept 13, 2026, by the author.
- Nieman Lab, "More than 340 local news outlets are limiting the Internet Archive's access to their journalism," May 20, 2026. niemanlab.org
- Internet Archive, "Today's News for Tomorrow" Press Forward grant announcement, July 16, 2025. blog.archive.org
- Press Gazette, AI deals and lawsuits tracker, updated Sept 7, 2026 (McClatchy v. Cohere, Gannett and Perplexity). pressgazette.co.uk
- Chartbeat, "Pageviews are down, but AI's impact is complicated," June 2026. chartbeat.com
- Axios, "Chartbeat: search traffic to small publishers down 60%," Mar 17, 2026. axios.com
- Pew Research Center, "Google users are less likely to click on links when an AI summary appears in the results," July 22, 2025. pewresearch.org
- TheWrap, "McClatchy Journalists Revolt Against AI," Apr 7, 2026; Nieman Lab, "The Centre Daily Times unionizes after backlash to McClatchy's AI tool," June 10, 2026; Northwest Labor Press, "AI, meet CBA," Sept 2, 2026.
- Presteligence, "Presteligence introduces AI tools to enhance newspaper workflows," Oct 26, 2025. presteligence.com
- Rebuild Local News, "California Legislature Passes Nation's Largest Proposed Investment in Community News," Aug 31, 2026; Los Angeles Times, Aug 29, 2026 (opposition and Department of Finance analysis); California Franchise Tax Board bill analysis, AB 2222 (credit amounts). rebuildlocalnews.org
- Lenfest Institute, "How The Miami Herald newsroom drove the creation of a sports-only subscription product," June 27, 2019; Poynter, Sept 17, 2018; Digiday, "With sports, local newspapers try unbundling the subscription," 2019; Better News, Feb 2024. lenfestinstitute.org; digiday.com
- Poynter, "How premium, ad-free subscriptions are working in local news," Aug 25, 2020. poynter.org
- Wayback Machine capture of timesnews.net/robots.txt, Feb 10, 2025. web.archive.org
- Digital Heroes, custom-development cost estimate for circulation software (vendor), June 26, 2026. digitalheroesco.com
- Nieman Lab, "This price was set by an algorithm," July 22, 2026. niemanlab.org; American Press Institute on Newsday cancellation flow, Oct 12, 2021; University of Hong Kong Business School, "Effects of click-to-cancel on a regional newspaper," Mar 12, 2026.
- The Washington Post, "Flexible Access White Paper" (week pass, day pass, and pay-per-article tests, 2024 to 2025). subscription.washingtonpost.com; The Audiencers, June 29, 2026; Nieman Lab, Sept 16, 2024. niemanlab.org
- Dustin Lifford, "Thirteen Years Behind the CMS of Local News," July 9, 2026. dustinlifford.com
- Rebuild Local News, "Supporting Local Reporting in Illinois: Early Lessons," Feb 5, 2026. rebuildlocalnews.org
- National Trust for Local News, Colorado restructure announcement, May 13, 2025; Free Press Initiative interview with the Trust's CEO, Aug 11, 2026.
- Carl Hiaasen, public Facebook post, Sept 11, 2026. Kevin Slimp and Steven Waldman, public LinkedIn comments, Sept 12, 2026.