Unadhi Guides

EU text and data mining opt-out check sheet for client websites: robots.txt groups for GPTBot, Google-Extended, ClaudeBot, Applebot-Extended and CCBot plus a website terms sentence, under Directive (EU) 2019/790 Article 4(3) and AI Act Article 53(1)(c), for accountants, web agencies and SMEs

Updated 9 Nov 2026 · Disclosure: the EURO AI Register mentioned below is an Unadhi product with a free edition. Legal facts come from Directive (EU) 2019/790 on copyright and related rights in the Digital Single Market (OJ L 130, 17.5.2019): Articles 2(2), 3, 4 and 7(1) and recital 18; and Regulation (EU) 2024/1689 (AI Act, OJ L, 2024/1689, 12.7.2024): Article 53(1) and (4), Article 111(3), Article 113 and recitals 104 to 108. EUR-Lex returned HTTP 202 with an empty body on 9 Nov 2026, so we read the same official Official Journal texts from the EU Publications Office repository (Cellar) on 9 Nov 2026. Crawler names come from each operator's own published crawler page and the robots.txt rules from RFC 9309, all read on 9 Nov 2026. We did not read any national copyright law or court ruling. This guide is for information only, not legal advice.

Partner programme opening soon: join the free waitlist. Planned terms are 50% of each referred sale (75% once you have 5+ paying customers), single level, paid through Gumroad only from revenue received.

What you get: a finished text and data mining opt-out check sheet, one row per website (your firm's own site and each client's site). You record whether the client wants to keep its texts and images out of AI training, what /robots.txt says for five AI training crawler tokens (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot), whether the AI search bots are blocked, and whether the website terms carry a reservation. The formulas return a robots.txt status that names each crawler not yet disallowed, a terms status, an AI search visibility warning, the next re-check date, a to-do list and a result (OK, FIX, RECHECK, CHECK BY HAND or NO RESERVATION). A third tab holds a ready robots.txt block. Also below: a terms sentence, a client email and a checklist. We tested the formulas in LibreOffice on fourteen made-up websites before publishing.

Short answer: In the EU, anyone may copy lawfully accessible content for text and data mining unless the rightholder has expressly reserved that use "in an appropriate manner, such as machine-readable means in the case of content made publicly available online" (Directive 2019/790, Article 4(3)). For online content, recital 18 says the reservation should be made "by the use of machine-readable means, including metadata and terms and conditions of a website or a service". Since 2 August 2025, providers of general-purpose AI models placed on the EU market must have a policy to "identify and comply with, including through state-of-the-art technologies" such reservations (AI Act Article 53(1)(c); Article 113(b)). Models placed on the market before that date must comply by 2 August 2027, 266 days from 9 November 2026 (Article 111(3)).

Why accountants, tax advisers and web agencies should look at this now

Step 1: decide, client by client

  1. Ask each client: do you want your website texts, photos and other content kept out of AI model training? Record Y, N or Undecided in column C. This is the client's decision as rightholder, not yours.
  2. Ask whether any content is licensed to an AI company by contract (column N). Recital 18 says that in other cases rights can also be reserved "by other means, such as contractual agreements or a unilateral declaration".
  3. Ask whether the client wants to appear in AI search answers (column L).

Step 2: check /robots.txt

  1. Open https://client-site.example/robots.txt. The rules must be in a file named "/robots.txt" at the top level of the site (RFC 9309, Section 2.3). If the address returns a 4xx error, the file counts as unavailable and "the crawler MAY access any resources on the server" (Section 2.3.1.3). Enter Y in column D only if the file loads.
  2. For each token below, tick Y if there is a group with that User-agent line and Disallow: / (or User-agent: * with Disallow: /, which blocks every crawler, search engines included). Crawlers must match the user-agent name case-insensitively (Section 2.2.1).
  3. Log the date in column E. Crawlers may cache robots.txt but "SHOULD NOT use the cached version for more than 24 hours" (Section 2.4). The sheet asks you to re-check every 90 days (Settings!B4, our choice), because sites get rebuilt and robots.txt files get overwritten.
TokenWhat the operator's own page says (read 9 Nov 2026)
GPTBot (OpenAI)Disallowing GPTBot indicates "that crawled content should not be used for training OpenAI's generative AI foundation models". OAI-SearchBot is a separate setting for search results.
Google-Extended (Google)A "standalone product token" to manage whether crawled content "may be used for training future generations of Gemini models" and for grounding. It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".
ClaudeBot (Anthropic)Restricting ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets". Claude-SearchBot and Claude-User are separate.
Applebot-Extended (Apple)Publishers "can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended".
CCBot (Common Crawl)"To prevent Common Crawl from crawling your website", add User-agent: CCBot and Disallow: /.
PerplexityBot (Perplexity)Surfaces sites in Perplexity search results; "It is not used to crawl content for AI foundation models." Counted in column K, not as a training crawler.

RFC 9309 also says: "These rules are not a form of access authorization." A crawler that ignores robots.txt is not stopped by it. The file is a signal and a dated record, not a lock.

Step 3: the robots.txt block and the terms sentence

# AI training crawlers: text and data mining reserved (Directive (EU) 2019/790, Art 4(3))
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Add it to the existing file; do not replace the groups for normal search engines. Sentence for the website terms (adapt with the client's lawyer if needed):

Text and data mining: [Company] expressly reserves the rights to make
reproductions and extractions of the content of this website for text
and data mining, including for training AI models, under Article 4(3)
of Directive (EU) 2019/790. This does not affect text and data mining
for scientific research under Article 3 of that Directive.

The research carve-out is in the law: a reservation does not affect Article 3 text and data mining by research organisations and cultural heritage institutions (Article 4(4)), and contract terms contrary to Article 3 are unenforceable (Article 7(1)).

The sheet: columns

Tab Sites (one row per website), tab Settings (B1 today, use =TODAY(); B2 = 2027-08-02; B3 = days left; B4 = 90-day re-check; B5 = 2026-08-02) and tab Robots block.

ColWhat to enter or what it showsSource
A–BClient, website URLYou
CClient wants to reserve text and data mining / AI training rights? Y/N/UndecidedArt 4(3)
D/robots.txt answers with HTTP 200? Y/NRFC 9309 Sec 2.3
Erobots.txt checked on (date)You
F–JGPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot disallowed? Y/NOperators' pages
KAI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) blocked? Y/NOperators' pages
LClient wants to appear in AI search answers? Y/NYou
MWebsite terms state a text and data mining reservation? Y/NRecital 18
NContent licensed to an AI company by contract? Y/NRecital 18
ONotesYou
PRobots.txt status, naming each token not yet disallowedArt 4(3); recital 18; RFC 9309
QTerms statusRecital 18
RAI search visibility warning or noteOperators' pages
SNext robots.txt check (E + 90 days), or OVERDUESettings
TTo-do listAs cited
UResult: OK, FIX, RECHECK, CHECK BY HAND, NO RESERVATIONSummary

Formulas (row 2; fill down). In Excel files saved by some tools TEXTJOIN shows as _xlfn.TEXTJOIN; that is the same function.

P2: =IF(A2="","",IF(C2="N",IF(COUNTIF(F2:J2,"Y")>0,"CHECK: CLIENT DOES NOT WANT A RESERVATION BUT ROBOTS.TXT BLOCKS "&COUNTIF(F2:J2,"Y")&" AI TRAINING CRAWLER(S)","NO RESERVATION WANTED: AI TRAINING CRAWLERS ALLOWED"),IF(C2<>"Y","CHECK BY HAND: ASK THE CLIENT WHETHER TO RESERVE TEXT AND DATA MINING RIGHTS (COLUMN C)",IF(D2<>"Y","MISSING: NO ROBOTS.TXT - IF /ROBOTS.TXT GIVES A 4XX ERROR, CRAWLERS MAY ACCESS ANY RESOURCE (RFC 9309 SEC 2.3.1.3)",IF(COUNTIF(F2:J2,"Y")=5,"OK: ALL 5 LISTED AI TRAINING CRAWLER TOKENS DISALLOWED (DIRECTIVE 2019/790 ART 4(3), RECITAL 18)","FIX: NOT DISALLOWED YET -"&IF(F2<>"Y"," GPTBot","")&IF(G2<>"Y"," Google-Extended","")&IF(H2<>"Y"," ClaudeBot","")&IF(I2<>"Y"," Applebot-Extended","")&IF(J2<>"Y"," CCBot","")&" (ART 4(3), RECITAL 18)")))))
Q2: =IF(C2="Y",IF(M2="Y","OK: WEBSITE TERMS STATE THE RESERVATION (RECITAL 18)","FIX: ADD AN EXPRESS TEXT AND DATA MINING RESERVATION TO THE WEBSITE TERMS (ART 4(3), RECITAL 18)"),"")
R2: =IF(A2="","",IF(AND(K2="Y",L2="Y"),"CHECK: AI SEARCH BOTS BLOCKED BUT THE CLIENT WANTS TO APPEAR IN AI ANSWERS - THE OPERATORS DOCUMENT THESE BOTS SEPARATELY FROM TRAINING",IF(AND(K2="N",L2="N"),"NOTE: AI SEARCH BOTS STILL ALLOWED - BLOCK THEM TOO IF THE CLIENT WANTS NO AI ANSWERS","")))
S2: =IF(C2<>"Y","",IF(ISNUMBER(E2),TEXT(E2+Settings!$B$4,"YYYY-MM-DD")&IF(Settings!$B$1>E2+Settings!$B$4," (OVERDUE)",""),"NO CHECK DATE LOGGED"))
T2: =IF(A2="","",TEXTJOIN("; ",TRUE,IF(LEFT(P2,7)="MISSING","Publish a robots.txt at /robots.txt with a Disallow group for each AI training crawler token",""),IF(LEFT(P2,3)="FIX","Add User-agent + Disallow: / groups for the tokens named in column P",""),IF(LEFT(Q2,3)="FIX","Add the reservation sentence to the website terms",""),IF(LEFT(P2,6)="CHECK:","Ask the client; remove the Disallow groups or change column C",""),IF(LEFT(R2,5)="CHECK","Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot, or confirm the client accepts not appearing in AI answers",""),IF(OR(RIGHT(S2,9)="(OVERDUE)",S2="NO CHECK DATE LOGGED"),"Re-open /robots.txt, re-tick F to K and log today in column E",""),IF(AND(N2="Y",C2="Y"),"Check the AI licence: keep the licensee's crawler allowed if the contract needs it",""),IF(C2="Y","Keep a dated copy of robots.txt and the terms as evidence of the reservation",""),IF(LEFT(P2,6)="CHECK:","",IF(C2="Y","",IF(C2="N","No reservation: note the client decision in column O","Ask the client: reserve text and data mining rights or not")))))
U2: =IF(A2="","",IF(AND(C2<>"Y",C2<>"N"),"CHECK BY HAND",IF(C2="N",IF(LEFT(P2,5)="CHECK","CHECK BY HAND","NO RESERVATION"),IF(OR(LEFT(P2,7)="MISSING",LEFT(P2,3)="FIX",LEFT(Q2,3)="FIX"),"FIX",IF(LEFT(R2,5)="CHECK","CHECK BY HAND",IF(OR(RIGHT(S2,9)="(OVERDUE)",S2="NO CHECK DATE LOGGED"),"RECHECK","OK"))))))

What we tested

Fourteen made-up websites in LibreOffice (xlsx → ods → csv), with today fixed at 9 Nov 2026: all five tokens disallowed and terms set (OK); no robots.txt (FIX); three of five tokens (FIX, naming Google-Extended and Applebot-Extended); robots.txt fine but terms silent (FIX); last check in June (RECHECK, overdue); no check date (RECHECK); client wants no reservation and nothing blocked (NO RESERVATION); client wants no reservation but GPTBot blocked (CHECK BY HAND); client undecided (CHECK BY HAND); search bots blocked although the client wants AI search answers (CHECK BY HAND); a check exactly 90 days ago (OK, due today); content licensed to one AI firm (OK with a licence note); column C blank (CHECK BY HAND); and a robots.txt with no AI groups (FIX, naming all five). 14/14 gave the intended result, the empty row stayed blank, and no cell showed an error.

Client checklist

Client email

Subject: Your website and AI training: decision needed

Hello,

EU copyright law lets anyone copy publicly available website content
for text and data mining, including AI training, unless the owner
reserves that use in a machine-readable way (Directive (EU) 2019/790,
Article 4(3)). Since August 2025, AI model providers selling in the EU
must have a policy to respect such reservations (AI Act, Article 53).

We checked [website]:
- robots.txt: [ ] of 5 AI training crawlers blocked
- website terms: [reservation present / missing]
- AI search bots: [allowed / blocked]

Please tell us by [date] whether you want your content kept out of AI
training. If yes, we will update robots.txt and the terms and keep a
dated copy for your records.

Best regards,
[Name]

Limits and our reading

FAQ

Can a business stop AI firms from training on its website under EU law?

Directive (EU) 2019/790, Article 4(3) lets the rightholder expressly reserve text and data mining 'in an appropriate manner, such as machine-readable means in the case of content made publicly available online'. Without a reservation, the general text and data mining exception applies to lawfully accessible content. A reservation does not affect text and data mining for scientific research under Article 3 (Article 4(4)).

Is robots.txt a valid reservation?

The Directive does not name robots.txt. Recital 18 says that for online content the reservation should be made 'by the use of machine-readable means, including metadata and terms and conditions of a website or a service'. Our reading is that a robots.txt group for each AI crawler is a machine-readable means; we pair it with a sentence in the website terms and flag this as our reading.

What does the AI Act add?

Article 53(1)(c) requires providers of general-purpose AI models to put in place a policy to comply with Union copyright law and to 'identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790'. Chapter V applies from 2 August 2025 (Article 113(b)); models placed on the market before that date must comply by 2 August 2027 (Article 111(3)).

Which crawler names does the sheet check?

Five tokens that the operators themselves describe as controlling AI training use: GPTBot (OpenAI), Google-Extended (Google), ClaudeBot (Anthropic), Applebot-Extended (Apple) and CCBot (Common Crawl). AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are checked in a separate column because their operators document them separately from training.

Will blocking Google-Extended remove the site from Google Search?

Google's crawler page says Google-Extended 'does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search'. Do not block Googlebot itself unless the client wants to leave Google Search.

Does robots.txt physically stop crawlers?

No. RFC 9309 says 'These rules are not a form of access authorization.' It is a signal that compliant crawlers follow and a dated record of the reservation. Keep dated copies of robots.txt and the website terms.

Sources (read 9 Nov 2026)

Related

Keep each client's AI tools and their suppliers in one register: the EURO AI Register has a free edition.

← All guides

Unadhi Guides are free, independent explainers. We correct errors as soon as we find them: write to mininglife7@gmail.com.