EU text and data mining opt-out check sheet for client websites: robots.txt groups for GPTBot, Google-Extended, ClaudeBot, Applebot-Extended and CCBot plus a website terms sentence, under Directive (EU) 2019/790 Article 4(3) and AI Act Article 53(1)(c), for accountants, web agencies and SMEs
Partner programme opening soon: join the free waitlist. Planned terms are 50% of each referred sale (75% once you have 5+ paying customers), single level, paid through Gumroad only from revenue received.
What you get: a finished text and data mining opt-out check sheet, one row per website (your firm's own site and each client's site). You record whether the client wants to keep its texts and images out of AI training, what /robots.txt says for five AI training crawler tokens (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot), whether the AI search bots are blocked, and whether the website terms carry a reservation. The formulas return a robots.txt status that names each crawler not yet disallowed, a terms status, an AI search visibility warning, the next re-check date, a to-do list and a result (OK, FIX, RECHECK, CHECK BY HAND or NO RESERVATION). A third tab holds a ready robots.txt block. Also below: a terms sentence, a client email and a checklist. We tested the formulas in LibreOffice on fourteen made-up websites before publishing.
Short answer: In the EU, anyone may copy lawfully accessible content for text and data mining unless the rightholder has expressly reserved that use "in an appropriate manner, such as machine-readable means in the case of content made publicly available online" (Directive 2019/790, Article 4(3)). For online content, recital 18 says the reservation should be made "by the use of machine-readable means, including metadata and terms and conditions of a website or a service". Since 2 August 2025, providers of general-purpose AI models placed on the EU market must have a policy to "identify and comply with, including through state-of-the-art technologies" such reservations (AI Act Article 53(1)(c); Article 113(b)). Models placed on the market before that date must comply by 2 August 2027, 266 days from 9 November 2026 (Article 111(3)).
Why accountants, tax advisers and web agencies should look at this now
- Clients ask "can we stop AI firms using our website?" The EU answer is a reservation under Article 4(3). Without one, the general text and data mining exception applies to lawfully accessible content.
- The reservation has to be readable by machines for online content (recital 18). A sentence buried in a PDF brochure is weak evidence. A robots.txt group plus a clear sentence in the website terms is a simple, dated way to show it.
- AI model providers must now look for it. Article 53(1)(c) makes respecting Article 4(3) reservations part of their legal duties, and recital 106 says this applies "regardless of the jurisdiction in which the copyright-relevant acts underpinning the training" take place.
- Getting it wrong costs visibility. Blocking the wrong bot can take a client out of AI search answers. The operators document their training crawlers and their search bots separately, so the sheet checks both.
- It is a short, billable job for any firm that already looks after client websites or content.
Step 1: decide, client by client
- Ask each client: do you want your website texts, photos and other content kept out of AI model training? Record
Y,NorUndecidedin column C. This is the client's decision as rightholder, not yours. - Ask whether any content is licensed to an AI company by contract (column N). Recital 18 says that in other cases rights can also be reserved "by other means, such as contractual agreements or a unilateral declaration".
- Ask whether the client wants to appear in AI search answers (column L).
Step 2: check /robots.txt
- Open
https://client-site.example/robots.txt. The rules must be in a file named "/robots.txt" at the top level of the site (RFC 9309, Section 2.3). If the address returns a 4xx error, the file counts as unavailable and "the crawler MAY access any resources on the server" (Section 2.3.1.3). EnterYin column D only if the file loads. - For each token below, tick
Yif there is a group with thatUser-agentline andDisallow: /(orUser-agent: *withDisallow: /, which blocks every crawler, search engines included). Crawlers must match the user-agent name case-insensitively (Section 2.2.1). - Log the date in column E. Crawlers may cache robots.txt but "SHOULD NOT use the cached version for more than 24 hours" (Section 2.4). The sheet asks you to re-check every 90 days (Settings!B4, our choice), because sites get rebuilt and robots.txt files get overwritten.
| Token | What the operator's own page says (read 9 Nov 2026) |
|---|---|
GPTBot (OpenAI) | Disallowing GPTBot indicates "that crawled content should not be used for training OpenAI's generative AI foundation models". OAI-SearchBot is a separate setting for search results. |
Google-Extended (Google) | A "standalone product token" to manage whether crawled content "may be used for training future generations of Gemini models" and for grounding. It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". |
ClaudeBot (Anthropic) | Restricting ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets". Claude-SearchBot and Claude-User are separate. |
Applebot-Extended (Apple) | Publishers "can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended". |
CCBot (Common Crawl) | "To prevent Common Crawl from crawling your website", add User-agent: CCBot and Disallow: /. |
PerplexityBot (Perplexity) | Surfaces sites in Perplexity search results; "It is not used to crawl content for AI foundation models." Counted in column K, not as a training crawler. |
RFC 9309 also says: "These rules are not a form of access authorization." A crawler that ignores robots.txt is not stopped by it. The file is a signal and a dated record, not a lock.
Step 3: the robots.txt block and the terms sentence
# AI training crawlers: text and data mining reserved (Directive (EU) 2019/790, Art 4(3)) User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: /
Add it to the existing file; do not replace the groups for normal search engines. Sentence for the website terms (adapt with the client's lawyer if needed):
Text and data mining: [Company] expressly reserves the rights to make reproductions and extractions of the content of this website for text and data mining, including for training AI models, under Article 4(3) of Directive (EU) 2019/790. This does not affect text and data mining for scientific research under Article 3 of that Directive.
The research carve-out is in the law: a reservation does not affect Article 3 text and data mining by research organisations and cultural heritage institutions (Article 4(4)), and contract terms contrary to Article 3 are unenforceable (Article 7(1)).
The sheet: columns
Tab Sites (one row per website), tab Settings (B1 today, use =TODAY(); B2 = 2027-08-02; B3 = days left; B4 = 90-day re-check; B5 = 2026-08-02) and tab Robots block.
| Col | What to enter or what it shows | Source |
|---|---|---|
| A–B | Client, website URL | You |
| C | Client wants to reserve text and data mining / AI training rights? Y/N/Undecided | Art 4(3) |
| D | /robots.txt answers with HTTP 200? Y/N | RFC 9309 Sec 2.3 |
| E | robots.txt checked on (date) | You |
| F–J | GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot disallowed? Y/N | Operators' pages |
| K | AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) blocked? Y/N | Operators' pages |
| L | Client wants to appear in AI search answers? Y/N | You |
| M | Website terms state a text and data mining reservation? Y/N | Recital 18 |
| N | Content licensed to an AI company by contract? Y/N | Recital 18 |
| O | Notes | You |
| P | Robots.txt status, naming each token not yet disallowed | Art 4(3); recital 18; RFC 9309 |
| Q | Terms status | Recital 18 |
| R | AI search visibility warning or note | Operators' pages |
| S | Next robots.txt check (E + 90 days), or OVERDUE | Settings |
| T | To-do list | As cited |
| U | Result: OK, FIX, RECHECK, CHECK BY HAND, NO RESERVATION | Summary |
Formulas (row 2; fill down). In Excel files saved by some tools TEXTJOIN shows as _xlfn.TEXTJOIN; that is the same function.
P2: =IF(A2="","",IF(C2="N",IF(COUNTIF(F2:J2,"Y")>0,"CHECK: CLIENT DOES NOT WANT A RESERVATION BUT ROBOTS.TXT BLOCKS "&COUNTIF(F2:J2,"Y")&" AI TRAINING CRAWLER(S)","NO RESERVATION WANTED: AI TRAINING CRAWLERS ALLOWED"),IF(C2<>"Y","CHECK BY HAND: ASK THE CLIENT WHETHER TO RESERVE TEXT AND DATA MINING RIGHTS (COLUMN C)",IF(D2<>"Y","MISSING: NO ROBOTS.TXT - IF /ROBOTS.TXT GIVES A 4XX ERROR, CRAWLERS MAY ACCESS ANY RESOURCE (RFC 9309 SEC 2.3.1.3)",IF(COUNTIF(F2:J2,"Y")=5,"OK: ALL 5 LISTED AI TRAINING CRAWLER TOKENS DISALLOWED (DIRECTIVE 2019/790 ART 4(3), RECITAL 18)","FIX: NOT DISALLOWED YET -"&IF(F2<>"Y"," GPTBot","")&IF(G2<>"Y"," Google-Extended","")&IF(H2<>"Y"," ClaudeBot","")&IF(I2<>"Y"," Applebot-Extended","")&IF(J2<>"Y"," CCBot","")&" (ART 4(3), RECITAL 18)")))))
Q2: =IF(C2="Y",IF(M2="Y","OK: WEBSITE TERMS STATE THE RESERVATION (RECITAL 18)","FIX: ADD AN EXPRESS TEXT AND DATA MINING RESERVATION TO THE WEBSITE TERMS (ART 4(3), RECITAL 18)"),"")
R2: =IF(A2="","",IF(AND(K2="Y",L2="Y"),"CHECK: AI SEARCH BOTS BLOCKED BUT THE CLIENT WANTS TO APPEAR IN AI ANSWERS - THE OPERATORS DOCUMENT THESE BOTS SEPARATELY FROM TRAINING",IF(AND(K2="N",L2="N"),"NOTE: AI SEARCH BOTS STILL ALLOWED - BLOCK THEM TOO IF THE CLIENT WANTS NO AI ANSWERS","")))
S2: =IF(C2<>"Y","",IF(ISNUMBER(E2),TEXT(E2+Settings!$B$4,"YYYY-MM-DD")&IF(Settings!$B$1>E2+Settings!$B$4," (OVERDUE)",""),"NO CHECK DATE LOGGED"))
T2: =IF(A2="","",TEXTJOIN("; ",TRUE,IF(LEFT(P2,7)="MISSING","Publish a robots.txt at /robots.txt with a Disallow group for each AI training crawler token",""),IF(LEFT(P2,3)="FIX","Add User-agent + Disallow: / groups for the tokens named in column P",""),IF(LEFT(Q2,3)="FIX","Add the reservation sentence to the website terms",""),IF(LEFT(P2,6)="CHECK:","Ask the client; remove the Disallow groups or change column C",""),IF(LEFT(R2,5)="CHECK","Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot, or confirm the client accepts not appearing in AI answers",""),IF(OR(RIGHT(S2,9)="(OVERDUE)",S2="NO CHECK DATE LOGGED"),"Re-open /robots.txt, re-tick F to K and log today in column E",""),IF(AND(N2="Y",C2="Y"),"Check the AI licence: keep the licensee's crawler allowed if the contract needs it",""),IF(C2="Y","Keep a dated copy of robots.txt and the terms as evidence of the reservation",""),IF(LEFT(P2,6)="CHECK:","",IF(C2="Y","",IF(C2="N","No reservation: note the client decision in column O","Ask the client: reserve text and data mining rights or not")))))
U2: =IF(A2="","",IF(AND(C2<>"Y",C2<>"N"),"CHECK BY HAND",IF(C2="N",IF(LEFT(P2,5)="CHECK","CHECK BY HAND","NO RESERVATION"),IF(OR(LEFT(P2,7)="MISSING",LEFT(P2,3)="FIX",LEFT(Q2,3)="FIX"),"FIX",IF(LEFT(R2,5)="CHECK","CHECK BY HAND",IF(OR(RIGHT(S2,9)="(OVERDUE)",S2="NO CHECK DATE LOGGED"),"RECHECK","OK"))))))
What we tested
Fourteen made-up websites in LibreOffice (xlsx → ods → csv), with today fixed at 9 Nov 2026: all five tokens disallowed and terms set (OK); no robots.txt (FIX); three of five tokens (FIX, naming Google-Extended and Applebot-Extended); robots.txt fine but terms silent (FIX); last check in June (RECHECK, overdue); no check date (RECHECK); client wants no reservation and nothing blocked (NO RESERVATION); client wants no reservation but GPTBot blocked (CHECK BY HAND); client undecided (CHECK BY HAND); search bots blocked although the client wants AI search answers (CHECK BY HAND); a check exactly 90 days ago (OK, due today); content licensed to one AI firm (OK with a licence note); column C blank (CHECK BY HAND); and a robots.txt with no AI groups (FIX, naming all five). 14/14 gave the intended result, the empty row stayed blank, and no cell showed an error.
Client checklist
- Decide: keep website content out of AI training, yes or no.
- Check /robots.txt loads at the top level of the site.
- Add a
Disallow: /group for GPTBot, Google-Extended, ClaudeBot, Applebot-Extended and CCBot. - Leave Googlebot and other search engine groups as they are; Google-Extended does not affect Google Search.
- Decide separately about OAI-SearchBot, Claude-SearchBot and PerplexityBot (AI search answers).
- Add the reservation sentence to the website terms.
- Keep dated copies of robots.txt and the terms.
- Re-check after every website rebuild and at least every 90 days.
Client email
Subject: Your website and AI training: decision needed Hello, EU copyright law lets anyone copy publicly available website content for text and data mining, including AI training, unless the owner reserves that use in a machine-readable way (Directive (EU) 2019/790, Article 4(3)). Since August 2025, AI model providers selling in the EU must have a policy to respect such reservations (AI Act, Article 53). We checked [website]: - robots.txt: [ ] of 5 AI training crawlers blocked - website terms: [reservation present / missing] - AI search bots: [allowed / blocked] Please tell us by [date] whether you want your content kept out of AI training. If yes, we will update robots.txt and the terms and keep a dated copy for your records. Best regards, [Name]
Limits and our reading
- National law decides the details. Article 4 is in a directive; each Member State's copyright law applies it. We did not read any national law or court ruling.
- Our reading, flagged: treating a robots.txt group per AI crawler as a "machine-readable means" under Article 4(3) and recital 18 (the Directive does not name robots.txt); treating the website terms as a second, human-readable layer; the 90-day re-check interval; and the choice of the five training tokens, which come from the operators' own pages, not from EU law. Other AI developers use other crawler names; add columns if you need them.
- A reservation looks forward. Anthropic's page speaks of "future materials". Nothing in robots.txt removes content already collected.
- Not covered: the scientific research exception in detail (Article 3), databases and software rights, the training-content summary under Article 53(1)(d), and the AI Office template and code of practice for model providers.
FAQ
Can a business stop AI firms from training on its website under EU law?
Directive (EU) 2019/790, Article 4(3) lets the rightholder expressly reserve text and data mining 'in an appropriate manner, such as machine-readable means in the case of content made publicly available online'. Without a reservation, the general text and data mining exception applies to lawfully accessible content. A reservation does not affect text and data mining for scientific research under Article 3 (Article 4(4)).
Is robots.txt a valid reservation?
The Directive does not name robots.txt. Recital 18 says that for online content the reservation should be made 'by the use of machine-readable means, including metadata and terms and conditions of a website or a service'. Our reading is that a robots.txt group for each AI crawler is a machine-readable means; we pair it with a sentence in the website terms and flag this as our reading.
What does the AI Act add?
Article 53(1)(c) requires providers of general-purpose AI models to put in place a policy to comply with Union copyright law and to 'identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790'. Chapter V applies from 2 August 2025 (Article 113(b)); models placed on the market before that date must comply by 2 August 2027 (Article 111(3)).
Which crawler names does the sheet check?
Five tokens that the operators themselves describe as controlling AI training use: GPTBot (OpenAI), Google-Extended (Google), ClaudeBot (Anthropic), Applebot-Extended (Apple) and CCBot (Common Crawl). AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are checked in a separate column because their operators document them separately from training.
Will blocking Google-Extended remove the site from Google Search?
Google's crawler page says Google-Extended 'does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search'. Do not block Googlebot itself unless the client wants to leave Google Search.
Does robots.txt physically stop crawlers?
No. RFC 9309 says 'These rules are not a form of access authorization.' It is a signal that compliant crawlers follow and a dated record of the reservation. Keep dated copies of robots.txt and the website terms.
Sources (read 9 Nov 2026)
- Directive (EU) 2019/790 of 17 April 2019 on copyright and related rights in the Digital Single Market, OJ L 130, 17.5.2019, p. 92: Article 2(2), Article 3, Article 4(1)–(4), Article 7(1), recital 18. Official text read from the EU Publications Office repository (Cellar, CELEX 32019L0790, English XHTML) because EUR-Lex returned an empty response.
- Regulation (EU) 2024/1689 (AI Act), OJ L, 2024/1689, 12.7.2024: Article 53(1)(c)–(d) and (4), Article 111(3), Article 113, recitals 104–108. Read from Cellar (CELEX 32024R1689, English XHTML).
- RFC 9309, Robots Exclusion Protocol (September 2022), Sections 1, 2.2.1, 2.3, 2.3.1.3, 2.4: rfc-editor.org.
- Crawler pages of OpenAI (developers.openai.com/api/docs/bots), Google (developers.google.com, Google common crawlers), Anthropic (support.claude.com article 8896518), Apple (support.apple.com/en-us/119829), Common Crawl (commoncrawl.org/ccbot) and Perplexity (docs.perplexity.ai, Perplexity crawlers).
Related
- AI Act Article 53: obligations of general-purpose AI model providers
- AI Act Article 50(2) machine-readable marking check sheet (2 Dec 2026)
- AI-generated marketing content label sheet
- Website legal notice (imprint) check sheet for client websites
- GDPR Article 13 privacy notice: AI section wording sheet
Keep each client's AI tools and their suppliers in one register: the EURO AI Register has a free edition.
Unadhi Guides are free, independent explainers. We correct errors as soon as we find them: write to mininglife7@gmail.com.