Website knowledge sheet for an AI chat assistant: check old prices, offers and short pages before your small business chatbot learns from your site

Updated 24 Oct 2026 · Disclosure: the website crawler linked below is an Unadhi product on the Apify Store. This guide is free, and the same sheet works with a crawl export from other tools. Current Apify pricing is only on the Store page. This page has no affiliate links and nobody paid to be mentioned.

The finished sheet

check tab (example rows are made up; old years setting: 2010 to 2024)

The Diwali 2024 offer page is the usual trap: it is still on the site, the crawler finds it, and without this check the assistant can quote last year's discount as if it were current.

Who this is for

Small businesses (salons, clinics, coaching centres, shops, accountants, trades) that are adding an AI chat assistant to their website or messaging channel and want it to answer from their own pages, and the freelancers and small agencies who set these up. Most chat tools let you upload files as knowledge. What you upload is what the assistant repeats, so a sheet that shows every page and flags the risky ones is quicker than reading the site page by page.

Step 1: crawl the site into a tab named crawl

With the website crawler

Open the Website Content Crawler (Half Price) on the Apify Store and use this input with your site's address:

{ "startUrls": [{ "url": "https://your-site.example/" }],
  "maxPagesPerSite": 300, "maxDepth": 5, "useSitemap": true,
  "minWords": 0, "outputMarkdown": true }

• outputMarkdown: keep it on. The markdown column is the page's main text with headings, lists and tables kept, and menus, footers and cookie banners removed. That is the text the assistant will learn from.
• maxPagesPerSite: set it above the number of pages on the site (the default is 50).
• minWords: set it to 0 so short pages such as Contact still appear; the sheet flags them instead of dropping them.
• useSitemap adds pages listed in your sitemap, including old offer pages that are no longer linked from the menu. Those are the ones you most need to see.

The crawler follows robots.txt. Run it on your own site or a client's site that you look after.

In the run, export the dataset as CSV. In your Google Sheet use File → Import → Upload, "Insert new sheet", and name the tab crawl. Do not rename the headers.

What the columns contain

• url, title, description, language, canonicalUrl, headings: the page address and its basic details.
• wordCount: words in the main content.
• markdown and text: the main content as Markdown and as plain text.
• httpStatus, depth and fetchedAt: the server response, how many clicks from the start page, and when the page was read.
• Pages whose main text exactly matches a page already delivered are skipped, so copies do not appear twice.

A Google Sheets cell holds at most 50,000 characters. A very long page (a big price list or terms page) can be cut short in the sheet; the sheet flags those so you can take their text from the JSON export instead.

Free routes

For a small site you can copy each page's text into a document by hand. Free desktop crawlers such as the free version of Screaming Frog SEO Spider (up to 500 addresses) list your pages; paste a crawl tab with the headers url, title, wordCount, httpStatus and markdown (page text) and the formulas below work unchanged.

Step 2: the settings tab

Add a tab named settings. In B1 type the pattern for years you treat as old: 20(1[0-9]|2[0-4]) matches 2010 to 2024. In B2 type 80 (fewer words than this counts as a short page). Next year, change 2[0-4] to 2[0-5].

Step 3: build the check tab

Add a tab named check. Row 1 holds the headers below; row 2 holds the formulas, which you copy down one row per row of crawl.

Notes on the formulas:

• Old year catches "2024" anywhere in the text, including a copyright line or "since 2012". Read the page before marking it No; often only one sentence needs changing on the website.
• Price only looks for a currency sign or code followed by a number. It is a prompt to check that the price on the page is today's price, not a judgement that the page is wrong.
• Status other than 200 means the page did not load normally when it was read; check it in a browser.

Step 4: the knowledge tab and the upload

1. Add a tab named knowledge and put =FILTER(check!J2:J, check!J2:J<>"") in A1. Each cell is one page: its title, its address and its text.
2. Fix the website first where you can (old offers, old prices, old hours), then crawl again, so the site and the assistant say the same thing.
3. Download the knowledge tab (File → Download → Comma-separated values, or Microsoft Excel) and upload it to your chat tool's knowledge or files section if it accepts that format. If it only takes documents, copy column A into a document and save it in a format the tool accepts. Check your tool's own help pages for file types and size limits.
4. Add a short first line of your own, such as your opening hours, booking link and "if a price is not listed here, ask the customer to call", so the assistant has a safe answer when the pages do not cover a question.
5. Re-run the crawl after any price or offer change and replace the upload. Note the date in the sheet.

If you serve customers in the EU

From 2 August 2026, Article 50 of the EU AI Act requires that people are told when they are talking to an AI system, unless it is obvious. Our chatbot disclosure checklist covers the wording.

Check before you act on it

• The number of rows is close to the number of pages you expect. If it is much lower, raise maxPagesPerSite or maxDepth.
• Open three Yes pages and compare the Text column with the live page: the main content should be there without menus or footers.
• Ask the assistant five real customer questions (price, hours, location, a service you no longer offer, a refund) and check each answer against the site before you switch it on for customers.

Partner programme opening soon

Partner programme opening soon: join the free waitlist. Planned terms are 50% of each referred sale (75% once you have 5+ paying customers), single level, paid through Gumroad only from revenue received.

Sources

• Website crawler inputs and outputs (startUrls, maxPagesPerSite default 50, maxDepth, useSitemap, minWords default 20, outputMarkdown; per-page url, site, depth, title, description, language, canonicalUrl, headings, wordCount, markdown, text, contentHash, httpStatus, fetchedAt; duplicate pages skipped; robots.txt followed): Apify Store listing.
• Google Sheets limits (50,000 characters per cell) and REGEXMATCH, TEXTJOIN, FILTER and INDEX/MATCH functions: Google Docs Editors Help.
• EU AI Act Article 50 transparency obligations and their application date: Regulation (EU) 2024/1689 on EUR-Lex.

Related

• Page title and meta description check sheet
• EU AI Act Article 50 chatbot disclosure checklist
• Web agency client website contact check sheet

← All guides

Unadhi Guides are free, independent explainers. We correct errors as soon as we find them: write to mininglife7@gmail.com.