Get your website AI ready: search, crawlers, and agents

An AI-ready website makes accurate public information easy to find, read, and link to. Start with useful pages and working access. An LLM, or large language model, may answer from older training knowledge, a web search, or a page it retrieves for someone. Those are different routes. Opening your website to a crawler does not guarantee a recommendation in any of them.

This guide is for an existing small-business website, whether it was built by a person, an AI assistant, or a website builder. Building with AI and being readable by AI are separate jobs. Use the website-building guide if you still need the site.

Download the AI-readiness checklist to record the checks, evidence, fixes, and next review. It is a working log, not a visibility score or certificate. Start with your homepage, main service page, About page, and contact route.

Start with five owner checks

1. Read a public page while signed out. Does it show your services, location, and contact details without a login, challenge, or mandatory interaction?
2. Find the facts in the page’s HTML. HTML is the document the website sends. Ask your website supplier to show where the important text exists before scripts run.
3. Review the actual crawl policy. Open your website’s main address with /robots.txt added at the end, then check the hosting or security dashboard for blocks that operate separately.
4. Follow a normal link to each important page. A customer should be able to reach it from your navigation or another relevant page. Submit your sitemap through the search setup guide.
5. Check the business facts an assistant could repeat. Service area, eligibility, currency, hours, and next steps must be explicit and current.

Replace the example domain with yours throughout this guide. Record an unknown result as unknown. A page opening in your browser proves less than a verified crawler receiving its content.

Choose search access, user access, and training separately

A crawler fetches pages automatically. A user agent is the name a requesting program sends to identify itself. Names alone do not prove identity. A scraper extracts information from a page; it might belong to a search provider, an agent helping a customer, or an unrelated operator.

These are current documented distinctions, checked October 4, 2026. Follow the linked provider documentation when changing settings; names and behavior can change.

ChatGPT and OpenAI

OAI-SearchBot supports ChatGPT search discovery. GPTBot collects content that may be used for model training. Their preferences are independent: you can allow search while declining training. ChatGPT-User handles some user-directed visits and does not determine search inclusion; OpenAI says robots rules may not apply to those visits. OpenAI crawler documentation.

Claude and Anthropic

Claude-SearchBot supports search indexing, Claude-User retrieves pages for user requests, and ClaudeBot collects content that could contribute to training. Anthropic documents robots compliance for these bots. Choose the relevant access for each purpose rather than treating ClaudeBot as the only Claude control. Anthropic crawler documentation.

Perplexity

PerplexityBot supports search results; Perplexity says it is not used for foundation-model training. Perplexity-User handles user-requested retrieval, and its documentation says those requests generally ignore robots rules. A robots entry is therefore not an access barrier for every Perplexity request. Perplexity crawler documentation.

Google AI Overviews and AI Mode

For Google’s AI features in Search, Googlebot access, indexing, and eligibility to show a snippet matter. Google-Extended controls training and grounding in some other Google systems; it is not the switch for AI Overviews. Google’s snippet and indexing controls can also restrict how content appears. Google’s AI features guidance.

Google and Bing search setup is still useful. It does not register your business with every LLM, and different products can use different sources. Keep the search dashboards alongside provider-specific checks.

Check robots.txt without replacing your whole policy

robots.txt publishes crawl preferences. It cannot authenticate visitors or protect confidential records. Private customer files, account pages, unpublished quotes, and credentials need actual access controls. Even a disallowed URL can appear in Google when other pages link to it. Google’s robots introduction.

If you want OpenAI search access, a supplier might add this illustrative group after reviewing your existing rules:

User-agent: OAI-SearchBot
Allow: /
Disallow: /go/

Sitemap: https://yourbusiness.example/sitemap.xml

Use your real sitemap address. The /go/ exclusion is relevant to this site’s outbound redirects; your site may have different utility routes. Carry over the exclusions you actually need. This example does not set your training preference or every other provider’s policy.

The easy mistake: a named crawler group does not automatically inherit the wildcard group’s exclusions. Keep necessary rules in the matching group. Repeated groups, path specificity, and an upstream injected policy can change the effective result. Google’s robots specification explains group selection and matching.

Review noindex in the page’s metadata and X-Robots-Tag in its response headers too. These are indexing controls, separate from crawl access. Do not remove a deliberate restriction from a private or staging page merely to satisfy a checklist.

Check hosting and Cloudflare rules as well

An allowed robots rule cannot override a firewall block, a login, or a browser challenge. Cloudflare can prepend managed robots instructions to an existing file. Read the served file, rather than only the copy in your website folder.

If you use Cloudflare, review AI Crawl Control, bot settings, and security events for the public page and relevant crawler. An Allow selection in one feature does not prove the whole request path succeeds. Other hosting platforms need the equivalent check.

Ask your supplier to identify the exact rule causing a legitimate block. Any exception should be narrow, based on the provider’s current verification method, and limited to the access you intend. OpenAI and Perplexity publish address ranges in their crawler documentation. A copied user-agent name is not proof that a request came from either company. Keep protection on customer accounts, forms, and payment routes.

Robots preferences cannot promise that every scraper will obey. Watch for repeated errors, excessive load, and abuse while keeping intended public access available.

Make your content understandable without special tools

Publish the facts a customer needs as text on the page. A service menu that exists only inside a photograph, a PDF, or an interactive widget gives a basic page reader less useful information. Keep the original image or document when helpful, but provide a maintained text explanation beside it.

Use descriptive page titles and headings, ordinary links with destinations, and concise answers to real questions. Google documents crawlable links as anchor elements with an href. A clickable element that relies only on a script is a different thing.

Ask your supplier to check page landmarks, heading order, link purpose, image descriptions, and form labels against W3C’s page structure guidance and forms tutorial. These make the page easier for people and software to interpret. They do not certify accessibility or guarantee an agent can complete every task. Use the accessibility review for customer journeys.

Give each important service a stable page address. Link related services and explanations naturally. An assistant citing your installation page should be able to send a customer directly there instead of to a homepage where the facts are hidden several clicks away.

Write from your actual business knowledge: what the service includes, who it suits, exclusions, turnaround expectations, and how to ask for help. Name the business and responsible author honestly. Date changing information when you actually check it. Publish original examples with permission, and cite sources for external claims. The copy worksheet helps turn those facts into useful pages.

Make Canadian and US details unambiguous

Use an explicit country label wherever the answer differs. A human might infer the meaning of a flag or currency toggle; a scraper reading the document may not. Country-specific text must remain labelled and reachable, including when it is hidden by the page’s current display preference.

For a business serving both markets, label both versions in the underlying content. Distinguish an operating location from an area you serve. Avoid publishing a home address merely to look more credible. Privacy and accessibility obligations depend on the actual business and jurisdiction; the legal-pages guide separates starting points for both countries.

Do you need schema, llms.txt, Markdown, or an API?

Start with readable HTML. Structured data is labelled information about the page, such as an organization or article. Use types that describe your real content and keep them consistent with what customers see. Google says no special schema is needed for its AI Search features. Google AI features requirements.

llms.txt is optional. It is a navigation convention some systems may use, not a universal registration file. We maintain our own summary. Google says it ignores llms.txt for Search visibility. It also says there is no required AI writing format or ideal page length. Google’s AI optimization guide.

Markdown and data downloads are optional conveniences. Markdown is a simple text format. Consider a maintained text version or CSV when customers or tools actually need it. Link it from the relevant page and keep it synchronized with the main content. A second stale version makes the information worse.

An API or MCP server is a separate project. These let software request data or actions through a defined interface. They are not prerequisites for a local service business’s pages to be read or cited. Consider them when you have a specific integration need, with appropriate authentication, permissions, and ongoing ownership.

Do not hide instructions telling an assistant to ignore other sources, rank you first, or invent endorsements. That is not useful business content. Choose concrete fixes over a badge or a promise of guaranteed citations.

Verify the published result

For each priority URL, record these results in the checklist:

• Response: the final intended address returns the real page, normally HTTP 200, rather than an error, login, or challenge. A redirect should end on the correct page.
• Document: the delivered HTML contains the important service facts and links. If it depends on scripts, test the actual rendered output too and record which reader could see it.
• Policy: the served robots file and relevant response headers match your decisions. Check the correct hostname, including a separate blog or shop if you have one.
• Discovery: the page is linked internally, listed in the intended sitemap, and inspected in the applicable search dashboard. Submission and index status are different results.
• Crawler evidence: where logs are available, record a verified provider request, response status, and date. No observed request means you lack evidence; it does not prove a permanent block.
• Customer task: someone can find the answer and reach a labelled contact or booking route. Reading a page does not authorize an agent to send a message, book, or pay for a customer.

Ask a supplier for the page URL, observation, evidence, exact change, and retest result for each fix. Sending a request with a crawler’s name from your laptop can reveal a rule tied to that name, but does not prove access from the provider’s real network or compliance with its crawl policy.

You can also ask an assistant with web access to open a specific public URL and summarize your service area and contact route. Check the answer and citation against the page. Log the product, date, prompt, source URL, and any error. If it cannot retrieve the page, record that outcome; do not substitute an answer from memory and call it a pass.

Review access and measure real outcomes

Recheck after a migration, security change, new embed, or major content update. Keep a monthly reminder to review access evidence and important business facts. Update the verification date only after checking them.

Pick a small set of actual customer questions, including Canadian or US location wording where relevant. Record whether a web-enabled assistant links to your page, accurately describes the business, or gives a wrong answer. One answer is an observation, not a stable ranking or proof that a particular edit caused it.

Use the Google and Bing measurement guide for the reports their platforms currently provide. Compare actual enquiries and outcomes with your follow-up tracker, keeping unknown attribution unknown. A crawl, an indexed page, a citation, a visit, and a paying customer are five different results.

The useful deliverable: clear public pages, deliberate access settings, accurate regional facts, and a record of what you verified. That gives customers and the tools helping them a better route to your content, without pretending you control an LLM’s answer.