How AISearch Global runs a technical-setup scan
Three public files per organisation, the way any visitor or search engine can: robots.txt, the homepage, and the llms.txt address. Nothing else. We do not log in, fill in forms, or look past the homepage.
How do we behave while reading?
One organisation at a time, with pauses between requests. We identify ourselves with a user-agent line that says what the scan is for and gives a contact address, and an organisation can ask us not to scan it again by emailing that address. (The scans reported in the November 2026 pilots used an earlier line without the contact address; later scans carry it.) We respect robots.txt. If a site refuses us, blocks us or shows a check page, we record the homepage as unread and move on. We do not disguise the scanner or try to get past a block.
How do we protect the organisations?
No organisation is named, and none will be. We publish counts only. No group smaller than 10 is shown, and counts under 10 are written as "fewer than 10". We keep private evidence (the list and the raw records) and do not publish either.
What do the terms mean?
- JSON-LD structured data: a block of code in a page that states facts in a fixed format, such as an organisation's name, address and services. Search engines and AI tools can read it directly.
- robots.txt: a public file that asks crawlers which parts of a site to leave alone. Having one does not mean a site blocks anyone.
- llms.txt: a proposed convention, a text file that points AI tools to a site's key pages. It is not a standard, and the page that proposes it presents it as a proposal.
- "Blocks": in our studies this means robots.txt says Disallow for the whole site. We did not test what a server does when a real AI crawler arrives.
What can these scans never tell you?
- They read page code, so structured data added later by JavaScript is invisible to us. Structured-data figures are a minimum.
- They cover homepages on one day. Sites change.
- Unread homepages tell us about our scanner, not about the organisation.
- They describe how a site is set up for machines. They are not an audit, and they say nothing about the quality of any organisation, service, clinician or product.
Sources
- llmstxt.org, "The /llms.txt file" (page opened 22 Nov 2026; presents llms.txt as a proposal).
Viveka Mohan Das
Answer Engine Optimisation (AEO) and GEO Strategist, Published Author & Founder, AISearch Global. More about Viveka
Check your own homepage
The free AI Visibility Toolkit reads your homepage the way a machine does and reports what is there.