About CernagoScanBot
CernagoScanBot is an automated crawler operated by Cernago Ltd. It collects publicly available technical information from company websites for a published research study on AI visibility in the UK professional AV sector.
If you have found this page because you saw our crawler in your server logs, everything you need is below.
What is the user agent?
CernagoScanBot/1.0 (+https://cernago.com/scanner)Requests also carry a From header with a monitored address, so you can identify us regardless of which header your logging captures.
What does it fetch?
Six requests to your root domain, plus at most one further page:
- Your homepage
/robots.txt/llms.txt/sitemap.xmland/sitemap_index.xml- Up to five child sitemaps, if your sitemap is an index
- One recent article or news page, selected from your sitemap
We do not log in, we do not submit forms, we do not fetch images or media, and we do not attempt to access anything behind authentication.
How often does it run?
Once per quarter. The same list of companies is rescanned each wave, so that change over time is measurable rather than an artefact of who happens to be in the sample.
How polite is it?
At least one second between requests to the same domain, a maximum of ten concurrent connections across all domains, and we honour Retry-After headers. A full scan of your site takes under a minute and amounts to seven requests.
Does it respect robots.txt?
Yes. We read your robots.txt first and follow it. If you disallow us, we record that and stop.
How do I block it?
Add this to your robots.txt:
User-agent: CernagoScanBot
Disallow: /We check this before every scan, so it takes effect from your next wave.
If you would rather be removed from the study entirely, email tech@cernago.com and we will exclude your domain permanently.
Why are you doing this?
AI assistants increasingly answer buyers' questions instead of returning a list of links. Whether a company can be read and quoted by those systems now depends on technical signals most businesses have never audited. Nobody has published what those signals look like across the UK professional AV sector, so we are measuring it.
What do you publish?
Aggregate figures only, by sector segment. No company is ever named in any published output. We report concentration rather than identity, for example "the three most-named firms account for a given share of all mentions", never who they are.
What do you store?
The raw responses from the requests listed above, held privately and used only to compute the metrics in our published methodology. We do not republish, sell, or share the content of anyone's website.
Do you collect personal data?
Our measurements are about websites, not people. However, some websites publish personal information in their markup, for example a named contact in structured data, and our stored responses may therefore contain personal data we did not seek.
Where we hold personal data:
- Controller
- Company Number 13377499, 75 Normanhurst Avenue, Bournemouth, England, BH8 9NW, ICO registration ZB234376
- What we hold
- Names, business email addresses and job titles, where published on your own website or in public business directories
- Where it came from
- Your website, and publicly available trade and business directories
- Why we hold it
- Market research, and business communications about our services
- How long we keep it
- Raw crawl data is retained for as long as it is needed to produce and verify the published study, and to allow comparison across waves. Contact information is kept until you ask us to remove it.
- Lawful basis
- Legitimate interests. Our assessment of that balance is available on request
- Your rights
- Access, rectification, erasure, restriction, and an absolute right to object to direct marketing. Email tech@cernago.com and we will act on it
- Complaints
- You may complain to the Information Commissioner's Office at ico.org.uk
Who runs this?
Cernago Ltd, Company Number 13377499, 75 Normanhurst Avenue, Bournemouth, England, BH8 9NW. Contact tech@cernago.com.
Last updated .
