QueryosityBot
QueryosityBot fetches pages for Queryosity's AI Search Audit. If it appears in your logs, this page says what it requests, how often, and how to block it.
How it identifies itself
Every request QueryosityBot makes carries the user agent QueryosityBot/1.0 (+https://queryosity.xyz/bot), and its name in robots.txt is QueryosityBot.
Its requests come from cloud task runners that do not use a published set of IP addresses, so recognize it by its user agent.
When it visits
QueryosityBot is the fetcher of Queryosity's AI Search Audit. It visits a site after a Queryosity customer has added it as a brand's website, most often the site's own team:
- When the brand is added, to find the site's pages and suggest which to audit.
- When an AI Search Audit runs, at most once every 7 days for each brand, started by hand or on a weekly schedule. Each audit begins by finding the site's pages again.
It does not crawl whole sites, and it does not visit between audits.
What it requests
Every visit starts with robots.txt. After that, QueryosityBot never requests a path your robots.txt disallows, whether the rule is written for QueryosityBot or for all crawlers. In an audit it requests:
- robots.txt, then the sitemaps it lists and
/sitemap.xml, to find pages. - The audited pages, at most 50 on standard plans, each once, as a plain request that runs no scripts: the page as an answer engine that does not run JavaScript sees it.
- The links on those pages that stay on your site, up to 300 per audit, each asked once with a HEAD request, or a GET where HEAD is refused, to find broken links.
- A few requests about the site as a whole: the homepage over http, to see whether it moves to https; the homepage with and without www, to see which one the site uses; and a page that does not exist, whose path starts with
/queryosity-check-, to see whether the site answers 404.
QueryosityBot keeps at most 2 requests to a site in flight and starts them at least 1 second apart.
Pages loaded in a browser for us
Two steps load pages in a full browser, which QueryosityBot does not do itself. Two providers do them for Queryosity, and their requests identify themselves in their own way:
- Finding pages: Firecrawl loads the homepage and the pages it links to, at most two links deep and within 60 seconds, to see links a site builds with JavaScript.
- The audit crawl: DataForSEO's site auditor loads the audited pages with JavaScript on. Its requests carry the user agent
Mozilla/5.0 (compatible; RSiteAuditor).
Neither provider is given a page your robots.txt disallows for QueryosityBot or for all crawlers.
How to block it
To keep QueryosityBot off your site, add this to your robots.txt:
User-agent: QueryosityBot
Disallow: /
From its next visit it reads your robots.txt and nothing else, and none of your pages are passed to the providers above. To keep it out of part of your site, disallow those paths instead. A page it may not request is reported to the customer as blocked, and is not fetched another way.
Contact
Questions about QueryosityBot, or a problem with its requests: write to gaurav@queryosity.xyz.