HullGaugeBot asks first.
You probably found this page in your server logs. Here is who we are, exactly what we read and keep, and how to stop us with two lines.
Hull Gauge publishes used-boat price statistics. To compute them we observe boats listed for sale on brokers’ and dealers’ own websites, and on portals only where their machine-readable policies allow it.
HullGaugeBot/1.0 (+https://hullgauge.com/bot)
User-agent: HullGaugeBot Disallow: /
bot@hullgauge.com
Every request passes three gates
In this order, before any page is fetched, and again for every redirect we are sent to. Fail either of the first two and we don’t read your site; fail the third and we don’t read that page. A fourth check records what you’ve signalled, as evidence.
Gate 1: Our own refusal list
A short list of sites we never read, whatever their robots rules say today: marketplaces whose terms forbid automated access, sites that name and deny AI crawlers, and anyone who has asked us by email.
Stop
Your site is on it. We never fetch a page, not even robots.txt.
Gate 2: Your AI reservations
We check whether your robots.txt denies your whole site to any of GPTBot, ClaudeBot, Claude-User, anthropic-ai, CCBot, PerplexityBot or Google-Extended. If it does, we treat that as a reservation of your rights and stay out, even though it doesn’t name us.
Stop
Any of those crawlers is denied your whole site.
Gate 3: Your robots.txt
Read before anything else at the start of every visit, never trusted for more than a day, and matched to the published standard (RFC 9309). If you have a group for HullGaugeBot, only that group applies to us; otherwise we follow the rules for all robots (*). No robots.txt at all means no rules. If it answers with a server error or doesn’t answer, we read nothing that visit.
Stop
The path is disallowed for HullGaugeBot, or for * when you have no group for us, or we couldn’t read your robots.txt.
And: We record your Content-Signal
If your robots.txt lets us read your home page and it sends a Content-Signal header, we record it, with the date of our latest check, alongside your robots rules, so what you allowed is on file.
Stop
Recorded as evidence; not yet enforced. To stop us, use robots.txt or email.
Then, politely
At most one request a second to your site, and slower if your robots.txt sets a longer Crawl-delay. If you answer 429 or 503 with a Retry-After, we send nothing more until it has passed; if it asks for more than ten minutes, we leave your site until our next visit. A server error or a timeout gets up to three retries, further apart each time. Conditional requests for pages we’ve seen before, so an unchanged page costs you a single short reply. Our scheduled visit starts at 00:30 UK time each night.
Is it really us?
Anyone can put our name in a user agent. Our crawler doesn’t run from fixed addresses yet, so there are no published ranges to check against: this check reads the user agent, and says what that can and can’t prove.
waiting
Paste a line to check it
We look for the user agent in it. Nothing you paste leaves this page.
Published ranges: none yet. We’ll publish them here, with reverse DNS, once the crawler runs from fixed addresses.
What we read, keep and publish
We read
- Pages listing boats for sale, and their sitemaps
- The facts on those pages: structured data (schema.org) where there is some, otherwise the labelled specification rows
- Your robots.txt, and your home page’s headers when robots.txt lets us read it
We keep
- Builder, model, year, dimensions, hull, engine and hours
- Asking price and currency, each time it changes
- VAT status, listing status, and where the boat lies: country, region and the page’s location line
- The page address, its title, the listing reference and your trading name, to recognise the same boat across sites and over time
- A fingerprint of the page, to tell whether it changed; never the page itself
We never keep or show
- Photographs
- Descriptions
- People’s names, phone numbers or email addresses: there is no field for a person’s name or contact details, and phone numbers and email addresses are stripped from the titles, location lines and references we keep before they are saved
- A median, range or other price statistic from fewer than five boats
Stopping us
Everything
User-agent: HullGaugeBot Disallow: /
Your change applies from our next visit: we read robots.txt before any page, and never rely on a copy more than a day old.
Only your listings
User-agent: HullGaugeBot Disallow: /boats-for-sale/
Use your own listings path. We come only for listings, so this stops us reading any (we may still read your sitemaps). Once you have a HullGaugeBot group, only that group applies to us, so copy into it any other Disallow lines you want us to follow.
AI crawlers, us included
User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot Disallow: /
Denying your whole site to any of the AI crawlers listed in gate 2 also stops us, without naming us.
Or email bot@hullgauge.com from your site’s domain, or use the opt-out form. We confirm within five working days and remove your site’s rows from all future figures. Brokers: what opting out means for your listings.
Our basis, and our record
Draft for review: this wording has not yet been checked by a solicitor.
We read facts, not creative work. We publish statistics, and individual boats only as derived facts such as model, year, asking price and month, never the listing. We treat a denial of your whole site to any of the AI crawlers listed above as a reservation of rights under Article 4 of the EU Digital Single Market Directive, even when our own name isn’t on it, and we record your robots rules and any Content-Signal header, with the date of our latest check, as evidence of what you allowed.
- Version 1.0 published: three gates, AI reservations honoured, Content-Signal recorded, Crawl-delay and Retry-After honoured, every redirect checked again; the same rules for every request we make, including to the web archive.
- NextAny change to what we read or keep is dated here before it takes effect.