SEO and Discovery

Babylon publishes three discovery files for your storefront automatically: sitemap.xml, which tells search engines what pages exist; robots.txt, which tells crawlers what they may visit; and llms.txt, a plain-language guide to your site for AI assistants. Settings → SEO & Discovery lets you adjust the last two. This guide also covers the Public catalog browsing switch, which decides whether guests can browse your catalogue at all and therefore what search engines can index.

Before you start

• You need the manage tenant settings permission. The standard Admin role has it. If SEO & Discovery is missing from the Settings menu, ask an administrator on your account to check your role (see Team members and roles).
• The page is available on every plan.
• The three files are served on your live storefront hostname, for example https://auctions.example.com/robots.txt. If you move to a custom domain, they move with it. See Connecting a custom domain.

How the page works

The page has three sections. The first two each end with a live preview (Preview: /llms.txt and Preview: /robots.txt) showing exactly what the file will contain, including edits you have not saved yet. Nothing changes on your storefront until you click Save at the bottom of the page.

llms.txt: AI and LLM crawler guide

llms.txt is a short Markdown file that helps AI assistants understand your site: your name, a one-line description, what you sell, the key pages (home, auctions, sell, about, FAQ, contact) and how buying and selling work. Babylon generates most of it from your account.

• Site description override: replaces the one-line description near the top of the file. Leave it blank to use your storefront description; if you have none, a generic line is used. Up to 500 characters. Write one plain sentence that says who you are and what you sell.
• Custom sections (Markdown): added word for word after the generated content. Use ## headings to add sections about your house, for example authentication tiers, bidding rules or specialist categories. Up to 5,000 characters.

Use the preview to check the result before saving. Remember that anything you add here is public.

robots.txt: search and bot crawl rules

Babylon's generated robots.txt always allows crawlers into your public pages, always blocks private areas such as account, checkout, orders and messaging pages, and ends with a link to your sitemap. You can add to it in two ways.

Block AI/LLM training crawlers

Tick any crawler you want to keep out of your whole site. Each ticked crawler receives Disallow: /. The options are:

• GPTBot (OpenAI)
• ChatGPT-User (OpenAI browsing)
• OAI-SearchBot (OpenAI search)
• CCBot (Common Crawl, used by many LLM pipelines)
• anthropic-ai (Anthropic)
• ClaudeBot (Anthropic)
• PerplexityBot (Perplexity AI)
• Bytespider (ByteDance / TikTok)
• meta-externalagent (Meta AI)
• Amazonbot (Amazon AI / Alexa)

Blocking these does not affect ordinary search engines. Bear in mind that some of them also power AI search and browsing features, so blocking them can stop your lots appearing in those answers. robots.txt is a request, not an enforcement mechanism: well-behaved crawlers follow it, others may not.

Additional robots.txt rules

Additional robots.txt rules is added word for word after the generated rules, so use standard robots.txt syntax. It is clearest to start your rules with a User-agent line saying which crawlers they apply to. For example:

User-agent: *
Crawl-delay: 10
Disallow: /private-collection

Up to 2,000 characters. Check the preview carefully: a stray Disallow: / here can remove your whole storefront from search results.

sitemap.xml

There is nothing to configure. The sitemap.xml section shows the address of your sitemap, for example https://auctions.example.com/sitemap.xml. It is built on request from your published lots and editorial articles, so it is always current. Submit that address in Google Search Console or Bing Webmaster Tools if you use them.

Public catalog browsing

This switch lives on a different page: Settings → Business Settings, on the Experience tab, in the Marketplace Visibility section.

• On (the default): guests can browse your listings, auctions, search, categories and collections without signing in. Search engines can index your catalogue.
• Off: browsing requires a signed-in, verified user. Babylon also adds rules to robots.txt asking crawlers to stay out of the catalogue, auctions, listings and collections pages, so your lots will drop out of search results over time.

Draft and pending listings stay hidden from guests either way. Change the switch and click Save Settings at the bottom of the Business Settings page. Turn it off only if you run a private or members-only sale; for most houses, public browsing is how new bidders find you.

Common questions

I saved my changes but the file looks the same. Search engines and browsers cache these files. Open the file directly in a private window to check, and allow time for crawlers to fetch the new version.

Can I upload my own robots.txt or sitemap? No. The files are generated for you. Use Additional robots.txt rules and Custom sections (Markdown) to add to them. If you need something they cannot express, contact Hammerd support.

Can I block a crawler that is not in the list? Yes, with Additional robots.txt rules, for example User-agent: ExampleBot on one line followed by Disallow: / on the next.

Where do page titles, logos and the tagline come from? From Settings → White Label and Branding. See White label and branding.