# /js/* is a separate service — a client-rendered SPA that answers every path
# with the same empty shell (/js/jobs, /js/job/<id>, /js/profile/<slug> and a
# deliberately invented path all return a byte-identical 1662-byte document).
# Nothing there is worth indexing and nothing there server-renders.
#
# It used to be `Disallow: /js`, and that is precisely why /js/jobs/ sat in
# Google as "Indexed, though blocked by robots.txt": the URL arrived from links,
# the content was never once readable, and a disallowed URL cannot be crawled —
# so no noindex, no redirect and no 410 placed on it would ever be seen. A
# directive nobody is allowed to read cannot remove anything from an index.
#
# So the crawl is allowed, and what a crawler finds when it arrives is one of
# two things (both in nginx.conf in seekerpitch-jobseeker-app):
#
#   - A RETIRED PATH ANSWERS 301. The jobseeker self-service product is gone;
#     the SPA is guest-link only. /js/jobs* redirects to /industries, every other
#     dead path to /. This is what moves the ranking those URLs still hold onto a
#     live page, which a noindex could never do — noindex only deletes.
#   - A LIVE GUEST ROUTE (/js/schedule, /js/guest-conversation, /js/survey)
#     serves the shell with `X-Robots-Tag: noindex, nofollow`. Three token URLs
#     nobody should reach from search; the header is all they need.
#
# Assets are untouched, so the app keeps working.
#
# ORDER MATTERS. Those must be live in production BEFORE this file changes.
# Removing the Disallow first means Google starts crawling /js/* and finds a
# shell carrying no directive at all — strictly worse than leaving it blocked.
#
# Do not re-add `Disallow: /js`. It would re-block the crawl, the 301 would stop
# being read, and whatever is in the index today would freeze there.
User-agent: *
Allow: /

User-agent: facebookexternalhit
User-agent: WhatsApp
User-agent: snapchat
User-agent: TelegramBot
User-agent: Twitterbot
User-agent: LinkedInBot
User-agent: Slackbot
User-agent: Discordbot
Allow: /

# AI model TRAINING crawlers.
#
# Jobseeker profiles used to be excluded here on privacy grounds — a profile
# carries name, work history and a video of the person, and agreeing to be
# findable in search is not agreeing to sit in a training corpus, which cannot
# be withdrawn later.
#
# The `Disallow: /js` below STAYS, deliberately, even though the group above
# dropped it. That was an SEO change: search engines have to be let in so the
# 301 gets read and the ranking moves. Training crawlers have nothing to gain —
# a redirect passes them no signal they use, and the shell behind it is empty.
# Nor would a directive help: `X-Robots-Tag: noindex` is a search instruction a
# training crawler is under no obligation to honour. If profiles ever come back,
# they come back excluded from these groups by default, as a privacy decision,
# not an SEO one.
User-agent: GPTBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: anthropic-ai
Allow: /
Disallow: /js

# AI SEARCH & CITATION crawlers — these fetch pages to answer and cite, which is
# a different job from the training crawlers above. Named explicitly so a future
# edit to this file cannot silently cut off AI visibility.
#
# These need the same access the `*` group now has, and for the same reason: an
# answer engine that cannot fetch /js/* never sees the 301 either, and keeps
# citing a dead URL from whatever it saw once. Bingbot is in this group and is a
# search engine outright — blocking it here would have left Bing indefinitely on
# the old URLs. Nothing under /js/ renders server-side, so allowing the fetch
# exposes no profile data; the shell is identical on every path, and the retired
# ones no longer serve it at all.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Applebot-Extended
User-agent: Bingbot
Allow: /

Sitemap: https://seekerpitch.com/sitemap.xml