Every Host You Own Gets Crawled

A thread on r/webdev this week has a title that is the whole question: Everything on XYZ.com or XYZ.com+app.XYZ.com. One domain for everything, or the apex plus an app. subdomain?

The usual arguments are real. Cookies: a separate host keeps session cookies away from marketing pages and their third-party scripts. Deploys: two hosts can ship on two schedules. CORS: split them and some same-origin calls become cross-origin.

Search rarely makes that list, and it sends its own bill. Every hostname and scheme you expose gets discovered, crawled and reported on, whether you meant it to be public or not, and each needs its own answer for redirects, canonicals, robots and what its root URL returns. Our position: split hosts deliberately, and give every host a policy before a crawler asks.

What search saw on our site

Our canonical host is the apex, metasignaltech.com. www redirects permanently to the apex, http redirects to https, and / redirects to a language prefix such as /en based on the Accept-Language header. App downloads come from download.metasignaltech.com and video (HLS playlists and segments) from video.metasignaltech.com, both served from object storage, so large files stay off the application's origin and cache. On the usual criteria, that split was right.

Our Search Console property is a Domain property, which covers every subdomain and both schemes, so one report mixed all of these hosts. With data as of 2026-09-21, the page-indexing report showed 177 pages indexed and 204 not indexed. Three of the not-indexed reasons are worth reading line by line.

Page with redirect: 23 URLs. www.metasignaltech.com/, http://www.metasignaltech.com/, http://metasignaltech.com/, the apex root /, locale roots with a trailing slash like /ja/, and the root carrying ?from=AppAgg.com&utm_campaign=AppAgg.com&utm_medium=referral&utm_source=AppAgg.com on both www and the apex, from an app directory site's link. All expected: this bucket shows Google following the redirects we designed.

Not found (404): 20 URLs. One was https://download.metasignaltech.com/, the root of a host that only serves files at specific paths. Others were /mo, /mois, /月 and /mês, which Google extracted from price-unit strings such as /mo that appeared on their own in our pages' source. We have since stopped emitting unit strings that start with a slash.

Crawled – currently not indexed: 8 URLs. An HLS playlist on the video host (…/hls/720p.m3u8), an old APK on the download host, a font file under /_next/static/media/, and old www URLs.

None of this is an emergency, but each line is a question nobody had answered on purpose. We split those hosts off for origin and cache reasons; the report asked what the download host should say at /, and whether a playlist should be crawlable at all.

One policy per host

Any site with more than one host should be able to fill in this table. The app. row is for the split in the question; the file-host rows are recommendations for any host that serves only bytes.

Three rules behind the table are easy to get wrong.

robots.txt is per host and per scheme. The apex's file says nothing about download. or video.. A host without one is crawlable in full.

A blocked URL can't show its answer. A crawler that robots.txt keeps out never sees the redirect or noindex header on that URL, so never block www or http. For files, pick one tool per path: a noindex header takes a file out of results but costs a fetch; a Disallow saves the fetch, which matters for large media segments, but a linked URL can still be listed without content.

Every host has a root, and something will request it. A 404 there is fine if it is a decision. A redirect to the page that links the files is kinder to whoever trimmed a URL to see what else lives there.

Decide once, write it down, test it

A policy that lives in someone's head survives until the next migration. Keep the table in the repository next to the routing config, and turn each row into a check that runs on every deploy:

# Non-canonical entry points: one permanent hop to the canonical URL
for u in http://example.com/ http://www.example.com/ https://www.example.com/pricing; do
  curl -s -o /dev/null -w "%{http_code} $u -> %{redirect_url}\n" "$u"
done
 
# File host: what does the root answer, and do files carry noindex?
curl -sI https://download.example.com/ | head -1
curl -sI https://download.example.com/app-1.2.0.zip | grep -i x-robots-tag
 
# Canonical host: every sitemap URL is 200 and names itself as canonical
curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*' | cut -c6- |
while read -r u; do
  code=$(curl -s -o /tmp/page -w '%{http_code}' "$u")
  canon=$(grep -o 'rel="canonical" href="[^"]*"' /tmp/page | cut -d'"' -f4)
  [ "$code" = 200 ] && [ "$canon" = "$u" ] || echo "BAD $code $u -> $canon"
done

After this month's fixes our sitemap lists 87 URLs, all on the canonical host, each returning 200 with a self-referencing canonical. The indexing report counted 381 (177 + 204) across every host. The sitemap is where you tell search which URLs you mean; the report is where you learn which ones it found anyway.

If the mixed report is hard to read, add a URL-prefix property per host, and keep the Domain view for the hosts you forgot you had.

Checklist

• List every hostname and scheme that answers, including www and http, and name one canonical host. The rest redirect to it or serve only files.
• Per host, write down what / returns, what redirects, how indexing is controlled, and whether it belongs in a sitemap.
• Give each file host its own robots.txt; use X-Robots-Tag for files, which have no HTML to carry a meta tag.
• Don't ship strings in page source that start with / and aren't routes.
• Re-run the checks on every deploy.