- Use a 503 with a Retry-After header for short planned downtime, never a 200 error page.
- Robots.txt controls crawling, not indexing; use noindex or authentication to keep pages out of search.
- Keep robots.txt small, well under size limits, and test every rule before deploying.
- Avoid square brackets in URLs where you can, and keep each URL form consistent in links, canonicals and sitemaps.
Some of the most expensive technical SEO problems come from small details nobody reviews: a status code returned during maintenance, a rule in a text file, a character in a URL. Three of these keep coming up in agency audits: how Google treats 503 responses, what robots.txt can and cannot do, and whether bracket characters in URLs are a risk. Here is a practical way to handle each.
What a 503 tells Google
A 503 Service Unavailable response says the server is temporarily unable to respond. That is exactly what you want during a short deployment or an overloaded moment, because it signals that the content has not gone away. Google generally slows crawling when it sees server errors and retries later.
The danger is duration. If 503s persist for days, Google may treat the problem as more than temporary, and URLs can begin dropping from the index. A 503 is a tool for hours, not weeks.
Do it correctly
- Return a real 503 status, not a friendly maintenance page served with a 200. A 200 teaches Google your thin placeholder is the page.
- Add a Retry-After header to suggest when crawlers should come back.
- Serve 503 only where needed. If only checkout is down, do not error the whole site.
- Never 503 your robots.txt file as a shortcut. If robots.txt itself returns a server error for long, Google may become cautious and reduce crawling of the whole host.
Also watch for accidental 503s: rate limiters, bot-protection rules and overloaded origins can return them to Googlebot while humans see a normal site. Check server logs for Googlebot hits with 5xx codes, and compare with the Crawl Stats report in Search Console.
Robots.txt: powerful, but narrow
Robots.txt tells compliant crawlers which paths they may fetch. That is all. It does not remove pages from search, does not secure anything, and does not stop a blocked URL from appearing as a bare listing if other sites link to it.
Common misunderstandings
- Blocking is not deindexing. If you disallow a page, Google cannot see a noindex tag on it. To remove a page, let it be crawled and add noindex, or return a 404/410, or put it behind a login.
- Size is limited. Google reads only a limited amount of a robots.txt file, and rules beyond that are ignored. Bloated files from years of accumulated rules are a real risk. Consolidate with patterns instead of listing thousands of paths.
- Only some directives are supported. Google follows allow, disallow, user-agent and sitemap. Unsupported lines such as crawl-delay are ignored, so do not rely on them.
- It is public. Anyone can read it, so do not list secret directories in it.
Be careful with blocking CSS, JavaScript or API endpoints that pages need to render. If Google cannot fetch them, it may misjudge your layout and content.
Bracket URLs: harmless, but handle with care
Square brackets show up in URLs thanks to frameworks that use them for dynamic routes, and in parameter arrays such as filter[color]=blue. Google can crawl such URLs, but brackets are technically reserved characters, and tools, proxies and analytics platforms do not always treat them consistently. The risk is not a penalty; it is duplication and confusion.
- Check encoded versus raw forms. The same page might be reachable as [ and as %5B. Pick one, redirect or canonicalize the other.
- Keep internal links, canonicals and sitemaps identical in format.
- Prefer clean paths for pages you want ranking, and reserve bracketed parameters for faceted or app states.
- Control parameter sprawl. Faceted bracket parameters can create endless crawlable combinations; canonicalize or disallow low-value ones.
A quick audit routine
- Pull a month of server logs and filter for Googlebot; list status codes by section.
- Fetch robots.txt as Googlebot would and confirm it returns 200 and loads fast.
- Test ten important URLs and ten parameter URLs against your rules using Search Console tools.
- Crawl the site and flag any URL containing brackets, encoded or not, then review canonicals.
- Document your maintenance process so developers know to send 503 with Retry-After.
How Growth Asteroid can help
These issues rarely announce themselves; they surface as slow indexing, crawl waste or sudden drops after a release. Growth Asteroid runs technical audits that combine log analysis, crawl data and Search Console, then turns findings into a prioritized fix list your developers can ship.
Conclusion
Treat 503s as short, honest signals, treat robots.txt as a crawl-control tool rather than an index-removal tool, and keep URL formats tidy and consistent. None of it is glamorous, but together these habits keep Google spending its crawl effort on pages that earn you traffic.
Frequently asked questions
How long can a site return 503 before it hurts SEO?
A few hours is generally safe. If errors last for days, Google may treat the outage as lasting and start dropping URLs, so restore service or fix the cause quickly.
Does robots.txt keep a page out of Google?
No. It only stops crawling. A blocked URL can still appear if linked elsewhere. Use noindex on a crawlable page, a removal status code, or authentication.
Are square brackets in URLs bad for SEO?
Not inherently. Google can crawl them, but inconsistent encoded and raw versions can cause duplicates. Standardize one form and use canonicals.