Robots.txt Explained: Crawling Rules and Common Mistakes
What does robots.txt actually block? It gives cooperating crawlers rules about access.
1:47 · No narration: the video shows each step as text on screen, and every step is written out below, with buttons that jump the video to it.
Prefer YouTube? Watch this video on our YouTube channel (opens in a new tab).
The six points in the video
This is a silent video. The headings, explanations and labels below reproduce the information on screen.
-
Step 1: It guides crawler access
Rules tell specified crawlers which URLs they may request.
On-screen labels: Crawler instructions; Allowed routes; Disallowed routes.
-
Step 2: It is a public file
Do not put secrets or private information in the rules.
On-screen labels: Public address; Visible instructions; No secret storage.
-
Step 3: Crawling is not indexing
A blocked address can still appear in search results.
On-screen labels: Crawl restriction; Possible URL listing; Different controls.
-
Step 4: Use access controls for privacy
Confidential content needs real protection such as sign-in.
On-screen labels: Private content; Authentication; Restricted access.
-
Step 5: Allow needed page resources
Blocking useful scripts or styles can affect rendering.
On-screen labels: Page content; Required resources; Rendered page check.
-
Step 6: Check launch settings
An old sitewide block can accidentally remain after testing.
On-screen labels: Production rules; Important pages; Search Console checks.
A practical check for your business
Review the exact robots.txt rule with the person responsible for search settings. A crawl restriction and an indexing instruction do different jobs. Do not rely on robots.txt to protect private customer or administration information.
Sources and further detail
Common questions
What does robots.txt actually block?
It gives cooperating crawlers rules about access. Review crawler rules before a site launch.
Go further
More video tips
All 205 videos →
XML Sitemaps Explained: Helping Search Engines Find Pages
Help important pages be found.
What Does Noindex Mean? Search Visibility Controls Explained
Noindex, made clear.