WEBVTT

00:00:00.000 --> 00:00:06.000
What does robots.txt actually block?

00:00:06.000 --> 00:00:13.000
It gives cooperating crawlers rules about access.

00:00:13.000 --> 00:00:18.000
Understand the crawler instructions.
6 things robots.txt does and does not do.

00:00:18.000 --> 00:00:30.000
It guides crawler access
Rules tell specified crawlers which URLs they may request.
Crawler instructions · Allowed routes · Disallowed routes

00:00:30.000 --> 00:00:42.000
It is a public file
Do not put secrets or private information in the rules.
Public address · Visible instructions · No secret storage

00:00:42.000 --> 00:00:54.000
Crawling is not indexing
A blocked address can still appear in search results.
Crawl restriction · Possible URL listing · Different controls

00:00:54.000 --> 00:01:06.000
Use access controls for privacy
Confidential content needs real protection such as sign-in.
Private content · Authentication · Restricted access

00:01:06.000 --> 00:01:18.000
Allow needed page resources
Blocking useful scripts or styles can affect rendering.
Page content · Required resources · Rendered page check

00:01:18.000 --> 00:01:30.000
Check launch settings
An old sitewide block can accidentally remain after testing.
Production rules · Important pages · Search Console checks

00:01:30.000 --> 00:01:39.000
Your checklist
It guides crawler access · It is a public file · Crawling is not indexing · Use access controls for privacy · Allow needed page resources · Check launch settings

00:01:39.000 --> 00:01:47.000
Review crawler rules before a site launch.
jerseyweb.je