HOW IT WORKS
Generate a simple robots.txt
Create rules for User-agent: * plus either one Disallow path or Allow: / and a sitemap URL.
Sitemap + optional pathrobots.txtRuns locally
01Sitemap + optional path
02Build crawler directives
03robots.txt
Leaving Disallow empty generates Allow: /.
robots.txt controls crawling, not guaranteed indexing
Search engines can use many signals beyond this file when deciding whether a URL appears in results.
Your content stays in the browser
Processing runs locally in the page without requiring the content to be sent to UtilityLake.
02
In short
What exactly does it do?
Create rules for User-agent: * plus either one Disallow path or Allow: / and a sitemap URL.
Example?
Leaving Disallow empty generates Allow: /.