URL Paths to Exclude. Using Excluded URLs is a method for restricting the crawler, and this method allows you to specify URLs or entire directories to avoid.

Navigating to Global Settings. From here, the 'Excluded External URLs' tab allows you to exclude Sitebulb from scheduling and crawling specific external URLs ...

The Advanced Settings allow you more granular control of how Sitebulb crawls and what data it collects. Crawler configuration · Advanced robots settings ...

There are 4 ways in which you can exclude particular internal URLs from being crawled: Excluding specific URLs or paths; Including specific URLs or paths ( ...

Learn everything you need to audit websites successfully with Sitebulb - everything from the basics through to advanced use cases, licensing and billing.

Virtual robots.txt. This setting allows you to over-ride the website's robots.txt file, to instead use a 'virtual robots.txt' file.

Excluding specific URL paths. Within Advanced Settings, navigate to the URLs tab, then select the Excluded URLs option, and on this screen, add in the folder ...

When setting up a new project or crawl navigate to the "Crawl Settings" Tab. Crawl settings tab. It should take you to this page, here can select "Limit URL ...