The webcrawler collector is included with Explorer as a standard.
The webcrawler collector has three configuration options.
This is the uri at which the collector will start crawling. This is usually a domain name.
A list of Regular expressions which are matched against the full url of each link found during crawling. If any link matches any exclude pattern it will be skipped.
Some webservers handle user-agents differently. Here you can pretend to be a specific browser or robot. If you leave the field empty it will use a default useragent.