I am looking at ways to speed up the crawling and processing of re-indexing our site because currently it is quite slow and it can take several days for an incremental update to happen.
If I were to use the noindex option to exclude the primary navigation, header and footer content, would that help in speeding up the crawling process as there is “technically” less for it to crawl?
For people who do exclude those primary menu links, how do you go about indexing those pages from the primary navigation? Do you create a “dummy” orphan page with links to those pages and get Funnelback to crawl that or do you add those individual urls to the list of “included” things to crawl?
Funnelback noindex comment tags won’t make any different to the time taken to crawl - they just hide areas of the page from what the indexer considers as page content. They also don’t stop the crawler from following links contained within the noindex area.
Robots.txt can make a difference as this can prevent the crawler from accessing parts of the site that shouldn’t be included in the index (but please note that this only affects include/exclude rules - the crawler doesn’t pay any attention to other directives about visit frequency).
Similarly use of page level robots directives (noindex/nofollow etc) can help too.
You can also adjust some other crawler settings to increase crawl speed or change the frequency of page visits.
Use include/exclude rules to ensure you only crawl the parts of the site you are interested in
Site profiles can also be used to add additional crawler threads (the maximum parallel requests setting). This can dramatically speed things up if your web server can handle the additional load, and the Funnelback server is appropriately resourced.