Darshan Hiranandani : How long does it take for Funnelback to complete a crawl for your web collections?

General Discussion
1

Hi Team,

I’ve been seeing Funnelback taking up to 48 hours to index a straightforward HTML collection of around 15,000 pages, which seems like a long time. I’m curious to know how long it typically takes for others to crawl similar collections.

Also, is there a single log file or method to track the exact time it takes from crawl initiation to completion? Any tips or recommendations on improving crawl speed or checking logs for performance would be really helpful.

Looking forward to your feedback and suggestions!

Thanks!
Darshan Hiranandani

2

The crawl time mostly depends on the speed of the server being crawled and how quickly it responds.

There are various crawl settings that can affect the crawl time, e.g. values like the crawler request delay.

You also can use things like site profile configuration to crawl more agressively if the server can handle more requests.

The collection update history (on the analyse tab) shows you a history graph showing where time is spent during updates.

It is also worth checking your crawl logs to see if you’re getting into a crawler trap (like crawling a calendar indefinitely).