Darshan Hiranandani : What is the average duration for crawling your web collections?

[darshanhiranandani23]

Hi team,

I’m currently seeing Funnelback take up to 48 hours to index a straightforward HTML collection of around 15,000 pages. This duration seems quite long to me. I’m curious about the experiences of others:

What indexing times are you observing for similar-sized collections?
Are there any optimizations or configurations that could reduce this time?
Additionally, is there a specific log file or method to check the exact duration from the start to completion of the indexing process?

Any insights or suggestions would be greatly appreciated!

Thanks!

Regards
Darshan Hiranandani

[plevan]

Hi Darshan,

You can view the update timings by looking at the crawler monitor graphs report which is displayed in the collection tools section of your web data source.

The time to crawl can be quite variable and depends on a number of factors. You can make the crawl more aggressive by increasing the number of concurrent requests made against your server, and also adjusting the delay between requests by configuring site profiles https://docs.squiz.net/funnelback/docs/latest/build/data-sources/web/site-profiles.html