the actual request delay may sometimes be less than this fixed setting. If you wish to ensure that this fixed setting is always used you will need to set
crawler.monitor_delay_typeto “fixed”.
Does crawler.request_delay only come into effect if crawler.monitor_delay_type=fixed?
Or does it somehow modify the dynamic delay also?
I’d like to increase the delay a little, but if I can still make use of the dynamic delay for efficiency that’d be great.
Hi Mr. Cook.
The way I read it is that it is possible to increase the delay by using the crawler.request_delay configuration even in dynamic mode but the delay is not guaranteed. i.e. Depending on the response, Funnelback will adapt to delay value slightly to make the most of the available resources .
If you want to ensure that the delay is “always” a fixed amount, you will need to set crawler.monitor_delay_type to fixed.
In my experience, I have not had any problems with simply setting the crawler.request_delay slightly higher for sites which a little less performant or sensitive.
Hope this helps.
Thanks,
~Gioan