Hello,
I am attempting to crawl an intranet site - but am finding that almost all the content pages are being redirected with the M = Meta Refresh (HTML) Redirect reason.
There is a meta refresh tag <meta http-equiv="refresh" content="3600; url=homepage> on each page.</p>
<p>Is there an simple way I can tell the crawler to ignore this type of redirect?</p>
<p>Cheers,</p>
<p>Karen</p>