Override robots meta tag

Developers Gathering & indexing
1

I’m trying to force Funnelback to index some pages even though they have the following metadata tag:
<meta name="robots" content="noindex,nofollow" />
They have this tag because it’s a public test system and we don’t want public crawlers to index it, but do want our test Funnelback instance to crawl it.

I considered trying to wrap the meta tag with <!-- noindex --> / <!-- endnoindex --> comments, but that would be tricky with this system, and might affect other search engines (eg Yandex).

I tried adding a workflow filter like this, but it doesn’t seem to have an effect:

// If crawling staging environment, ignore robots meta (force FB crawl for testing environment, but don't let Google crawl it).
if (urlContains("testserver.example")) {
    if ((contentContains("<meta name=\"robots\" content=\"noindex,nofollow\" />"))) {
        insertMetaTag("robots", "index,follow");
    }
}

So have done a bit more browsing around the docs and found some more options to try, which I think should have the same effect as wrapping noindex comments around the meta tag.

Will report back if it works…

2

Ok I’m not sure those methods worked, I ended up with errors and I’m not sure what was happening..

But anyway after going around in circles a couple of times I’ve been given a workflow.cfg that works:

if (urlContains("testserver.example")) {
    replaceContent "<meta name=\"robots\" content=\".*\">", ""
}

It seems “replaceContent” means any part of the HTML. And perhaps workflow.cfg is actually a Groovy script, despite the file extension.

Another option would have been to write a custom Groovy filter, but thankfully that wasn’t necessary.