Collection update running queries on the meta collection

[kgarske]

Hello,

We have a web collection which is indexing json (via the json to XML filter).

Each time this web collection is updated it results in a series of automated queries being made on the parent meta collection. These queries look to be running for each record in the web collection and have the format of !padrenull id:10527

Mon May 18 03:01:42 2020,43.245.42.63,!padrenull id:10527,,,1,1,S2x,1,1417,20,_default,-
Mon May 18 03:01:43 2020,43.245.42.63,!padrenull id:10596,,,1,1,S2x,1,1417,20,_default,-
Mon May 18 03:01:44 2020,43.245.42.63,!padrenull id:10475,,,1,1,S2x,1,1417,19,_default,-
Mon May 18 03:01:45 2020,43.245.42.63,!padrenull id:11155,,,1,1,S2x,1,1417,19,_default,-
Mon May 18 03:01:46 2020,43.245.42.63,!padrenull id:12854,,,1,1,S2x,1,1417,19,_default,-
Mon May 18 03:01:46 2020,43.245.42.63,!padrenull id:11236,,,1,1,S2x,1,1417,20,_default,-
Mon May 18 03:01:47 2020,43.245.42.63,!padrenull id:10461,,,1,1,S2x,1,1417,20,_default,-

Just a a few questions about this:

1. What could be the reason for these queries on the meta collection are being run (is it to gather spelling suggestions?)
2. How can we prevent these from hitting the meta collections analytics? Can we turn them into system queries or modify the profile that they are run on somehow.

Many Thanks,

Karen

[plevan]

Hi Karen,

There's no reason as far as I'm aware that Funnelback would query the meta collection and log requests as part of a collection update.

Is there any workflow or filters on the web collection that cause queries to run against the meta collection? e.g. a filter that performs some lookup for each record it processes.

Once you know the source of the requests you can update these so that the requests don't result in requests being logged for analytics. The easiest way to achieve this is to create a new profile on the meta collection that is designed for use for system type lookups. In this profile you can set -log=off in padre_opt.cfg to disable any requests to that profile.

[kgarske]

Thanks Peter

The only filter that is being used is the ForceJSONMime:JSONToXML

There is no workflow.

[kgarske]

Just for the record we have discovered where these queries were coming from thanks to Peter's help.

There was a web collection that was updating at a similar time. It was crawling URLs which triggered a Funnelback query in order to serve up their content.

Now we know where these queries are coming from we have modified the profile they use so as to redirect them away from the live analytics.