Prevent Facebook comments from being indexed

bewilderbeest 2017-04-27 23:45:39 UTC #1

Hi. We have what I'd call a standard Funnelback collection set up for a Facebook page. An issue we're having is user comments are being indexed for each post on the Facebook page. Ideally, it would be just the contents of the post that are being indexed, and not the user comments, names of commenters etc,. How can we prevent these from being indexed, given we can't control the Facebook HTML structure and use the noindex tags?

gordongrace 2017-04-28 11:10:40 UTC #2

Hi @bewilderbeest -

The scenario you're describing (selectively indexing regions from external web resources) isn't unique to Facebook content - the built-in InjectNoIndexFilterProvider should assist you here:

Refer:
https://docs.funnelback.com/15.10/more/extra/filter_classes_collection_cfg.html
https://docs.funnelback.com/15.10/more/extra/injectnoindexfilterprovider.html

A full regather/refilter will be required, once you've got your URL RegEx and CSS selector correct.

bewilderbeest 2017-05-02 01:32:53 UTC #3

Thanks @gordongrace for the info.

This is what I've added to the collection.cfg file:

filter.classes=CombinerFilterProvider,TikaFilterProvider,ExternalFilterProvider:JSoupProcessingFilterProvider:DocumentFixerFilterProvider
filter.noindex.1=.* script,form
filter.noindex.2=div.UFIComment

And I've updated the collection but still am having problems with comments being included within the search.

Is there anything I've missed or do you have any documentation specific to Facebook collections?

gordongrace 2017-05-02 07:40:15 UTC #4

If this is just a web collection (standard seed URLs, include/exclude patterns), then the InjectNoIndexFilters will be run against that gathered HTML content. You can view the cached copies of filtered content to confirm whether (and where) the NOINDEX tags are being injected.

If you're using a Facebook Collection, there's additional options controlled by the xml.cfg file to indicate which fields being gathered from the Facebook API are to be indexed.