Analytics data not present after FB 15.10 upgrade

[Mirek]

Hello Guys,

recently I have upgraded Funnelback to 15.10 (from 15.8), latest patch (17) has been deployed and I started experiencing a weird situation where analytics reports (clicks, queries) are not presented (as I don't think they're empty).

The analytics data is not being presented:
* via Marketing Dashboard screen,
* via request URL pasted in browser (clicksAndCounts/queries JSON arrays are empty),
* via curl to admin API (with auth token, same as above),
* via Swagger Admin API (what is more interesting here I got the 0 response code stating that there was no response from the server),
* Trends Report prints the message that "Analytics has not been run on this collection".

Some screenshots attached:

What I can confirm is:
* analytics update executes fine (full and incremental, via admin and manually),
* update_reports.log and update_reports_launch.log show nothing special in debug mode (one warning about NO_HTTP_REFERRER),
* clicks/queries are being tracked/stored fine, they are being processed ok as per the above logs,
* enabled .facts files are not corrupted,
* so is the database, I've checked few queries against click_queries, query_clicks and query_words tables and all seemed to be fine,
* I've checked various profiles on different collections (new ones as well),
* when previewing the Marketing Dashboard analytics screens there are no GET calls being logged for "/admin-api/analytics/v1/" in YYYY_MM_DD.request.admin.log and audit.admin-api.log (when making the exact calls using the token or via the browser they are logged though).

Have you got any thoughts on what can be wrong? Is there any way I can debug what is happening between the database and actual calls to admin api? Are there any other log files that can help me find any related errors?

FB is behind the Openresty which operates on 80 and 443 (Jetty runs on 8080 and 8443+8444). It's probably possible the analytics data is not reachable on the server but no proxy settings have been changed.

Last thing is that just after the upgrade all the collections and analytics failed to update. The problem is that the updater doesn't inherit the collection settings related to heap size (which were probably introduced in >15.8?), so I had to add them explicitly in collection.cfg (update-pipeline.max_heap_size and analytics.max_heap_size in particular). The problem doesn't exist on any other known FB 15.10 system. Did I miss something here? Can it be somehow related to the analytics problem?

Many thanks for any help,
Mirek

[gordongrace]

Hi @Mirek -

There's another layer you haven't mentioned above that may be worth investigating, assuming you've got SSH access to this box.

After click and query logs have been processed, they're written into a SQLite database for downstream use by the Analytics API.

sqlite3 $SEARCH_HOME/admin/reports/COLLECTION/reports.sqlitedb
sqlite> SELECT * FROM agg_q_cpd JOIN query_words ON query_words.id=agg_q_cpd.query_words_id;

If successful, you should see a list of unique queries per time interval (default is day), indicating that the problem is further downstream.

Direct access to the analytics database is generally not encouraged or required, but there are scenarios where it may be unavoidable: https://community.funnelback.com/knowledge-base/implementation/rename-a-collection

[Mirek]

@gordongrace The database looks okay. I checked some queries before and also the one you mentioned and I can confirm that there are unique queries with corresponding time intervals.

Is there any way I can see what's happening just before the database is being requested?