recently I have upgraded Funnelback to 15.10 (from 15.8), latest patch (17) has been deployed and I started experiencing a weird situation where analytics reports (clicks, queries) are not presented (as I don't think they're empty).
The analytics data is not being presented:
* via Marketing Dashboard screen,
* via request URL pasted in browser (clicksAndCounts/queries JSON arrays are empty),
* via curl to admin API (with auth token, same as above),
* via Swagger Admin API (what is more interesting here I got the 0 response code stating that there was no response from the server),
* Trends Report prints the message that "Analytics has not been run on this collection".
Some screenshots attached:
What I can confirm is:
* analytics update executes fine (full and incremental, via admin and manually),
* update_reports.log and update_reports_launch.log show nothing special in debug mode (one warning about NO_HTTP_REFERRER),
* clicks/queries are being tracked/stored fine, they are being processed ok as per the above logs,
* enabled .facts files are not corrupted,
* so is the database, I've checked few queries against click_queries, query_clicks and query_words tables and all seemed to be fine,
* I've checked various profiles on different collections (new ones as well),
* when previewing the Marketing Dashboard analytics screens there are no GET calls being logged for "/admin-api/analytics/v1/" in YYYY_MM_DD.request.admin.log and audit.admin-api.log (when making the exact calls using the token or via the browser they are logged though).
Have you got any thoughts on what can be wrong? Is there any way I can debug what is happening between the database and actual calls to admin api? Are there any other log files that can help me find any related errors?
FB is behind the Openresty which operates on 80 and 443 (Jetty runs on 8080 and 8443+8444). It's probably possible the analytics data is not reachable on the server but no proxy settings have been changed.
Last thing is that just after the upgrade all the collections and analytics failed to update. The problem is that the updater doesn't inherit the collection settings related to heap size (which were probably introduced in >15.8?), so I had to add them explicitly in collection.cfg (update-pipeline.max_heap_size and analytics.max_heap_size in particular). The problem doesn't exist on any other known FB 15.10 system. Did I miss something here? Can it be somehow related to the analytics problem?
Many thanks for any help,