Question

A retail company runs multiple microservices on GKE and Compute Engine. The analytics team wants to run ad hoc SQL queries across the last 18 months of application and audit logs to identify usage trends and correlate errors with business events. Logs are currently retained only in Cloud Logging's default buckets. The platform team needs a low-maintenance, cost-effective way to make this historical log data queryable at scale without impacting the running services. What should they recommend?

Accepted Answer

Create an aggregated log sink at the organization level that exports the relevant logs to a BigQuery dataset, and set an appropriate table partition expiration for 18-month retention.

Answer

Increase the retention period of the default Cloud Logging bucket to 18 months and have analysts use the Logs Explorer query interface for SQL-style analysis.

Answer

Export logs to Cloud Storage via a log sink and have analysts download and parse the JSON files with a custom script whenever they need to run analysis.

Answer

Install the Ops Agent on every VM and node to forward logs to a self-managed Elasticsearch cluster on Compute Engine for querying.