Question

A DevOps team runs a fleet of application servers that write structured JSON logs to a single CloudWatch Logs log group. Each log event contains fields for 'statusCode', 'service', and 'region'. The team creates a metric filter to count HTTP 5xx errors and wants the resulting CloudWatch metric to be broken down separately by the 'service' and 'region' field values so they can alarm per-service. After deploying the metric filter with dimensions configured for 'service' and 'region', they notice that some emitted metrics are missing dimension values and total counts appear incomplete. What is the MOST likely cause and correct approach?

Accepted Answer

Metric filter dimensions are only populated when the specified JSON fields are present in every matching log event; the team should ensure all events include 'service' and 'region', and be aware that only up to 3 dimensions are supported per metric filter.

Answer

CloudWatch Logs metric filters do not support dimensions at all; the team must use CloudWatch Logs Insights scheduled queries to generate per-service metrics.

Answer

The metric filter is being throttled because high-cardinality dimensions require the Embedded Metric Format; they should switch all logging to EMF to fix missing values.

Answer

Dimensions on metric filters are populated asynchronously and can take up to 24 hours to backfill; the team should wait before evaluating completeness.