How to Create Custom Metrics on Log Groups Using IaC

Your application logs are full of useful data, but CloudWatch doesn't automatically turn them into metrics. You need metric filters to extract numbers from log events and create alarms that actually matter.

This guide shows you how to set up custom metrics from log groups using Infrastructure as Code.

Understanding Metric Filters

Metric filters scan log events and extract numerical data based on patterns you define. When a log event matches your filter pattern, it increments a custom CloudWatch metric.

Implementation with Terraform

Here's a complete example that creates a metric filter to track application errors and sets up an alarm:

# Create the log group
resource "aws_cloudwatch_log_group" "app_logs" {
  name              = "/aws/lambda/my-application"
  retention_in_days = 7
}

# Create metric filter for error tracking
resource "aws_cloudwatch_log_metric_filter" "error_filter" {
  name           = "application-error-count"
  log_group_name = aws_cloudwatch_log_group.app_logs.name
  pattern        = "[timestamp, request_id, ERROR, message]"
  
  metric_transformation {
    name      = "ApplicationErrors"
    namespace = "MyApp/Errors"
    value     = "1"
  }
}

# Create alarm for error metric
resource "aws_cloudwatch_metric_alarm" "error_alarm" {
  alarm_name          = "high-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "ApplicationErrors"
  namespace           = "MyApp/Errors"
  period              = "300"
  statistic           = "Sum"
  threshold           = "10"
  alarm_description   = "This metric monitors application error rate"
  
  alarm_actions = [aws_sns_topic.alerts.arn]
}

# SNS topic for notifications
resource "aws_sns_topic" "alerts" {
  name = "app-error-alerts"
}

Advanced Metric Patterns

Track specific error types with different patterns:

# Database connection errors
resource "aws_cloudwatch_log_metric_filter" "db_errors" {
  name           = "database-connection-errors"
  log_group_name = aws_cloudwatch_log_group.app_logs.name
  pattern        = "?ERROR ?Database ?connection"
  
  metric_transformation {
    name      = "DatabaseErrors"
    namespace = "MyApp/Database"
    value     = "1"
  }
}

# Response time tracking
resource "aws_cloudwatch_log_metric_filter" "response_time" {
  name           = "api-response-time"
  log_group_name = aws_cloudwatch_log_group.app_logs.name
  pattern        = "[timestamp, request_id, duration_ms > 1000]"
  
  metric_transformation {
    name      = "SlowRequests"
    namespace = "MyApp/Performance"
    value     = "$duration_ms"
  }
}

Testing Your Setup

Verify metric filters work correctly:

# Generate test log entry
aws logs put-log-events \
  --log-group-name "/aws/lambda/my-application" \
  --log-stream-name "test-stream" \
  --log-events timestamp=$(date +%s000),message="ERROR Database connection failed"

# Check metric data
aws cloudwatch get-metric-statistics \
  --namespace "MyApp/Errors" \
  --metric-name "ApplicationErrors" \
  --start-time $(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Sum

Key Configuration Tips

Filter Patterns: Use specific patterns to avoid false positives. Test patterns with sample log data before deployment.

Metric Namespaces: Organize metrics with clear namespaces like MyApp/Errors or MyApp/Performance for easier management.

Alarm Thresholds: Start with conservative thresholds and adjust based on actual traffic patterns.

Cost Optimization: Limit metric filter scope to essential log groups. Each custom metric costs $0.30 per month.

Frequently Asked Questions

1. What are CloudWatch metric filters?

Metric filters scan log events and extract values to create custom metrics. They allow you to monitor specific patterns like errors or performance issues.

2. Why create custom metrics from log groups?

Custom metrics help you track important application events like errors, latency, or usage patterns that are not available by default in CloudWatch.

3. How does Terraform help in creating metric filters?

Terraform allows you to define metric filters, alarms, and log groups as code, making your monitoring setup reproducible and version-controlled.

4. Can I create alerts based on custom metrics?

Yes, you can create CloudWatch alarms that trigger notifications when custom metric thresholds are exceeded, helping detect issues early.

Conclusion

Custom metrics from log groups provide real-time visibility into application behavior. Using IaC ensures your monitoring setup is reproducible and version-controlled. Start with basic error tracking, then expand to performance and business metrics as needed.

The combination of metric filters and alarms creates an automated monitoring system that alerts you to issues before they impact users.

Read More from KubeBlogs

Fix GCP 403 Permission Errors in Compute Engine VMs
https://www.kubeblogs.com/gcp-permissions-and-scopes-for-compute-engine-vms/

Run Streamlit Reliably in Production using Supervisor
https://www.kubeblogs.com/run-streamlit-reliably-in-production-using-supervisor/

Docker Storage Full? Move It to an External Volume
https://www.kubeblogs.com/docker-storage-on-an-external-volume/