Skip to content

CloudWatch

General Info

Use cases Monitoring service for AWS resources/applications:
  1. Collects monitoring and operational data in the form of logs/metrics/events
  2. Provides a unified view of AWS resources, applications and services
  3. Can set high resolution alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to optimize applications
Enables
  • Resource utilization, operational performance monitoring
    • Provides: disk read operations, CPU usage, and inbound network traffic
    • Does not provide memory usage
  • Real-time monitoring
  • Log aggregation and basic analysis
Limits
  • 5,000 alarms for each AWS account
  • Metrics data retained for 2 weeks by default

CloudWatch Logs

  • Allows to aggregate and monitor logs from applications and systems
  • Sources
    • Pushed from some AWS services (including CloudTrail)
    • Pushed from your apps/systems
    • Metrics from log entry matches

Elements

Type Description
Log Event
  • A record representing an event
  • Contains
    • Timestamp (date and time when the event occurred)
    • Raw message (specific data thr record is reporting)
Log Stream A sequence of log events that share the same source
Log Group A collection of log streams that share the same retention, monitoring, and access control settings
Subscription Filters Define a filter pattern that matches events in a particular log group, send them to Kinesis Data Firehose stream, Kinesis stream, or a Lambda function

Log Group Configurations

Configuration Description
Encryption Can Use KMS, CMK (Customer Key)
Retention
  • By default, there is no retention limit (does not use S3), unless you set a retention period
  • Can configure the retention period in days (1 - 3,653)
  • Can export Log Groups (in a particular time range) to S3 (not real time, up to 12h)

Integrations

  • CloudWatch Agent
    • Can be installed on a host (e.g. via SSM) to push logs to CloudWatch Logs
  • Cross-Account
    • Can receive events from other accounts by creating a destination in CloudWatch, which references a receiving Kinesis stream
    • The destination has a resource-based policy that controls which accounts can write to the destination
    • CloudWatch Logs on the sender side can then stream to the other account
  • CloudWatch Logs Insights
    • Provides a query syntax to apply to one or more (up to 20) log groups
    • Aware of the system fields that are automatically generated when a log event arrives to CloudWatch Logs:

      Field Description
      @message raw log event
      @timestamp when the event occurred
      @ingestionTime when the event was ingested
      @logStream the name of the log stream
      @log the unique ID of the log group

Metrics

  • CloudWatch as a metric repository service: records information in a time sequence by storing a number (metric) for each period of time
  • Ingestion

    • At the time of ingesting data into CloudWatch, the sources define:

      1. A namespace
      2. A metric name
      3. Metric dimensions to group the information (optionals)
      4. Resolution

        Resolution Granularity Notes
        Standard 1m Default
        High 1s Only available for custom metric
    • Retains the metrics for up to 15 months

    • CloudWatch Application Insights
      • Collect information about specific apps via an SSM Agent
      • Compatible with .NET and SQL Server apps
    • Alarms:
      • Defined thresholds for a specific metric
      • Once a metric crosses the threshold for a custom-defined number of times, the alarm is triggered
      • Can configure actions to execute and notifications to send when an alarm is triggered

CloudWatch Monitoring

Characteristics
  • Provides monitoring of performance metrics (real time)
  • Types

    Type Description
    BASIC
    • Sends data points to CloudWatch every 5mins
    • For a limited number of metrics
    • No charge
    DETAILED
    • Every 1min
    • Allows data aggregation (across AZ within a region)
    • Additional charge
  • Metrics

    • Hypervisor visible metrics (CPU) (NO MEMORY)
    • default = CPU util/network util
    • custom = disk space/RAM utilization
  • Alarms = if CPU > 80% for 5 mins —> alarm
  • Notifications = SNS
  • Can support on-premise services (doesn't need to be an AWS service)

CloudWatch Events

Characteristics

  • Provides a near real-time stream of events within your AWS account which can be used to trigger actions (such as a Lambda function) to perform a task
  • Near real-time stream of system events (event-driven security)
Events
  • AWS CloudTrail (API calls)
  • AWS resources state change
  • Custom events (code)
  • Scheduled
Rules
  • Match incoming events and route them to 1+ targets
  • Rules that trigger from either event patterns or a schedule
  • Rules send JSON to one or more targets
Targets
  • AWS lambda functions, SNS topics, SQS queues, Kinesis Streams
  • Lambda function can kick in a delete the newly created EC2 instance
Access control
  • Use IAM policies to restrict access to CloudWatch and the Actions they can perform
  • However, data is decoupled from its source, therefore you are not able to restrict access by the originating resource

EventBridge

Use Cases
  • Delivers a stream of events representing changes to resources (preferred option over CloudWatch Events for managing events from AWS resources)
  • Provides services and applications with an easy way to react without the need to configure the response actions inside themselves
  • Provides abstracted event buses and adds capabilities to connect sources of events with action executors

Rule Components

Component Description
Event Buses
  • Default
    • Automatically receives events from AWS resources
    • Unique bus per region and per account
    • You don't create it and you cannot delete it because it is part of the EventBridge infrastructure
  • Partners
  • Custom
Event Pattern An expression to match the events of interest
Target
  • A service that will be invoked
  • The target must be in the same region as the rule
  • For the same rule, you can configure several targets
  • For each target you can define an event transformation that describes the JSON object that will be delivered to the target
{
    "Entries": [
        {
        "Detail": {
            "custom-app-id": "id-2131580",
            "role": "accounting"
        },
        "DetailType": "Custom App Sample Event",
        "EventBusName": "my-event-bus",
        "Resources": [
            "chile:santiago:west-1:sensor/12345"
        ],
        "Source": "custom.app.001",
        "Time": 1589627829
        }
    ]
}
{
    "account": [
        "123456789012"
    ],
    "source": [
        "mycustom.app.001"
    ],
    "detail-type": [
        "My Custom App Sample Event"
    ],
    "detail": {
        "mycustom-app-id": [
        "id-2131580"
        ]
    }
}
{
    "source": [
        "aws.iam"
    ],
    "detail-type": [
        "AWS API Call via CloudTrail"
    ],
    "detail": {
        "eventSource": [
        "iam.amazonaws.com"
        ]
    }
}