Monitoring, Audit & Performance

1. Three services, three different questions

Once a workload runs on AWS you end up answering three questions that look similar but are not: how fast or slow is the system behaving, who called which API, and how has the configuration changed and does it still comply with your rules. Each question belongs to its own service.

• Amazon CloudWatch answers the performance question: metrics, logs, dashboards, alarms.
• AWS CloudTrail answers the “who did it” question: a history of API calls made inside the account.
• AWS Config answers the configuration question: what a resource looked like over time and whether it satisfies your compliance rules.

Keep that mapping in mind; most exam questions in this area are really asking you to pick the right one of the three.

2. CloudWatch Metrics

Every service in AWS feeds metrics into CloudWatch. Treat a metric as one variable you follow over time — CPUUtilization or NetworkIn, for instance — and each such variable belongs to a namespace.

A dimension is an attribute attached to a metric that lets you slice it: the instance id, the environment name, and similar. One metric carries up to 30 dimensions, so the same measurement can be broken out per instance, per environment and so on rather than becoming a separate metric each time. Every data point is stamped with a timestamp, which is what turns a metric into a time series, and metrics can be assembled into CloudWatch dashboards.

When AWS does not publish the metric you need, you publish it yourself as a Custom Metric. The classic example is RAM usage, which is not among the metrics EC2 emits on its own.

Exam tip

Out of the box, EC2 gives you high-level disk, CPU and network metrics. Memory is not among them. Whenever a question needs RAM, the answer involves a custom metric or the CloudWatch Unified Agent.

CloudWatch Metric Streams

Rather than pulling metrics through the API on a schedule, a Metric Stream keeps pushing them towards a destination you nominate, with low latency and a near-real-time lag behind the metric itself. The stream’s target is Amazon Kinesis Data Firehose, so Firehose’s own destinations follow on from there: Amazon S3, Amazon Redshift, Amazon OpenSearch, or S3 data queried with Athena. Third-party providers are supported directly too — Datadog, Dynatrace, New Relic, Splunk, Sumo Logic. And if you do not want the lot, a filter narrows the stream down to a subset of metrics.

3. CloudWatch Logs

CloudWatch Logs organizes log data in two levels:

• a log group, which carries an arbitrary name and usually represents one application;
• a log stream inside that group, which represents one instance, one log file or one container.

Each log group has an expiration policy: you can keep logs forever or set a retention between one day and ten years. Logs are encrypted by default, and you can switch to KMS-based encryption with your own keys.

From CloudWatch Logs the data can be sent onward to Amazon S3 (as an export), Kinesis Data Streams, Kinesis Data Firehose, AWS Lambda or OpenSearch.

Logs reach CloudWatch Logs from a long list of sources, and it is worth knowing what each one actually contributes:

• the SDK, the CloudWatch Logs Agent and the CloudWatch Unified Agent, for anything you push yourself;
• Elastic Beanstalk, which collects the logs of the application it hosts;
• ECS, which collects them from the containers;
• AWS Lambda, which collects the function logs;
• VPC Flow Logs, the VPC-specific traffic logs;
• API Gateway;
• CloudTrail, based on a filter rather than wholesale;
• Route 53, which logs DNS queries.

4. CloudWatch Logs Insights

Logs Insights searches and analyzes the log data already stored in CloudWatch Logs — finding a specific IP inside a log, counting occurrences of ERROR, and so on. It offers a purpose-built query language and automatically discovers fields from AWS service logs and from JSON log events.

A query can fetch selected event fields, filter on conditions, compute aggregate statistics, sort events and limit the number returned. Saved queries can be added to CloudWatch dashboards, and one query can span multiple log groups in different AWS accounts.

Exam tip

Logs Insights is a query engine, not a real-time engine. If the scenario needs live processing of log events as they arrive, the answer is Logs Subscriptions, not Logs Insights.

5. Getting logs out: S3 Export vs Subscriptions

Both mechanisms move log data out of CloudWatch Logs, and the exam cares about the difference in latency.

Mechanism	API / target	Latency
S3 Export	CreateExportTask to an S3 bucket	Log data can take up to 12 hours to become available for export
Subscriptions	Kinesis Data Streams, Kinesis Data Firehose, Lambda	Real-time to Kinesis Data Streams and Lambda; near real-time where Firehose delivers onward to S3 or OpenSearch

A Subscription Filter decides which log events are delivered to the destination. From Kinesis Data Firehose the events can flow on to Amazon S3 or OpenSearch (near real-time); Kinesis Data Streams feeds real-time consumers such as Lambda, Kinesis Data Analytics, EC2 or Firehose.

Subscriptions are also the building block for log aggregation across accounts and regions: subscription filters in several accounts and regions all point at one Kinesis Data Stream, which feeds Kinesis Data Firehose and finally a central S3 bucket.

For a cross-account subscription, log events from a sender account are delivered to a destination (a Kinesis Data Stream or Firehose) in a recipient account. The recipient creates a subscription destination with a destination access policy that names the sender account, plus a cross-account IAM role the sender can assume with permission to PutRecord on the stream.

Real-world example

A company runs one AWS account per team in three regions and wants a single searchable archive of application logs. Each account attaches a subscription filter to its log groups pointing at a Kinesis Data Stream in the logging account; Firehose writes the aggregated stream to one S3 bucket that the security team queries.

6. EC2 logs and the agents

By default no logs leave an EC2 machine for CloudWatch. You must run a CloudWatch agent on the instance, tell it which log files to push, and make sure the instance IAM permissions allow it. The same agent can be installed on on-premises servers.

There are two agents, both aimed at virtual servers (EC2 instances or on-premises machines):

• the CloudWatch Logs Agent, the older version, which can only send logs to CloudWatch Logs;
• the CloudWatch Unified Agent, which additionally collects system-level metrics such as RAM and processes, still collects logs, and can be configured centrally through SSM Parameter Store.

The Unified Agent collects a rich set of metrics directly on the Linux server or EC2 instance: CPU (active, guest, idle, system, user, steal), disk metrics (free, used, total) and disk IO (writes, reads, bytes, iops), RAM (free, inactive, used, total, cached), netstat data (numbers of TCP and UDP connections, packets, bytes), process counts (total, dead, blocked, idle, running, sleeping) and swap space (free, used, used percentage).

Exam tip

“Centralised agent configuration” points at the Unified Agent plus SSM Parameter Store. “Collect memory and process metrics from EC2” points at the Unified Agent, never the old Logs Agent.

7. CloudWatch Alarms

An alarm exists to trigger a notification off any metric. How the metric is evaluated is up to you — sampling, a percentage, the maximum, the minimum and so on — and whatever the evaluation produces, the alarm always sits in exactly one of three states:

• OK
• INSUFFICIENT_DATA
• ALARM

The period is the length of time in seconds over which the metric is evaluated. For high-resolution custom metrics the period can be 10 seconds, 30 seconds, or a multiple of 60 seconds.

Alarm targets

An alarm can stop, terminate, reboot or recover an EC2 instance; trigger an Auto Scaling action; or send a notification to Amazon SNS, from which you can drive essentially anything else.

Composite Alarms

A normal CloudWatch alarm watches a single metric. A Composite Alarm watches the states of several other alarms and combines them with AND and OR conditions. That is how you cut down “alarm noise”: alarm A watches CPU, alarm B watches IOPS, and the composite alarm notifies SNS only when both are in ALARM.

EC2 Instance Recovery

EC2 exposes status checks at three levels: the instance status checks the EC2 VM itself, the system status checks the underlying hardware, and the attached EBS status checks the attached EBS volumes. An alarm on StatusCheckFailed_System can trigger EC2 Instance Recovery, which alerts an SNS topic and brings the instance back with the same private IP, public IP, Elastic IP, metadata and placement group.

Two things worth knowing

First, alarms can be created on CloudWatch Logs Metric Filters, which is how a log becomes something you can alarm on at all: the metric filter matches a pattern in the log events arriving in CloudWatch Logs and turns the number of matches into a CloudWatch metric, an ordinary alarm then watches that metric, and the alert goes out through SNS. That chain is the answer whenever a scenario says “notify me when this string appears in the logs”.

Second, you do not have to wait for a real breach to check that the chain works. Forcing the alarm state from the CLI exercises the whole notification path:

aws cloudwatch set-alarm-state --alarm-name "myalarm" --state-value ALARM --state-reason "testing purposes"

8. Amazon EventBridge

Amazon EventBridge, formerly CloudWatch Events, reacts to things happening. It works in two modes:

• Schedule — cron jobs, for scripts that must run on a timetable, for example triggering a Lambda function every hour.
• Event Pattern — rules that react to a service doing something, for example sending an email through SNS when the IAM root user signs in.

Sources of events include EC2 instance state changes, a failed CodeBuild build, an S3 object upload, a new Trusted Advisor finding, any API call captured by CloudTrail, or a schedule. EventBridge can optionally filter events, hands the event over as JSON, and routes it to compute targets (Lambda, AWS Batch, ECS task), integration targets (SQS, SNS, Kinesis Data Streams), orchestration targets (Step Functions, CodePipeline, CodeBuild) and maintenance targets (SSM, EC2 actions).

Buses, archive and replay

EventBridge has a default event bus for AWS services, partner event buses for SaaS partners, and custom event buses for your own applications. Event buses can be made accessible to other AWS accounts through resource-based policies. You can archive the events sent to a bus (all of them or a filtered subset, indefinitely or for a set period) and later replay them.

A resource-based policy manages permissions on a specific event bus — allowing or denying events from another AWS account or region. The typical use case is aggregating every event from an AWS Organization into one central bus in a single account, where another account calls PutEvents and a Lambda function consumes the result.

Schema Registry

EventBridge can analyze the events flowing through a bus and infer their schema. The Schema Registry stores those schemas, lets you version them, and generates code for your application so it knows the structure of the data in advance.

EventBridge with CloudTrail

Because CloudTrail records any API call, pairing the two lets you alert on specific administrative actions. A DeleteTable call against DynamoDB is logged by CloudTrail, matched by an EventBridge rule and pushed to SNS. The same pattern catches an AssumeRole against an IAM role, or an AuthorizeSecurityGroupIngress that edits a security group’s inbound rules.

9. The four CloudWatch Insights products

The name “Insights” is reused for four unrelated features, which is exactly why the exam likes it.

Feature	What it does	Scope
Container Insights	Collects, aggregates and summarizes metrics and logs from containers	ECS, EKS, Kubernetes on EC2, Fargate for both ECS and EKS; on EKS and Kubernetes it uses a containerised CloudWatch Agent to discover containers
Lambda Insights	Monitoring and troubleshooting for serverless applications: system-level metrics (CPU time, memory, disk, network) plus diagnostics such as cold starts and worker shutdowns	AWS Lambda, delivered as a Lambda Layer
Contributor Insights	Analyzes log data and builds time series of contributor data, showing the top-N contributors, the number of unique contributors and their usage	Any AWS-generated logs (VPC, DNS and so on); build rules from scratch or use AWS sample rules, plus built-in rules for other services’ metrics
Application Insights	Automated dashboards that surface potential problems with a monitored application so you can isolate ongoing issues	Applications on EC2 with selected technologies (Java, .NET, Microsoft IIS, databases) plus related resources such as EBS, RDS, ELB, ASG, Lambda, SQS, DynamoDB, S3, ECS, EKS, SNS, API Gateway

Contributor Insights is the one that finds bad hosts, the heaviest network users, or the URLs generating the most errors — feed it VPC Flow Logs and it returns the top ten IP addresses. Application Insights is powered by SageMaker and sends its findings and alerts to Amazon EventBridge and SSM OpsCenter, and the visibility it adds into application health is aimed squarely at cutting the time it takes to troubleshoot and repair.

Boiled down to one line each: Container Insights is metrics and logs from containers, and needs the agent on Kubernetes; Lambda Insights is detailed metrics for troubleshooting serverless applications; Contributor Insights finds the top-N contributors through CloudWatch Logs; Application Insights builds the dashboard automatically for an application and the AWS services around it.

10. AWS CloudTrail

CloudTrail is the governance, compliance and audit layer of an AWS account, and you do not have to turn it on — it is enabled by default. What it keeps is a history of the events and API calls raised inside the account, whether they came from the console, the SDK, the CLI or an AWS service acting on your behalf. Those records can be delivered into CloudWatch Logs or an S3 bucket, and a trail covers all regions (the default) or one single region.

The flow is worth picturing: IAM users and IAM roles act through the console, the CLI or an SDK, CloudTrail records what they called, and you then inspect and audit the result — in the CloudTrail console for recent activity, or in CloudWatch Logs and an S3 bucket for whatever you have routed there.

Exam tip

If a resource disappears and nobody admits to deleting it, investigate CloudTrail first.

Event types

• Management Events cover operations performed on resources in the account: configuring security (AttachRolePolicy in IAM), configuring routing (CreateSubnet in EC2), setting up logging (CreateTrail in CloudTrail). Trails log management events by default, and you can separate read events, which do not modify resources, from write events, which may.
• Data Events are not logged by default because they are high volume. They cover S3 object-level activity (GetObject, DeleteObject, PutObject, again separable into read and write) and Lambda function execution activity (the Invoke API).
• CloudTrail Insights Events are the third category, described below.

CloudTrail Insights

Turn CloudTrail Insights on and CloudTrail starts looking for activity that does not fit the account’s habits: resources provisioned inaccurately, service limits being hit, a sudden burst of AWS IAM actions, or a gap where periodic maintenance activity should have happened. It gets there by analyzing normal management events first to establish a baseline, then analyzing write events continuously against it. What it finds shows up as an anomaly in the CloudTrail console, the event is delivered to Amazon S3, and an EventBridge event is generated as well so a response can be automated.

Retention

Events are kept for 90 days inside CloudTrail. To keep them longer — whether management, data or Insights events — log them to S3 and analyze them with Athena.

11. AWS Config

AWS Config helps you audit and record how compliant your AWS resources are, and it records their configurations and the changes to them over time. The kind of question it settles: whether any security group leaves SSH open to the world, whether a bucket has public access, and what has changed in an ALB’s configuration since last month. Any change can raise an SNS notification to you.

Config is a per-region service, though its data can be aggregated across regions and accounts, and the configuration data can be stored in S3 and analyzed with Athena.

Config Rules

You can use the AWS managed rules (over 75 of them) or write custom rules, which must be defined in AWS Lambda — for instance to evaluate whether every EBS disk is of type gp2, or whether every EC2 instance is a t2.micro. Rules are evaluated on each configuration change, at regular intervals, or both.

Exam tip

Config Rules do not prevent actions from happening — there is no deny. They detect and report.

Pricing has no free tier: $0.003 per configuration item recorded per region and $0.001 per rule evaluation per region.

For any recorded resource, Config gives you three views over time, and they answer three different questions: its compliance over time tells you when it drifted out of line with your rules, its configuration over time tells you what it actually looked like at each point, and the CloudTrail API calls on that resource over time tell you which calls produced those changes.

Remediation

Non-compliant resources can be remediated automatically using SSM Automation Documents, either AWS-managed ones or custom documents you write — and a custom document can invoke a Lambda function. You can also set remediation retries for the case where the resource is still non-compliant after the first attempt. The canonical example is an expired IAM access key marked NON_COMPLIANT, which triggers the managed document AWSConfigRemediation-RevokeUnusedIAMUserCredentials with five retries to deactivate it.

Notifications

Use EventBridge to trigger notifications when resources become non-compliant, routing to Lambda, SNS, SQS and so on. Config can also send configuration changes and compliance state notifications directly to SNS — but that channel sends all events, so filter with SNS filtering or on the client side.

12. CloudWatch vs CloudTrail vs Config

Service	Responsibility
CloudWatch	Performance monitoring (metrics, CPU, network) and dashboards; events and alerting; log aggregation and analysis
CloudTrail	Records API calls made in the account by everyone; trails can target specific resources; global service
Config	Records configuration changes, evaluates resources against compliance rules, and gives a timeline of changes and compliance

Real-world example

Take a single Elastic Load Balancer. CloudWatch monitors the incoming connections metric, visualizes error codes as a percentage over time and drives a performance dashboard. Config tracks the load balancer’s security group rules, tracks its configuration changes and enforces that an SSL certificate is always assigned. CloudTrail tells you who made each change, through which API call.

Quick recap

Item	What to remember for the exam
CloudWatch Metrics	Every service has metrics; up to 30 dimensions per metric; custom metrics for what AWS does not publish, such as RAM
Metric Streams	Near-real-time streaming of metrics to Kinesis Data Firehose destinations or third parties, with optional filtering
CloudWatch Logs	Log group (application) contains log streams (instance, file, container); encrypted by default, optional KMS keys; retention from 1 day to 10 years
Logs Insights	Query engine with its own language over stored logs; multi-log-group, multi-account; not real-time
Export vs Subscriptions	CreateExportTask to S3 can take up to 12 hours; for real-time use subscriptions to KDS, KDF or Lambda
Agents	Old Logs Agent sends logs only; Unified Agent adds RAM, disk, netstat and process metrics and is configured centrally via SSM Parameter Store
Alarms	States OK, INSUFFICIENT_DATA, ALARM; targets are EC2 actions, Auto Scaling actions and SNS; composite alarms combine alarms with AND/OR
EC2 Instance Recovery	Alarm on StatusCheckFailed_System; recovery keeps private, public and Elastic IP, metadata and placement group
EventBridge	Schedule or event pattern; archive and replay events; resource-based policies on buses for cross-account aggregation; Schema Registry infers and versions event schemas
Insights family	Container (ECS/EKS/Fargate), Lambda (a layer), Contributor (top-N from logs), Application (automatic dashboards, SageMaker-powered)
CloudTrail	On by default, global; management events logged by default, data events are not; 90-day retention, longer needs S3 plus Athena; Insights detects unusual write activity
AWS Config	Per-region, aggregatable; over 75 managed rules, custom rules in Lambda; cannot deny an action; remediation via SSM Automation Documents
Choosing between them	Performance means CloudWatch; who did what means CloudTrail; what changed and is it compliant means Config