Integration & Messaging – SQS, SNS, Kinesis

1. Why applications need decoupling

As soon as you deploy more than one application, those applications inevitably have to talk to each other. The deck reduces that conversation to two patterns:

1. Synchronous communication – one application calls another directly. A Buying Service invokes a Shipping Service and waits for the answer.
2. Asynchronous / event-based communication – the producing application writes into a queue, and the consuming application reads from that queue at its own pace. The same Buying Service puts the order on a queue and the Shipping Service picks it up later.

Synchronous calls look simple until traffic spikes. The deck’s illustration is video encoding: you normally encode 10 videos, and suddenly you need to encode 1,000. With a direct call the downstream service is overwhelmed, because it has to absorb the spike at exactly the rate the caller produces it.

The fix is to decouple the two sides so they no longer share a fate:

• SQS gives you a queue model.
• SNS gives you a publish/subscribe model.
• Kinesis gives you a real-time streaming model.

The reason this works is that all three services scale independently of your application — the buffer in the middle absorbs the spike while your consumers catch up.

2. Amazon SQS – Standard Queue

A queue is the simplest of the three shapes: any number of producers send messages into an SQS queue, and any number of consumers poll that queue to take messages out.

The Standard queue is AWS’s oldest offering — over ten years old — and it is fully managed, existing for exactly one purpose: decoupling applications. The attributes worth memorizing:

• Unlimited throughput and an unlimited number of messages in the queue.
• Message retention is 4 days by default, and 14 days maximum.
• Low latency, under 10 ms on both publish and receive.
• A hard limit of 256 KB per message sent.
• Delivery is at least once, so you can occasionally get duplicate messages.
• Ordering is best effort, so messages can arrive out of order.

Producing messages

Producers push to SQS with the SDK, calling the SendMessage API. Once accepted, the message is persisted in SQS until a consumer deletes it — nothing else removes it except retention expiry (4 days by default, up to 14). A message can be up to 256 KB, and its body is whatever you decide: the deck’s example sends an order to be processed, carrying an order id, a customer id, and any attributes you care about. Standard queues place no throughput ceiling on this.

Consuming messages

Consumers can be anything that can call an API: EC2 instances, plain servers, or AWS Lambda functions. The consumption cycle has three steps:

1. Poll SQS for messages, receiving up to 10 messages at a time.
2. Process them — the deck’s example is inserting the message into an RDS database.
3. Delete them with the DeleteMessage API.

The third step is not optional bookkeeping. A message that is never deleted comes back.

Scaling consumers horizontally

Nothing stops several EC2 instances from polling the same queue. They receive and process messages in parallel, each one deleting what it has finished with. The consequences are the ones already listed: delivery is at least once and ordering is best effort. The upside is that you scale consumers horizontally to increase processing throughput — the queue itself never becomes the bottleneck.

Exam tip

When a question says “messages may be processed more than once” or “the order of processing does not matter”, it is describing a Standard queue. When it insists on strict ordering or exactly-once sending, the answer is FIFO, and you accept the throughput limits that come with it.

3. SQS with an Auto Scaling Group

The natural way to scale consumers is to let the queue depth drive the group. SQS publishes a CloudWatch metric for queue length, ApproximateNumberOfMessages. You attach a CloudWatch Alarm to that metric, and the alarm’s breach triggers a scaling action on an Auto Scaling Group of EC2 instances that poll the queue.

The loop is self-correcting: a backlog builds, the metric rises, the alarm fires, more instances poll, the backlog drains, and the group scales back down.

4. Architectures built on SQS

Decoupling application tiers

A front-end web application takes requests and calls SendMessage; behind an infinitely scalable queue, a back-end processing application calls ReceiveMessages. Both tiers sit in their own Auto Scaling Group, and neither has to match the other’s capacity.

SQS as a buffer in front of a database

Without a queue, an auto-scaling application inserting transactions straight into Amazon RDS, Amazon Aurora or Amazon DynamoDB will lose transactions when the load is too big — the database simply cannot absorb the write rate.

With a queue in between, the front tier enqueues each transaction, and a separate auto-scaling tier dequeues and inserts at whatever rate the database can sustain. The queue absorbs the spike instead of the database rejecting it.

Real-world example

A flash sale drives ten times the usual order rate for twenty minutes. Order submission writes to SQS and returns immediately, so nobody sees an error page; the fulfillment workers drain the backlog over the following hour. Nothing was lost and no tier had to be sized for the peak.

5. Amazon SQS security

Encryption comes in three layers:

• In-flight encryption through the HTTPS API.
• At-rest encryption using KMS keys.
• Client-side encryption, if the client wants to handle encryption and decryption itself.

Access control has two layers as well. IAM policies regulate who may call the SQS API, and SQS Access Policies — resource policies, conceptually the same thing as S3 bucket policies — are attached to the queue. Those access policies are what you reach for in two situations: granting cross-account access to a queue, and allowing other AWS services such as SNS or S3 to write into the queue.

6. Message visibility timeout

When a consumer polls a message, that message becomes invisible to other consumers for a period called the message visibility timeout, which is 30 seconds by default. The consumer therefore has 30 seconds to finish processing and delete the message. If the timeout expires first, the message becomes visible in the queue again and is returned to the next ReceiveMessage request.

The behavior reads like a timeline: a first ReceiveMessage returns the message; subsequent requests during the visibility window do not return it; once the window closes, a later request returns it again.

The practical consequences:

• A message not processed within the visibility timeout will be processed twice.
• A consumer that needs more time can call the ChangeMessageVisibility API to extend it.
• If the visibility timeout is very high — hours — and a consumer crashes, reprocessing takes a long time because nothing else may touch the message until the window closes.
• If the visibility timeout is too low — seconds — you get duplicates, because slow-but-healthy consumers lose their messages mid-flight.

Exam tip

Two symptoms map to this setting. “The same message is being processed multiple times” means the visibility timeout is too short, or the consumer forgot ChangeMessageVisibility. “A failed instance’s messages took hours to be picked up again” means the visibility timeout is too long.

7. SQS long polling

When a consumer asks for messages and the queue is empty, it can optionally wait for messages to arrive instead of coming back empty-handed. That is long polling.

Waiting instead of returning empty-handed pays off twice: your application makes far fewer API calls to SQS, and it sees lower latency, because a message that lands mid-wait is handed straight over rather than waiting for the next poll cycle.

The wait time is configurable between 1 and 20 seconds, and 20 seconds is preferable. Long polling is preferable to short polling in general. You enable it at the queue level, or per request at the API level with the WaitTimeSeconds parameter.

8. Amazon SQS – FIFO queue

FIFO means First In First Out: messages come out of the queue in the order they went in, and the consumer processes 1, 2, 3, 4 in that order.

What you trade for that guarantee:

• Limited throughput: 300 messages per second without batching, 3,000 messages per second with batching.
• Exactly-once send capability, achieved by removing duplicates using a Deduplication ID.
• Messages are processed in order by the consumer.
• Ordering is scoped by Message Group ID — all messages sharing a group ID are ordered relative to each other — and this parameter is mandatory.

Exam tip

Remember the two numbers together with the two mechanisms: 300/3,000 msg/s, Deduplication ID for exactly-once, Message Group ID for ordering. A scenario demanding both strict ordering and very high throughput is a trap — FIFO cannot give you both.

9. Amazon SNS – the pub/sub model

SQS answers “one message, one consumer”. SNS exists for the opposite need: one message that has to reach many recipients at once.

Without SNS, a Buying Service has to integrate directly with each destination — an email notification, a fraud service, a shipping service, an SQS queue — and every new destination means changing the producer. With SNS, the event producer sends the message to a single SNS topic, and as many event receivers (subscriptions) as you like listen to that topic.

Each subscriber to the topic receives every message published to it, although message filtering (below) can narrow that down. The scale limits are worth remembering: up to 12,500,000 subscriptions per topic, and a limit of 100,000 topics.

Subscribers can be SQS queues, Lambda functions, Kinesis Data Firehose, HTTP(S) endpoints, SMS and mobile notifications, or email.

SNS integrates with many AWS services

A long list of AWS services can hand data straight to a topic for notification, which is why SNS so often turns out to be plumbing you never had to write. The publishers the deck draws: CloudWatch Alarms; S3 Bucket (Events); Auto Scaling Group (Notifications); CloudFormation (State Changes); AWS Budgets; Lambda; AWS DMS (New Replic); DynamoDB; RDS Events — and the slide’s trailing ellipses say the list is not exhaustive.

Two ways to publish

Topic publish, using the SDK — the ordinary path:

1. Create a topic.
2. Create one or more subscriptions.
3. Publish to the topic.

Direct publish, for mobile app SDKs:

1. Create a platform application.
2. Create a platform endpoint.
3. Publish to the platform endpoint.

Direct publish works with Google GCM, Apple APNS, Amazon ADM and similar push platforms.

SNS security

SNS security mirrors SQS almost line for line. Encryption is in-flight via the HTTPS API, at rest using KMS keys, and client-side if the client prefers to do it itself. Access control is IAM policies over the SNS API plus SNS Access Policies on the topic, which are what you use for cross-account access to topics and for allowing other services such as S3 to write to a topic.

10. SNS + SQS: the fan-out pattern

Fan-out is the combination the exam loves: publish once into SNS, and every subscribing SQS queue receives a copy.

Why it is a good pattern:

• The architecture is fully decoupled, with no data loss — SNS alone does not persist messages, but the SQS queues do.
• SQS adds data persistence, delayed processing and retries of work.
• You can add more SQS subscribers over time without touching the producer.
• Cross-region delivery works: the subscribing queues can live in other regions.

The one operational detail: make sure the SQS queue access policy allows SNS to write to it.

Application: S3 events to multiple queues

S3 imposes a structural limit: a given event type paired with a given prefix — object created under images/, say — supports exactly one S3 event rule. So when that same event has to land in several SQS queues, declaring extra rules is not an option. The single rule you are allowed points at an SNS topic, and the topic fans the event out to the queues, and to a Lambda function too if you want one.

Getting topic messages into S3 via Firehose

SNS can send to Kinesis Data Firehose, which unlocks a useful pipeline: a Buying Service publishes to an SNS topic, the topic delivers to Amazon Data Firehose, and Firehose writes to Amazon S3 — or to any other supported Firehose destination.

SNS FIFO topics

SNS also has FIFO topics, which order the messages in the topic the way SQS FIFO orders messages in the queue. The features mirror SQS FIFO:

• Ordering by Message Group ID.
• Duplicates removed either from an explicit Deduplication ID or from the message body itself (content-based deduplication).
• SQS Standard and FIFO queues can both be subscribers.
• Limited throughput, the same throughput as SQS FIFO.

Combining an SNS FIFO topic with SQS FIFO queues is the answer when you need fan-out and ordering and deduplication at the same time.

SNS message filtering

A JSON filter policy attached to a subscription decides which messages that subscription receives. A subscription without a filter policy receives every message.

The deck’s example publishes a new transaction carrying a State attribute. One SQS queue has a filter policy for State: Placed, another for State: Declined, an email subscription for State: Cancelled, and a further queue with no policy at all receives everything.

Real-world example

An order topic feeds four consumers. Fulfillment only cares about placed orders, the risk team only about declined ones, customer service is emailed about cancellations, and the analytics queue takes all of them. One publish, four filter policies, no branching logic in the producer.

11. Amazon Kinesis Data Streams

Kinesis Data Streams collects and stores streaming data in real time. Its producers are things like click streams, IoT devices, metrics and logs, and the Kinesis Agent; its consumers are applications, Lambda, Amazon Data Firehose, and Managed Service for Apache Flink.

The properties that distinguish it from SQS and SNS:

• Retention of up to 365 days.
• Consumers can reprocess (replay) data, because reading does not consume it.
• Data cannot be deleted from Kinesis; it goes away when it expires.
• Records are up to 1 MB — the typical use case is a lot of small real-time records.
• Ordering is guaranteed for data sharing the same Partition ID.
• At-rest KMS encryption and in-flight HTTPS encryption.
• The Kinesis Producer Library (KPL) helps you write an optimized producer, and the Kinesis Client Library (KCL) an optimized consumer.

Capacity modes

Provisioned mode — you choose the number of shards:

• Each shard takes 1 MB/s in, or 1,000 records per second.
• Each shard gives 2 MB/s out.
• You scale manually by adding or removing shards.
• You pay per shard provisioned per hour.

On-demand mode — no capacity to provision or manage:

• Default provisioned capacity is 4 MB/s in, or 4,000 records per second.
• It scales automatically based on the observed throughput peak during the last 30 days.
• You pay per stream per hour plus data in/out per GB.

Exam tip

“Replay the data” and “retain for up to 365 days” are Kinesis Data Streams giveaways — neither SQS nor SNS nor Firehose can do it. If a question gives you a steady, predictable record rate, provisioned mode is cheaper; if the workload is unpredictable or brand new, on-demand is the safe answer.

12. Amazon Data Firehose

Amazon Data Firehose — formerly called Kinesis Data Firehose, and the exam may still use the old name — is a fully managed delivery service. Producers (applications, clients, the Kinesis Agent, the SDK, Kinesis Data Streams, Amazon CloudWatch Logs and Events, AWS IoT) push records of up to 1 MB, and Firehose does batch writes to a destination.

Destinations fall into three groups:

• AWS destinations: Amazon S3, Amazon Redshift, Amazon OpenSearch Service.
• Third-party partner destinations: Splunk, MongoDB, Datadog, New Relic and others.
• Custom HTTP endpoints.

The rest of what you need to know:

• Automatic scaling, serverless, pay for what you use.
• Near real-time, with buffering based on size or time — the buffer is why it is “near” real time rather than real time.
• Supports CSV, JSON, Parquet, Avro, raw text and binary data.
• Can convert to Parquet or ORC and compress with gzip or snappy.
• Custom data transformations using AWS Lambda, for example converting CSV to JSON.
• An S3 backup bucket can receive all data or only the failed data.

Kinesis Data Streams vs Amazon Data Firehose

Kinesis Data Streams	Amazon Data Firehose
Streaming data collection	Loads streaming data into S3 / Redshift / OpenSearch / third party / custom HTTP
You write producer and consumer code	Fully managed
Real-time	Near real-time
Provisioned or on-demand mode	Automatic scaling
Data storage up to 365 days	No data storage
Replay capability	No replay capability

13. SQS vs SNS vs Kinesis

This is the comparison to know cold, because exam questions describe a behavior and expect you to name the service:

SQS	SNS	Kinesis
Consumers pull data	Pushes data to many subscribers	Standard: pull data — 2 MB per shard
Consumption removes the message	Up to 12,500,000 subscribers	Enhanced fan-out: push data — 2 MB per shard per consumer
As many workers (consumers) as you want	Data is not persisted — lost if not delivered	Possibility to replay data
Throughput needs no provisioning	Pub/Sub	Built for real-time big data, analytics and ETL
Ordering guarantees only on FIFO queues	Up to 100,000 topics	Ordering at the shard level
Individual message delay capability	No need to provision throughput	Data expires after X days
	Integrates with SQS for the fan-out pattern	Provisioned mode or on-demand capacity mode
	FIFO capability for SQS FIFO

14. Amazon MQ

SQS and SNS are cloud-native services built on proprietary AWS protocols. Traditional applications running on premises usually speak open protocols instead: MQTT, AMQP, STOMP, Openwire, WSS.

When such an application migrates to the cloud, you have two options: re-engineer it to use SQS and SNS, or keep its protocol and run a managed broker. Amazon MQ is the second option — a managed message broker service.

What to remember about it:

• It does not scale nearly as far as SQS or SNS.
• It runs on servers, and can run Multi-AZ with failover.
• It has both queue features (comparable to SQS) and topic features (comparable to SNS).

Amazon MQ high availability

Inside a region such as us-east-1, one Amazon MQ broker runs ACTIVE in one Availability Zone (us-east-1a) and a second runs STANDBY in another (us-east-1b). Both use Amazon EFS as shared storage, so when the active broker fails, the client fails over to the standby and the message state is intact.

Exam tip

The trigger words are the protocol names — MQTT, AMQP, STOMP, Openwire, WSS — and any phrasing about migrating an existing application without rewriting it. Those point to Amazon MQ. A new cloud-native application points to SQS or SNS instead.

Quick recap

Item	What to remember for the exam
Why decouple	Synchronous calls break under spikes; decouple with SQS (queue), SNS (pub/sub), Kinesis (streaming), all of which scale independently of your app
SQS Standard	Unlimited throughput, 256 KB per message, retention 4 days default / 14 days max, latency < 10 ms, at-least-once delivery, best-effort ordering
SQS consuming	Poll up to 10 messages at a time, process, then call DeleteMessage; scale consumers horizontally
Visibility timeout	30 seconds by default; too short means duplicates, too long means slow reprocessing after a crash; extend with ChangeMessageVisibility
Long polling	Wait 1–20 seconds (20 preferred), set at queue level or with WaitTimeSeconds; fewer API calls, lower latency
SQS FIFO	300 msg/s without batching, 3,000 msg/s with; exactly-once send via Deduplication ID; Message Group ID mandatory for ordering
Scaling consumers	CloudWatch metric ApproximateNumberOfMessages drives a CloudWatch Alarm which scales the Auto Scaling Group
SQS security	HTTPS in flight, KMS at rest, client-side optional; IAM policies plus SQS Access Policies for cross-account and service writes
SNS	Up to 12,500,000 subscriptions per topic and 100,000 topics; subscribers include SQS, Lambda, Firehose, HTTP(S), SMS/mobile, email; data is not persisted
SNS publishing	Topic publish via SDK, or direct publish to a platform endpoint for GCM / APNS / ADM
Fan out	One publish to SNS reaches every subscribing SQS queue; allow SNS in the SQS access policy; works cross-region
S3 events	Only one S3 event rule per event type + prefix combination, so use SNS fan-out to reach several queues
SNS FIFO	Ordering by Message Group ID, deduplication by ID or content; combine with SQS FIFO for fan-out plus ordering
SNS filtering	JSON filter policy per subscription; no policy means the subscription receives every message
Kinesis Data Streams	Retention up to 365 days, replay supported, records up to 1 MB, ordering by Partition ID; provisioned = 1 MB/s in (1,000 rec/s) and 2 MB/s out per shard; on-demand = 4 MB/s (4,000 rec/s) default, auto-scales on the 30-day peak
Amazon Data Firehose	Fully managed, near real-time, no storage and no replay; S3 / Redshift / OpenSearch / third party / custom HTTP; Lambda transformations; Parquet/ORC conversion, gzip/snappy compression
Amazon MQ	Managed broker for MQTT, AMQP, STOMP, Openwire, WSS; runs on servers, Multi-AZ active/standby over Amazon EFS; queue and topic features, but far less scalable than SQS/SNS