Amazon S3 – Advanced Features

1. Moving objects between storage classes

The S3 basics chapter covered the storage classes and their different prices and availability numbers. The next idea is that an object does not have to stay in the class it was uploaded to. You can transition objects from one class to another over time, because data usually cools down: this month’s log files are read constantly, two-year-old log files are opened by nobody.

The deck’s guidance is simple:

• For infrequently accessed objects, move them to Standard-IA.
• For archive objects you do not need fast access to, move them to Glacier or Glacier Deep Archive.

The full set of destinations is Standard, Standard-IA, Intelligent-Tiering, One Zone-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval and Glacier Deep Archive. And rather than doing this by hand, moving objects can be automated using lifecycle rules.

2. S3 Lifecycle Rules

A lifecycle rule attaches to a bucket and performs two kinds of action.

Transition actions configure objects to move to another storage class:

• move objects to Standard-IA 60 days after creation,
• move to Glacier for archiving after 6 months.

Expiration actions configure objects to expire (be deleted) after some time:

• access log files can be set to delete after 365 days,
• can be used to delete old versions of files, if versioning is enabled,
• can be used to delete incomplete multi-part uploads, which otherwise sit there costing money invisibly.

Rules do not have to apply to the whole bucket:

• they can be scoped to a prefix, for example s3://mybucket/mp3/*,
• they can be scoped to object tags, for example Department: Finance.

Scenario 1: thumbnails and source images

The setup: profile photos land in S3, and an EC2 workload derives a thumbnail from each one. Two retention requirements follow. The thumbnails are disposable — regenerating one is cheap — and nothing needs them past the 60-day mark. The originals have to come back instantly during that same 60-day window; once it closes, a wait of as long as 6 hours is acceptable. What storage layout satisfies both?

• Source images go on Standard, with a lifecycle configuration that transitions them to Glacier after 60 days. Glacier’s Standard retrieval of 3–5 hours fits inside the 6-hour tolerance.
• Thumbnails go on One Zone-IA, because they can be recreated, with a lifecycle configuration that expires (deletes) them after 60 days.

Scenario 2: recovering deleted objects

The requirement, written as internal policy: for the first 30 days after something is deleted from S3, getting it back has to be instant — even though anyone actually asking is a rare event. From day 30 out to day 365, the same recovery still has to be possible, but a 48-hour turnaround is good enough.

• Enable S3 Versioning, so that “deleted objects” are really just current versions hidden behind a delete marker, and can be recovered.
• Transition the noncurrent versions to Standard-IA — immediate retrieval, lower cost, matching the “rarely, but instantly” requirement.
• Then transition the noncurrent versions to Glacier Deep Archive, whose 48-hour Bulk retrieval matches the second half of the rule.

Exam tip

Both scenarios hinge on reading the retrieval time tolerance in the question. “Immediately” rules out every Glacier class except Instant Retrieval; “up to 6 hours” fits Glacier Flexible Retrieval Standard; “within 48 hours” is the Deep Archive Bulk giveaway. Also notice scenario 2 applies the rules to noncurrent versions, not to current objects.

3. S3 Analytics – Storage Class Analysis

Before writing lifecycle rules, it helps to know how your data is actually used. S3 Analytics – Storage Class Analysis does that: it helps you decide when to transition objects to the right storage class.

• It produces recommendations for Standard and Standard-IA.
• It does NOT work for One Zone-IA or Glacier.
• The report is a .csv delivered and updated daily, with columns such as date, storage class and object age.
• It takes 24 to 48 hours before you start seeing data.

It is described as a good first step for putting lifecycle rules together, or for improving the ones you already have.

4. S3 Requester Pays

Normally, bucket owners pay for all S3 storage and data transfer costs associated with their bucket. With a Requester Pays bucket, the requester — not the bucket owner — pays the cost of the request and of the data download. The owner still pays for storage.

• Helpful when you want to share large datasets with other accounts without paying their egress bill.
• The requester must be authenticated in AWS — it cannot be an anonymous caller.

Real-world example

A research organization publishes a multi-terabyte genomics dataset. Making the bucket plain-public would mean paying the bandwidth bill for every download in the world. Turning on Requester Pays keeps the data open to any authenticated AWS account while each consumer pays for their own transfers.

5. S3 Event Notifications

S3 can emit events when things happen to objects, and route them to other services.

• Event types include S3:ObjectCreated, S3:ObjectRemoved, S3:ObjectRestore, S3:Replication and more.
• Object name filtering is possible, for example only *.jpg.
• Typical use case: generate thumbnails of images uploaded to S3.
• You can create as many S3 events as you want.
• Events typically arrive in seconds, but can sometimes take a minute or longer — they are not a hard real-time guarantee.

The classic destinations are Lambda functions, SQS queues and SNS topics.

IAM permissions for event notifications

The permission model here is resource-based, and it runs in the direction people do not expect: the destination grants S3 the right to send to it.

• A Lambda resource policy allows S3 to invoke the function.
• An SNS resource (access) policy allows S3 to publish to the topic.
• An SQS resource (access) policy allows S3 to send to the queue.

Exam tip

You do not attach an IAM role to the bucket for event notifications. If the answer options include “create an IAM role for S3”, prefer the one that adds a resource policy on the destination (Lambda, SNS or SQS).

With Amazon EventBridge

S3 can instead send all events to Amazon EventBridge, which unlocks a bigger routing engine:

• Advanced filtering with JSON rules, on metadata, object size, name and so on.
• Multiple destinations — over 18 AWS services, including Step Functions and Kinesis Data Streams / Firehose.
• EventBridge capabilities: archive, replay events, and reliable delivery.

6. S3 baseline performance

Amazon S3 automatically scales to high request rates, with a latency of 100–200 ms.

The numbers to memorize:

• At least 3,500 PUT/COPY/POST/DELETE requests per second per prefix in a bucket.
• At least 5,500 GET/HEAD requests per second per prefix.
• There is no limit to the number of prefixes in a bucket.

The prefix is the path between the bucket name and the file name:

bucket/folder1/sub1/file   => prefix /folder1/sub1/
bucket/folder1/sub2/file   => prefix /folder1/sub2/
bucket/1/file              => prefix /1/
bucket/2/file              => prefix /2/

If you spread reads evenly across all four prefixes, you reach 22,000 GET/HEAD requests per second (4 × 5,500).

Exam tip

The limit is per prefix, not per bucket. A question describing a throttled workload whose objects all sit under one prefix is answered by spreading keys across more prefixes, not by requesting a quota increase and not by creating more buckets.

7. Optimizing upload and download performance

Multi-part upload

• Recommended for files larger than 100 MB, and mandatory for files larger than 5 GB.
• It splits a big file into parts that are uploaded in parallel, which speeds up the transfer and makes a failure cheap to retry.

S3 Transfer Acceleration

• It increases transfer speed by sending the file to an AWS edge location, which then forwards the data to the S3 bucket in the target region over the AWS private network.
• The public-Internet portion of the trip is short (client to nearby edge), and the long-haul portion runs on AWS’s own network.
• It is compatible with multi-part upload, so the two are commonly combined.

Byte-range fetches

This one is about downloads. You parallelize GETs by requesting specific byte ranges of an object.

• Splitting a file into parts and requesting them in parallel speeds up downloads, and gives better resilience in case of failures because only the failed range needs retrying.
• It can also be used to retrieve only partial data, for example just the header — the first XX bytes — of a file.

Real-world example

A media company ingests 40 GB video masters from studios worldwide. Uploads use Transfer Acceleration plus multi-part upload. Their catalog service only needs each file’s metadata header, so instead of downloading whole objects it issues a byte-range fetch for the first few kilobytes.

8. S3 Batch Operations

S3 Batch Operations performs bulk operations on existing S3 objects with a single request. Examples from the deck:

• modify object metadata and properties,
• copy objects between S3 buckets,
• encrypt un-encrypted objects,
• modify ACLs and tags,
• restore objects from S3 Glacier,
• invoke a Lambda function to perform a custom action on each object.

A job consists of a list of objects, the action to perform, and optional parameters. S3 Batch Operations then manages retries, tracks progress, sends completion notifications and generates reports — which is the real reason to use it rather than scripting a loop.

To build the object list, you can use S3 Inventory to produce a report of your objects, and Athena to query and filter that report down to the set you actually want to act on.

9. S3 Storage Lens

S3 Storage Lens helps you understand, analyze and optimize storage across an entire AWS Organization.

• It discovers anomalies, identifies cost efficiencies and applies data-protection best practices across the organization, using 30 days of usage and activity metrics.
• It can aggregate data for the organization, specific accounts, regions, buckets or prefixes.
• You get a default dashboard and can create your own dashboards.
• It can be configured to export metrics daily to an S3 bucket, in CSV or Parquet.

Default dashboard

• Visualizes summarized insights and trends for both free and advanced metrics.
• Shows multi-region and multi-account data.
• It is preconfigured by Amazon S3, and cannot be deleted — but it can be disabled.

Metric categories

Category	What it covers	Example metrics
Summary	General insights about your storage; identify the fastest-growing or unused buckets and prefixes	StorageBytes, ObjectCount
Cost-Optimization	Managing and reducing storage cost; find incomplete multipart uploads older than 7 days, or objects to move to a cheaper class	NonCurrentVersionStorageBytes, IncompleteMultipartUploadStorageBytes
Data-Protection	Whether buckets follow data-protection best practices	VersioningEnabledBucketCount, MFADeleteEnabledBucketCount, SSEKMSEnabledBucketCount, CrossRegionReplicationRuleCount
Access-management	S3 Object Ownership settings in use	ObjectOwnershipBucketOwnerEnforcedBucketCount
Event	Which buckets have S3 Event Notifications configured	EventNotificationEnabledBucketCount
Performance	S3 Transfer Acceleration adoption	TransferAccelerationEnabledBucketCount
Activity	How your storage is requested	AllRequests, GetRequests, PutRequests, ListRequests, BytesDownloaded
Detailed Status Code	HTTP status codes returned	200OKStatusCount, 403ForbiddenErrorCount, 404NotFoundErrorCount

Free vs paid

• Free metrics are automatically available for all customers, contain around 28 usage metrics, and are queryable for 14 days.
• Advanced Metrics and Recommendations are paid, and add Activity, Advanced Cost Optimization, Advanced Data Protection and Status Code metrics, CloudWatch publishing (access the metrics in CloudWatch at no additional charge), and prefix aggregation (collect metrics at the prefix level). Data is queryable for 15 months.

Exam tip

Do not confuse the three analysis tools. Storage Class Analysis answers “which objects should change class in this bucket”. S3 Inventory produces the object list. Storage Lens is the organization-wide dashboard across accounts, regions and buckets.

Quick recap

Item	What to remember for the exam
Transitions	Cold objects go to Standard-IA; archive to Glacier / Deep Archive; automate with lifecycle rules
Lifecycle rules	Transition and expiration actions; can delete old versions and incomplete multi-part uploads; scoped by prefix or tag
Scenario reflex	Match the retrieval time tolerance in the question to the class; noncurrent-version rules need versioning
Storage Class Analysis	Recommends for Standard and Standard-IA only, not One Zone-IA or Glacier; daily .csv, 24–48 h before data appears
Requester Pays	Requester pays request + download, owner still pays storage; requester must be authenticated
Event Notifications	ObjectCreated, ObjectRemoved, ObjectRestore, Replication; name filtering; destinations Lambda / SQS / SNS via their resource policies; delivery in seconds, sometimes a minute or more
EventBridge	All events, JSON advanced filtering, 18+ destinations, archive / replay / reliable delivery
Baseline performance	3,500 write and 5,500 read requests per second per prefix; unlimited prefixes; 4 prefixes evenly used = 22,000 GET/s; latency 100–200 ms
Multi-part upload	Recommended > 100 MB, mandatory > 5 GB
Transfer Acceleration	Client to edge location over public Internet, then AWS private network to the bucket’s region; combines with multi-part
Byte-range fetches	Parallel ranged GETs to speed up downloads, add failure resilience, or fetch only a file header
Batch Operations	Bulk copy, encrypt, retag, restore from Glacier, or invoke Lambda; handles retries, progress, notifications, reports; object list from S3 Inventory + Athena
Storage Lens	Org-wide analysis, 30 days of metrics, aggregate by org/account/region/bucket/prefix; default dashboard cannot be deleted, only disabled
Storage Lens tiers	Free: ~28 usage metrics, 14 days retention. Paid: advanced metrics, CloudWatch publishing, prefix aggregation, 15 months retention