AWS Storage Extras

1. AWS Snowball

You cannot always push data into the cloud over the network. AWS Snowball is a family of highly secure, portable devices that AWS ships to you to collect and process data at the edge and to migrate data into and out of AWS. They help migrate up to petabytes of data.

Two device models:

Snowball Edge device	Compute	Memory	Storage (SSD)
Snowball Edge Storage Optimized	104 vCPUs	416 GB	210 TB
Snowball Edge Compute Optimized	104 vCPUs	416 GB	28 TB

Why offline migration exists

The problem is not that networks cannot move data, it is how long they take. The deck’s table makes it obvious:

Amount	100 Mbps	1 Gbps	10 Gbps
10 TB	12 days	30 hours	3 hours
100 TB	124 days	12 days	30 hours
1 PB	3 years	124 days	12 days

And those numbers assume an ideal line. Real migrations also face limited connectivity, limited bandwidth, high network cost, shared bandwidth (you can never use the whole line), and connection stability problems.

Hence the rule of thumb: if it would take more than a week to transfer over the network, use Snowball devices.

The two patterns side by side: a direct upload to S3 sends the data from the client straight into the bucket over the Internet; with Snowball, the client copies data onto the device, ships it, and AWS imports it into the S3 bucket (and the same devices support export out of AWS).

Exam tip

“More than a week over the network” is the trigger phrase for Snowball. Watch for questions that give you a bandwidth and a data volume — do the rough arithmetic before choosing between Snowball and Direct Connect or DataSync.

2. Edge computing with Snowball Edge

Edge computing means processing data while it is being created, at the edge location — a truck on the road, a ship at sea, a mining station underground. These places may have limited internet and no access to computing power.

You set up a Snowball Edge device to do edge computing there. The Compute Optimized model is dedicated to this use case, and the Storage Optimized model can do it as well. On the device you can run EC2 instances or Lambda functions at the edge.

Use cases: preprocess data, machine learning, transcoding media — so that only the useful result ever has to travel.

Snowball into Glacier

An architecture point worth memorizing: Snowball cannot import directly into Glacier. You must import into Amazon S3 first, then use an S3 lifecycle policy to move the data into Glacier.

3. Amazon FSx – overview

Amazon FSx lets you launch third-party high-performance file systems on AWS as a fully managed service. There are four:

• FSx for Windows File Server
• FSx for Lustre
• FSx for NetApp ONTAP
• FSx for OpenZFS

FSx for Windows File Server

A fully managed Windows file system share drive.

• Speaks SMB and sits on Windows NTFS.
• Plugs into Microsoft Active Directory, and brings ACLs and user quotas with it.
• Linux EC2 instances can mount it too, not just Windows ones.
• Microsoft’s Distributed File System (DFS) Namespaces is supported, which is how you present files from several file systems as one tree.
• Ceilings are high: tens of GB/s, millions of IOPS, hundreds of PB of data.
• Two storage options. SSD is the latency-sensitive choice — databases, media processing, data analytics. HDD covers the broad middle, home directories and CMS workloads among them.
• Reachable from your own data center, over VPN or Direct Connect.
• Multi-AZ is a configuration option, for high availability.
• A daily backup lands in S3 without you arranging it.

FSx for Lustre

Lustre is a parallel distributed file system for large-scale computing; the name comes from “Linux” and “cluster”.

• Targets machine learning and High Performance Computing (HPC), plus video processing, financial modeling, electronic design automation.
• Scales to hundreds of GB/s, millions of IOPS, sub-millisecond latencies.
• Storage options: SSD for low-latency, IOPS-intensive workloads with small and random file operations; HDD for throughput-intensive workloads with large and sequential file operations.
• Seamless integration with S3: it can “read S3” as a file system through FSx, and write the output of computations back to S3 through FSx.
• Can be used from on-premises servers over VPN or Direct Connect.

Lustre deployment options

	Scratch file system	Persistent file system
Storage	Temporary	Long-term
Replication	Not replicated — data does not persist if the file server fails	Replicated within the same AZ; failed files replaced within minutes
Performance	High burst — 6x faster, 200 MBps per TiB	—
Usage	Short-term processing, optimize costs	Long-term processing, sensitive data

Both deployment options sit in a region with compute instances in one or more AZs, and both can use an S3 bucket as an optional data repository.

Exam tip

Scratch versus Persistent is decided by what happens on failure. “The data can be recomputed, we want it cheap and fast” is Scratch; “the data must survive a file server failure” is Persistent. Note that Persistent replicates within the same AZ, not across AZs.

FSx for NetApp ONTAP

Managed NetApp ONTAP on AWS.

• Three protocols are on offer at once: NFS, SMB and iSCSI.
• The point of it is lifting an existing ONTAP or NAS deployment into AWS unchanged.
• Client coverage is wide — Linux, Windows, macOS, VMware Cloud on AWS, Amazon WorkSpaces and AppStream 2.0, and Amazon EC2, ECS and EKS.
• Capacity follows the data: storage grows and shrinks on its own.
• You get snapshots, replication, compression and data de-duplication, at low cost.
• Cloning is point-in-time and instantaneous — a cheap way to stand up a copy for testing a new workload.

FSx for OpenZFS

Managed OpenZFS file system on AWS.

• One protocol here: NFS, in versions v3, v4, v4.1 and v4.2.
• Aimed at ZFS workloads you want to relocate onto AWS.
• The same client list applies — Linux, Windows, macOS, VMware Cloud on AWS, WorkSpaces and AppStream 2.0, EC2, ECS and EKS.
• Performance reaches 1,000,000 IOPS at under 0.5 ms latency.
• Snapshots and compression are included, it is low cost, and cloning is again point-in-time and instantaneous.

Exam tip

Pick FSx by protocol and heritage: SMB / NTFS / Active Directory → FSx for Windows; HPC, ML, sub-ms, S3 integration → FSx for Lustre; NFS + SMB + iSCSI together, highest OS compatibility, de-duplication → FSx for NetApp ONTAP; NFS only, existing ZFS workloads → FSx for OpenZFS.

4. Hybrid cloud and AWS Storage Gateway

AWS is pushing for hybrid cloud: part of your infrastructure in the cloud, part of it on-premises. That split can be caused by long cloud migrations, security requirements, compliance requirements or simply IT strategy.

This creates a problem. S3 is a proprietary storage technology, unlike EFS or NFS, so how do you expose S3 data on-premises? The answer is AWS Storage Gateway.

It helps to keep the cloud-native storage options in mind first:

Category	Services
Block	Amazon EBS, EC2 Instance Store
File	Amazon EFS, Amazon FSx
Object	Amazon S3, Amazon Glacier

AWS Storage Gateway is the bridge between on-premises data and cloud data. Use cases: disaster recovery, backup and restore, tiered storage, on-premises cache and low-latency file access. There are three types: S3 File Gateway, Volume Gateway and Tape Gateway.

Amazon S3 File Gateway

• Configured S3 buckets become accessible using the NFS and SMB protocols.
• The most recently used data is cached in the file gateway, which is what gives on-premises applications low latency.
• Supports S3 Standard, S3 Standard-IA, S3 One Zone-IA and S3 Intelligent-Tiering; you transition to S3 Glacier using a lifecycle policy.
• Bucket access uses IAM roles for each File Gateway.
• The SMB protocol integrates with Active Directory for user authentication.

Volume Gateway

• Block storage using the iSCSI protocol, backed by S3.
• Backed by EBS snapshots, which can help restore on-premises volumes.
• Cached volumes: low-latency access to the most recent data, the bulk living in AWS.
• Stored volumes: the entire dataset is on premises, with scheduled backups to S3.

Tape Gateway

• Some companies still run backup processes built around physical tapes. Tape Gateway lets them use the same processes in the cloud.
• It presents a Virtual Tape Library (VTL) backed by Amazon S3 and Glacier.
• You back up data using existing tape-based processes over an iSCSI interface.
• It works with leading backup software vendors.

Deployment and storage targets

The gateways run on-premises as a virtual machine on VMware, Hyper-V or KVM, each keeping a local cache, and they talk to AWS over the Internet or Direct Connect with encryption in transit.

Gateway	On-premises protocol	AWS target
File Gateway	NFS / SMB for user and group file shares	Amazon S3, excluding Glacier and Glacier Deep Archive
Volume Gateway	iSCSI from the application server	Amazon S3, with AWS EBS snapshots
Tape Gateway	iSCSI VTL from the backup application	Amazon S3 tape library (any S3 storage class, including Glacier); tapes ejected from the backup application move to the tape archive in Glacier and Glacier Deep Archive

Real-world example

A hospital keeps its imaging archive on an on-premises NAS that is running out of space. They deploy an S3 File Gateway VM: radiologists keep using the same SMB share, recent studies are served from the local cache, and everything is really stored in S3, with a lifecycle policy pushing studies older than a year into Glacier.

5. AWS Transfer Family

AWS Transfer Family is a fully managed service for file transfers into and out of Amazon S3 or Amazon EFS using the FTP protocol family.

Supported protocols:

• AWS Transfer for FTP — File Transfer Protocol.
• AWS Transfer for FTPS — FTP over SSL.
• AWS Transfer for SFTP — Secure File Transfer Protocol.

Other characteristics:

• Managed infrastructure, scalable, reliable, highly available (multi-AZ).
• Pricing is per provisioned endpoint per hour, plus data transfers in GB.
• You can store and manage users’ credentials within the service.
• It integrates with existing authentication systems: Microsoft Active Directory, LDAP, Okta, Amazon Cognito, or a custom provider.
• Usage: sharing files, public datasets, CRM, ERP.

One deployment note from the diagram: of the three, AWS Transfer for FTP is the one marked “only within VPC”. The diagram also shows Route 53 as an optional front end and an IAM role controlling access to the S3 or EFS backend.

6. AWS DataSync

AWS DataSync moves large amounts of data to and from AWS.

• On-premises or another cloud to AWS — over NFS, SMB, HDFS, the S3 API and others. This direction needs an agent.
• AWS to AWS, between different storage services — no agent needed.

It can synchronize to:

• Amazon S3, any storage class including Glacier,
• Amazon EFS,
• Amazon FSx (Windows, Lustre, NetApp, OpenZFS).

Operational details:

• Replication tasks can be scheduled hourly, daily or weekly — it is not a continuous replication service.
• File permissions and metadata are preserved (NFS POSIX, SMB and so on).
• One agent task can use 10 Gbps, and you can set a bandwidth limit.
• The agent also comes pre-installed on AWS Snowcone, which is handy for disconnected sites.

Exam tip

Two distinctions get tested. DataSync preserves file permissions and metadata — that phrase in a question almost always points at DataSync. And DataSync schedules run hourly, daily or weekly; a requirement for continuous, near real-time replication is not DataSync.

7. Comparing every storage service

A single slide ends the section by reducing each service to one line. Nothing else in these chapters is this efficient to revise from.

Service	What it is
S3	Object storage
S3 Glacier	Object archival
EBS volumes	Network storage for one EC2 instance at a time
Instance Store	Physical storage attached to your EC2 instance (high IOPS)
EFS	Network file system for Linux instances, POSIX filesystem
FSx for Windows	Network file system for Windows servers
FSx for Lustre	High Performance Computing Linux file system
FSx for NetApp ONTAP	High OS compatibility
FSx for OpenZFS	Managed ZFS file system
Storage Gateway	S3 and FSx File Gateway, Volume Gateway (cached and stored), Tape Gateway
Transfer Family	FTP, FTPS, SFTP interface on top of Amazon S3 or Amazon EFS
DataSync	Scheduled data sync from on-premises to AWS, or AWS to AWS
Snowcone / Snowball / Snowmobile	Move large amounts of data to the cloud physically
Database	For specific workloads, usually with indexing and querying

Quick recap

Item	What to remember for the exam
Snowball	Portable secure devices, up to petabytes; Storage Optimized 210 TB, Compute Optimized 28 TB, both 104 vCPUs / 416 GB
When to use Snowball	If the transfer would take more than a week over the network
Edge computing	Run EC2 instances or Lambda functions on a Snowball Edge at a disconnected site; preprocessing, ML, media transcoding
Snowball to Glacier	Not supported directly — import to S3 first, then an S3 lifecycle policy into Glacier
FSx for Windows	SMB + NTFS, Active Directory, ACLs, quotas, DFS Namespaces; SSD or HDD; Multi-AZ; daily backup to S3; mountable from Linux and on-premises
FSx for Lustre	HPC and ML, sub-ms latency, hundreds of GB/s; S3 integration both ways; Scratch (not replicated, 6x burst, 200 MBps per TiB) vs Persistent (replicated in the same AZ)
FSx for NetApp ONTAP	NFS, SMB, iSCSI; storage grows and shrinks automatically; de-duplication, compression, instantaneous cloning
FSx for OpenZFS	NFS v3–v4.2; up to 1,000,000 IOPS, < 0.5 ms; snapshots, compression, cloning
Storage Gateway	Hybrid bridge because S3 is proprietary; runs as a VM with a local cache; File / Volume / Tape
S3 File Gateway	NFS and SMB onto S3 (Standard, Standard-IA, One Zone-IA, Intelligent-Tiering); Glacier via lifecycle policy; IAM role per gateway; SMB integrates with AD
Volume Gateway	iSCSI block storage backed by S3 with EBS snapshots; cached (recent data local) vs stored (full dataset local, scheduled backups)
Tape Gateway	Virtual Tape Library over iSCSI, backed by S3 and Glacier; keeps existing tape backup processes
Transfer Family	Managed FTP / FTPS / SFTP in front of S3 or EFS; multi-AZ; per endpoint-hour + data; auth via AD, LDAP, Okta, Cognito, custom; plain FTP is VPC-only
DataSync	Agent for on-premises or other clouds, no agent AWS to AWS; targets S3 (any class), EFS, FSx; hourly / daily / weekly; preserves permissions and metadata; 10 Gbps per task with an optional bandwidth limit