Skip to content

Other

EBS

General Info

  • Persistent Block-Level storage for EC2
  • Automatically replicated within a single AZ (not across regions/zones)
  • Multiple EBS can be attached to 1 EC2, but an EBS volume can only attach to a single instance at one time
  • Use Cases
    • Primary storage for DB
    • Filesystem

EBS Volume Types

Name Description Use Cases IOPS Volume Size THR
Standard HDD
  • Low cost storage is a requirement
  • Charged on data space PROVISIONED, not used
  • Sequential reads
  • Cold workloads where data is accessed infrequently
 1GB - 1TB 90MiB/s
Cold HDD (sc1)
  • Low-cost storage for infrequently accessed data
 250 500GB - 16TB 250 MiB/s
Throughput-optimised HDD (st1)
  • Low-cost solution designed for applications that are throughput intensive
  • Data warehousing / Big data
  • Log processing
  • Video/data streaming
 500 500GB - 16TB 500 MiB/s
General-purpose SSD (gp2)
  • Good balance for performance and cost
  • Disk Performance not critical
  • Charged on data space PROVISIONED, not used
  • Boot Volumes
  • Small/medium sized DBs
  • dev/test
 16,000 1GB - 16TB 128-250 MiB/s
Provisioned IOPS SSD (io1)
  • Best-performing SSD
  • Any application that needs consistent IOPS above 16,000
  • Also charged for provisioned IOPS
  • Database workloads
  • I/O intensive workload
 32,000 - 64,000 4GB - 16TB 500-1000 MiB/s
IOPS Performance - General Purpose SSD

Baseline 3 IOPS/GBprovisioned capped at 10k IOPS:

1TB --> baseline 3K IOPS
5TB --> hit cap at 10k IOPS

if <1TB can BURST to 3K IOPS:

500G --> baseline = 1,500 IOPS
         when not used accumulated as credits
         when heavy traffic --> up to 3K IOPS till credits depleted

Protecting Data

Encryption
  • Native in EBS with KMS (wrapped data key stored alongside volume)
  • Transparent (same IOPS)
  • Occurs on the servers that host the EC2 instance, so data is encrypted in transit and at rest
  • Snapshots from encrypted volumes are automatically encrypted
  • ec2:CreateVolume action paired with ec2:Encrypted condition key can enforce use of encrypted volumes
Backup & Recovery
  • Backup data by taking point-in-time snapshots (incremental backups)
  • Set the root EBS volume to persist after instance termination: DeleteOnTermination = false
  • Data for snapshot
    • Stored in S3 (AWS-controlled)
    • Constrained in the original region
    • To create (restore) in different region, 1st copy snapshot in that region
  • Creating EBS volumes from snapshots
    • Volume created immediately
    • Data loaded lazily
    • If data requested not restored yet → restored upon 1st request
    • Best practice to access all blocks when initialising

EFS

Use cases
  • Shared storage → appropriate for large data sets of larger file sizes
  • Petabyte-scale NFS filesystem → no need to provision size
  • NFS v4.0 and 4.1 (NFSv4)

Characteristics
  • Stores data at a region scope, replicating that data into at least 3 AZs within the region → < availability, > durability than EBS
  • Can be configured with either General Purpose or MaxIO performance modes
  • Mount targets appear as Endpoints in a VPC → Security Groups can control access
  • Charged for data used
Type THR Notes
EFS File system resource 10GB/s+
EFS General Purpose performance mode 35000 READ, 7000 WRITE IOPS Lowest metadata latency
EFS MaxIO performance mode 500k+ IOPS Highest metadata latency
Security
  • IAM only used for administration
  • Standard posix permissions
  • Encryption
    • EFS file systems must be encrypted at resource creation (check the Encrypted File System option)
    • Transparent encryption at rest with KMS
    • NFS over TLS is an option with the EFS mount helper (stunnel)

Backup

Use cases
  • Centralise backups across RDS, DynamoDB, EBS, EFS, Storage Gateway
    • Filesystem and file level
    • Backups in the same region (AWS Data can be used for cross-region copy)
  • Uses those services' native capabilities (snapshots etc)
  • Resources: plans, vaults, recovery points
Characteristics
  • Encryption
    • can be encrypted in transit and at rest
    • for EFS, where the backup functionality comes from Backup itself, it uses KMS
    • for others, uses the service's native encryption capabilities
  • Resource-based policy for vaults
    • these only constrain vault access
    • not access to the underlying backup like an EBS or RDS snapshot

Storage Gateway

Description
  • Hybrid storage service that enables on-premises virtual appliances to seamlessly use AWS cloud storage
  • Appliance = VM image to install on a host in the data center

Configurations
Gateway-Cached Volumes (GCV)
  • Allows to expand local storage capacity into S3
  • All data stored on a CGV is moved to S3, while recently read data is retained locally to provide low latency
  • Size
    • 1 volume = max 32TB
    • 1 gateway = up to 32 volumes → max storage = 1PB
  • Backup via snapshots (incremental)
  • Security
    • Native encryption: SSL/SSE (server-side encryption)
    • Cannot directly access this data with S3 API
Gateway-Stored Volumes (GSV)
  • Allows to store data on your on-premises storage and async backup data to S3
  • Backups as EBS snapshots (incremental)
  • Size
    • 1 volume = max 16TB
    • 1 gateway = up to 32 volumes → max storage = 512TB
  • Security
    • Native encryption: SSL/SSE (server-side encryption)
    • Cannot directly access this data with S3 API
File Gateway
  • Provides an NFS-style protocol for transferring data to and from the gateway
  • Best suited for backup scenarios
Gateway Virtual Tape Libraries (GVTL)
  • Lets you leverage tape-based backup infrastructure to store data on virtual tape cartridges that you create on your GVTL
  • Ideal for replacing off-site tape directories
  • When tape SW ejects a tape → archived on a VIRTUAL TAPE SHELF (VTS) and stored in Glacier
  • Size
    • 1 gateway = up to 1500 tapes → max storage = 1PB
    • 1! VTS per region
    • Multiple gateways in the same region can share VTS