Skip to content

S3 & Glacier

S3

General Info

Object storage

Features

Feature Description
Regionality
  • Buckets are global (no region or account ID in their ARN)
  • Automatically replicated within a region
Data Consistency
  • PUT to new object = read-after-write consistency
  • PUT to existing object = eventual consistency
  • DELETE object = eventual consistency
MFA Delete
  • Applies to deleting objects, not buckets
  • Only the root account can enable MFA Delete
Versioning
  • The bucket owner, root account, and all authorized IAM users of a bucket are allowed to enable versioning
  • Once enabled, it is not possible to disable or turn off versioning (it can only be suspended)
S3 Transfer Acceleration Speeds up transfers to S3
Limits
  • S3 Per-prefix performance
    • PUT/POST/COPY/DELETE: 3500/sec
    • GET/HEAD: 5500/sec
  • S3 Latency performance: 100-200ms for first-byte-out

Naming

Use Case Naming Example
Direct bucket access bucket name after FQDN (path-style) https://s3.us-east-2.amazonaws.com/testBucket/images
Website hosting bucket name part of FQDN (virtual-hosted-style) https://testBucket.s3.us-east-2.amazonaws.com/images

Storage Classes

Class Description Availability Durability
S3 Standard Frequently accessed data 99.99 11 9s
S3 Intelligent-Tiering
  • Transitions objects between S3 Standard and S3-IA, depending on how often an object is accessed
  • Monitoring and automation charges
 99.99 11 9s
S3 Standard-IA
  • Infrequently accessed data (same performance)
  • Stored for at least 30days
  • Charges all objects as if they are at least 128 KB in size
 99.9 11 9s
S3 One Zone-IA
  • Reduced Redundancy Storage
  • Lower durability for derived data that can be replaced
 99.5 11 9s
Glacier
  • Long term archives
  • Retrieve time of several hours (3-5)
  • Issue RESTORE command → 5h later the object is copied into S3 RRS (original remains in Glacier)
 99.99 11 9s
Glacier Deep Archive 12-48 hours latency 99.99 11 9s

Access Control

The account (not user) that creates objects/buckets owns them, even if the bucket is hosted by a different account

Contexts

Context Description
User needs to have permission Using identity policies (or user is the root of an account)
For bucket operations: bucket needs to have permission
  • User in a different account → just bucket policy/ACL
  • User in the same account → both bucket policy/ACL and identity policy
For object operations: User has to have permission (or be root)
  • Bucket policy/ACL has to not deny
  • Object ACL (or bucket policy) has to allow

Grant Access

Option Description
IAM Policies
  • Can only grant users within your AWS account
S3 ACLs
  • Legacy: bucket and object resource-based policy
  • Default ACL grants the owner account full control
  • List of grants, each grant gives a grantee (an AWS account or predefined group, NOT specific users) a permission
  • Grantee groups:
    • Authenticated Users = any AWS user
    • All Users = include anonymous
    • Log Delivery = S3 audit logs
  • Permissions: READ, WRITE (only applies to buckets - allows overwriting and deleting objects), FULL CONTROL (all of the above)
S3 Bucket policies
  • Recommended: Bucket resource-based policy
  • Add/Deny permissions across some or all objects in one bucket
  • Resource needs to end in "/*"
  • Policies can be attached to: users, groups, S3 buckets
  • Differences with IAM:
    • Associated with bucket, no IAM principal
    • Explicit reference to IAM principal, which can belong to different account
Query String Auth
  • Can use query params to provide request info, including auth info
  • This part of URL (named PRE-SIGNED URL) share objects through URLs
Conditions
  • Specified by policy keys
  • Options
    • Request time (DATE Condition)
    • If over SSL (BOOLEAN Condition)
    • Client IP (IP Condition)
    • Client app (STRING Condition)

Policy Precedence (IAM Policy / S3 Bucket Policy / S3 ACL)

  1. Decisions ALWAYS default to DENY
    • If no method specifies an ALLOW —> DENY by default
  2. An EXPLICIT DENY ALWAYS override an ALLOW
  3. Only if no method specify a DENY && 1+ methods specify an ALLOW —> ALLOW
  4. S3 bucket policies has precedence over IAM policy
  • Example
    • IAM policy grants access to an object
    • S3 bucket policy denies access to that object
    • No S3 ACL
    • —> access denied (S3 bucket policy win)
  • Example
    • 1 bucket policy deny all ops to all users (EXPLICIT DENY)
    • 1 bucket policy allowing one specific user to upload
    • -> EXPLICIT DENY ALWAYS OVERRIDE AN ALLOW

Encryption

Server Side
SSE-S3 AWS S3-managed Keys
  • AWS handles key management/protection
  • Each object is encrypted with unique key
  • Object key encrypted with master key
SSE-KMS AWS KMS-managed Keys
  • AWS handles key management/protection but you manage the keys
  • Separate permissions for using the master key
  • Provides auditing
SSE-C Customer-provided Keys
  • AWS only performs encryption/decryption of the object
  • You send the plaintext encryption key in the request

Client Side
CSE-KMS AWS KMS-managed Customer Master Key
CSE-C Client-side Master Key

Monitoring

  • CloudTrail by default records bucket-level actions
  • Can enable CloudTrail logging of object-level actions by setting that property on a bucket in S3 (can choose read/write)
  • Server access logging: separate audit log, configured per-bucket, that stores events in a bucket

Specific Setups

Force TLS
Add bucket policy 
"Condition": {
    "Bool": {
        { "aws:SecureTransport": false } }}
Pre-signed URLs
Give access (temporarily) to a specific object since by default buckets are private 
$ aws s3 presign s3://<bucket>/file.txt —expires-in 300 # 300 seconds (default is 1h)
Block Public Access

Applied to specific buckets, or all buckets in an account
BlockPublicAcls Can't create new public bucket or object ACLs
IgnorePublicAcls Existing (and new) public ACLs are ignored
BlockPublicPolicy Can't create public bucket policies
RestrictPublicBuckets Blocks all anonymous and cross-account access to a bucket
Cross region replication
  • Features
    • Uses SSL by default (no need for policy)
    • You can replicate objects from a source bucket to only one destination
    • After S3 replicates an object, the object cannot be replicated again
    • Enabling cross-region replication does not copy existing objects by default → need to manually copy existing objects
  • Conditions
    • Object owner must grant the bucket owner the READ and READ_ACP permissions via the object ACL
    • The owner of the destination bucket must grant the owner of the source bucket permissions to replicate objects with a bucket policy
    • Versioning must be enabled
  • What replicated
    • Unencrypted objects
    • Objects encrypted using SSE-S3 and SSE-KMS (no SSE-C)
    • Object metadata
    • Any object ACL updates
    • Any object tags
    • Only object in the source bucket for which the bucket owner has permission to read objects and read ACLs
    • NOT REPLICATED deletes to a particular VERSION of an object (security mechanism)

Glacier

Use cases
  • Files stored as Archives, Archives stored in Vaults
  • Data Retrieval requires job initiation then getting the output from the job
Characteristics
Data Storage
  • Automatically encrypts data (AES-256)
  • Regular data integrity checks
  • Only your account can access data
  • Can use IAM to specify which users within the account have rights to operate on a given Vault
Archive
  • Stores data <40TB
  • Each assigned a unique ID
  • Automatically encrypted
  • Immutable (after creation can't be modified)
Vault
  • Container for Archives (can store infinite number)
  • Max 1000 Vaults per region
Vault Lock Policies
  • A vault access policy that can be locked to prevent changes to it
    • configure and enforce compliance controls for individual Glacier Vaults (~ IAM policy)
  • Use cases
    • configure WORM (write once read many) archives
    • create data retention policies
  • 2 steps to configure
    • initiate the lock by attaching a vault lock policy to the vault (sets the lock to an "in-progress" state)
    • you then have 24h to validate the lock policy
    • once validated, lock policies are immutable

  1. S3 Encryption ↩

  2. IAM Policies and Bucket Policies and ACLs! Oh, My! (Controlling Access to S3 Resources) ↩

Back to top