Skip to content

S3 & Glacier

S3

General Info

Object storage

Features

Feature Description
Regionality
  • Buckets are global (no region or account ID in their ARN)
  • Automatically replicated within a region
Data Consistency
  • PUT to new object = read-after-write consistency
  • PUT to existing object = eventual consistency
  • DELETE object = eventual consistency
MFA Delete
  • Applies to deleting objects, not buckets
  • Only the root account can enable MFA Delete
Versioning
  • The bucket owner, root account, and all authorized IAM users of a bucket are allowed to enable versioning
  • Once enabled, it is not possible to disable or turn off versioning (it can only be suspended)
S3 Transfer Acceleration Speeds up transfers to S3
Limits
  • S3 Per-prefix performance
    • PUT/POST/COPY/DELETE: 3500/sec
    • GET/HEAD: 5500/sec
  • S3 Latency performance: 100-200ms for first-byte-out

Naming

Use Case Naming Example
Direct bucket access bucket name after FQDN (path-style) https://s3.us-east-2.amazonaws.com/testBucket/images
Website hosting bucket name part of FQDN (virtual-hosted-style) https://testBucket.s3.us-east-2.amazonaws.com/images

Storage Classes

Class Description Availability Durability
S3 Standard Frequently accessed data 99.99 11 9s
S3 Intelligent-Tiering
  • Transitions objects between S3 Standard and S3-IA, depending on how often an object is accessed
  • Monitoring and automation charges
99.99 11 9s
S3 Standard-IA
  • Infrequently accessed data (same performance)
  • Stored for at least 30days
  • Charges all objects as if they are at least 128 KB in size
99.9 11 9s
S3 One Zone-IA
  • Reduced Redundancy Storage
  • Lower durability for derived data that can be replaced
99.5 11 9s
Glacier
  • Long term archives
  • Retrieve time of several hours (3-5)
  • Issue RESTORE command → 5h later the object is copied into S3 RRS (original remains in Glacier)
99.99 11 9s
Glacier Deep Archive 12-48 hours latency 99.99 11 9s

Access Control

The account (not user) that creates objects/buckets owns them, even if the bucket is hosted by a different account

Contexts

Context Description
User needs to have permission Using identity policies (or user is the root of an account)
For bucket operations: bucket needs to have permission
  • User in a different account → just bucket policy/ACL
  • User in the same account → both bucket policy/ACL and identity policy
For object operations: User has to have permission (or be root)
  • Bucket policy/ACL has to not deny
  • Object ACL (or bucket policy) has to allow

Grant Access

Option Description
IAM Policies
  • Can only grant users within your AWS account
S3 ACLs
  • Legacy: bucket and object resource-based policy
  • If you need to define object-specific permissions you must use ACLs
  • Default ACL grants the owner account full control
  • List of grants, each grant gives a grantee (an AWS account or predefined group, NOT specific users) a permission
  • Grantee groups:
    • Authenticated Users = any AWS user
    • All Users = include anonymous
    • Log Delivery = S3 audit logs
  • Permissions: READ, WRITE (only applies to buckets - allows overwriting and deleting objects), FULL CONTROL (all of the above)
S3 Bucket policies
  • Recommended: Bucket resource-based policy
  • Permissions are applied at the bucket-level (no object-level granularity)
  • Resource needs to end in "/*"
  • Policies can be attached to: users, groups, S3 buckets
  • Differences with IAM:
    • Associated with bucket, no IAM principal
    • Explicit reference to IAM principal, which can belong to different account (allows to define cross-account access without setting up IAM roles)
Query String Auth
  • Can use query params to provide request info, including auth info
  • This part of URL (named PRE-SIGNED URL) share objects through URLs
Conditions
  • Specified by policy keys
  • Options
    • Request time (DATE Condition)
    • If over SSL (BOOLEAN Condition)
    • Client IP (IP Condition)
    • Client app (STRING Condition)
<?xml version="1.0" encoding="UTF-8"?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
        <ID>*** Owner-Canonical-User-ID ***</ID>
        <DisplayName>owner-display-name</DisplayName>
    </Owner>
    <AccessControlList>
        <Grant>
            <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                     xsi:type="Canonical User">
                <ID>*** Owner-Canonical-User-ID ***</ID>
                <DisplayName>display-name</DisplayName>
            </Grantee>
            <Permission>FULL_CONTROL</Permission>
        </Grant>
    </AccessControlList>
</AccessControlPolicy>
// Allows full read permissions to anonymous users
{
    "Version":"2012-10-17",
    "Statement": [
        {
            "Effect":"Allow",
            "Principal": "*",
            "Action":["s3:GetObject"],
            "Resource":["arn:aws:s3:::my-bucket/*"]
        }
    ]
}

Policy Precedence

  1. Decisions ALWAYS default to DENY
    • If no method specifies an ALLOW —> DENY by default
  2. An EXPLICIT DENY ALWAYS override an ALLOW
  3. Only if no method specify a DENY && 1+ methods specify an ALLOW —> ALLOW
  4. S3 bucket policies has precedence over IAM policy
  • Example
    • IAM policy grants access to an object
    • S3 bucket policy denies access to that object
    • No S3 ACL
    • —> access denied (S3 bucket policy win)
  • Example
    • 1 bucket policy deny all ops to all users (EXPLICIT DENY)
    • 1 bucket policy allowing one specific user to upload
    • -> EXPLICIT DENY ALWAYS OVERRIDE AN ALLOW

Encryption

Server Side

SSE-S3 AWS S3-managed Keys
  • AWS handles key management/protection
  • Each object is encrypted with unique key
  • Object key encrypted with master key
SSE-KMS AWS KMS-managed Keys
  • AWS handles key management/protection but you manage the keys
  • Separate permissions for using the master key
  • Provides auditing
SSE-C Customer-provided Keys
  • AWS only performs encryption/decryption of the object
  • You send the plaintext encryption key in the request
DSSE-KMS Dual-layer encryption with keys stored in KMS
  • Applies two layers of encryption to objects when they are uploaded to an S3 bucket

Client Side

CSE-KMS AWS KMS-managed Customer Master Key
CSE-C Client-side Master Key

Monitoring

  • CloudTrail by default records bucket-level actions
  • Can enable CloudTrail logging of object-level actions by setting that property on a bucket in S3 (can choose read/write)
  • Server access logging: separate audit log, configured per-bucket, that stores events in a bucket

Specific Setups

Force TLS
Add bucket policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowRequests",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::your-s3-bucket/*"
        },
        {
            "Sid": "ForceSSLRequests",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::your-s3-bucket/*",
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        }
    ]
}
Pre-signed URLs
  • Give temporarily access to a specific object (since by default buckets are private)
  • Anyone who has access to a pre-signed URL will have access to the object
  • The illustrated guide to S3 pre-signed URLs
  • Must provide
    • Security credentials (must have the right permissions to access the S3 object)
    • Bucket name
    • Object key
    • Expiration (default 3600s, can be up to 7d)
$ aws s3 presign s3://<bucket>/file.txt --expires-in 300
Block Public Access

Applied to specific buckets, or all buckets in an account

BlockPublicAcls Can't create new public bucket or object ACLs
IgnorePublicAcls Existing (and new) public ACLs are ignored
BlockPublicPolicy Can't create public bucket policies
RestrictPublicBuckets Blocks all anonymous and cross-account access to a bucket
Cross region replication
  • Replicates objects from one region to another in an asynchronous fashion
  • Features
    • You can replicate objects from a source bucket to only one destination
    • After S3 replicates an object, the object cannot be replicated again
    • Enabling cross-region replication does not copy existing objects by default → need to manually copy existing objects
    • Uses SSL by default (no need for policy)
  • Conditions
    • Versioning must be enabled in both source and destination
    • The replication role gives all the necessary permissions to replicate and access the objects on the source bucket
    • If the destination bucket is owned by a different AWS account, the account owner of the destination must grant the required permission to store the replicated objects
      • It is also possible to change object ownership during replication to the destination bucket
    • If the owner of the source bucket does not have permissions on the stored objects, the read and read_acp permissions should be granted (via ACL)
    • Bucket owner must have permissions to access objects (ACLs) to perform the replication
  • What replicated
    • New objects created after the replication is activated
      • Old objects will not be replicated unless they are changed after the replication was activated
    • Unencrypted objects
    • Objects encrypted using SSE-S3 and SSE-KMS (no SSE-C)
    • Object metadata, ACL updates, existing tags, and lock retention information
    • Only object in the source bucket for which the bucket owner has permission to read objects and read ACLs
    • If you perform a delete operation to mark the object as deleted, this marker will be replicated
  • Not replicated
    • Objects created before the CRR was enabled
    • Objects created with SSE-C, or customer-provided encryption keys
    • Objects deleted through a delete operation with a version ID specified (particular object version) → prevents malicious deletion of data


Glacier

Use cases

  • Files stored as Archives, Archives stored in Vaults
  • Data Retrieval requires job initiation then getting the output from the job

Characteristics

Data Storage
  • Automatically encrypts data (AES-256)
  • Regular data integrity checks
  • Only your account can access data
  • Can use IAM to specify which users within the account have rights to operate on a given Vault
Archive
  • Stores data <40TB
  • Each assigned a unique ID
  • Automatically encrypted
  • Immutable (after creation can't be modified)
Vault
  • Container for Archives (can store infinite number)
  • Max 1000 Vaults per region
Vault Lock Policies
  • A vault access policy that can be locked to prevent changes to it
    • configure and enforce compliance controls for individual Glacier Vaults (~ IAM policy)
  • Use cases
    • configure WORM (write once read many) archives
    • create data retention policies
  • 2 steps to configure
    • initiate the lock by attaching a vault lock policy to the vault (sets the lock to an "in-progress" state)
    • you then have 24h to validate the lock policy
    • once validated, lock policies are immutable