S3 & Glacier

S3

General Info

Object storage

Features

Feature	Description
Regionality	Buckets are global (no region or account ID in their ARN) Automatically replicated within a region
Data Consistency	PUT to new object = read-after-write consistency PUT to existing object = eventual consistency DELETE object = eventual consistency
MFA Delete	Applies to deleting objects, not buckets Only the root account can enable MFA Delete
Versioning	The bucket owner, root account, and all authorized IAM users of a bucket are allowed to enable versioning Once enabled, it is not possible to disable or turn off versioning (it can only be suspended)
S3 Transfer Acceleration	Speeds up transfers to S3
Limits	S3 Per-prefix performance PUT/POST/COPY/DELETE: 3500/sec GET/HEAD: 5500/sec S3 Latency performance: 100-200ms for first-byte-out

Naming

Use Case	Naming	Example
Direct bucket access	bucket name after FQDN (path-style)	https://s3.us-east-2.amazonaws.com/testBucket/images
Website hosting	bucket name part of FQDN (virtual-hosted-style)	https://testBucket.s3.us-east-2.amazonaws.com/images

Storage Classes

Class	Description	Availability	Durability
S3 Standard	Frequently accessed data	99.99	11 9s
S3 Intelligent-Tiering	Transitions objects between S3 Standard and S3-IA, depending on how often an object is accessed Monitoring and automation charges	99.99	11 9s
S3 Standard-IA	Infrequently accessed data (same performance) Stored for at least 30days Charges all objects as if they are at least 128 KB in size	99.9	11 9s
S3 One Zone-IA	Reduced Redundancy Storage Lower durability for derived data that can be replaced	99.5	11 9s
Glacier	Long term archives Retrieve time of several hours (3-5) Issue RESTORE command → 5h later the object is copied into S3 RRS (original remains in Glacier)	99.99	11 9s
Glacier Deep Archive	12-48 hours latency	99.99	11 9s

Access Control

The account (not user) that creates objects/buckets owns them, even if the bucket is hosted by a different account

Contexts

Context	Description
User needs to have permission	Using identity policies (or user is the root of an account)
For bucket operations: bucket needs to have permission	User in a different account → just bucket policy/ACL User in the same account → both bucket policy/ACL and identity policy
For object operations: User has to have permission (or be root)	Bucket policy/ACL has to not deny Object ACL (or bucket policy) has to allow

Grant Access

Option	Description
IAM Policies	Can only grant users within your AWS account
S3 ACLs	Legacy: bucket and object resource-based policy If you need to define object-specific permissions you must use ACLs Default ACL grants the owner account full control List of grants, each grant gives a grantee (an AWS account or predefined group, NOT specific users) a permission Grantee groups: Authenticated Users = any AWS user All Users = include anonymous Log Delivery = S3 audit logs Permissions: READ, WRITE (only applies to buckets - allows overwriting and deleting objects), FULL CONTROL (all of the above)
S3 Bucket policies	Recommended: Bucket resource-based policy Permissions are applied at the bucket-level (no object-level granularity) Resource needs to end in "/*" Policies can be attached to: users, groups, S3 buckets Differences with IAM: Associated with bucket, no IAM principal Explicit reference to IAM principal, which can belong to different account (allows to define cross-account access without setting up IAM roles)
Query String Auth	Can use query params to provide request info, including auth info This part of URL (named PRE-SIGNED URL) share objects through URLs
Conditions	Specified by policy keys Options Request time (DATE Condition) If over SSL (BOOLEAN Condition) Client IP (IP Condition) Client app (STRING Condition)

ACLBucket Policy

<?xml version="1.0" encoding="UTF-8"?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
        <ID>*** Owner-Canonical-User-ID ***</ID>
        <DisplayName>owner-display-name</DisplayName>
    </Owner>
    <AccessControlList>
        <Grant>
            <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                     xsi:type="Canonical User">
                <ID>*** Owner-Canonical-User-ID ***</ID>
                <DisplayName>display-name</DisplayName>
            </Grantee>
            <Permission>FULL_CONTROL</Permission>
        </Grant>
    </AccessControlList>
</AccessControlPolicy>

// Allows full read permissions to anonymous users
{
    "Version":"2012-10-17",
    "Statement": [
        {
            "Effect":"Allow",
            "Principal": "*",
            "Action":["s3:GetObject"],
            "Resource":["arn:aws:s3:::my-bucket/*"]
        }
    ]
}

Policy Precedence

RulesExamples

1. Decisions ALWAYS default to DENY
   • If no method specifies an ALLOW —> DENY by default
2. An EXPLICIT DENY ALWAYS override an ALLOW
3. Only if no method specify a DENY && 1+ methods specify an ALLOW —> ALLOW
4. S3 bucket policies has precedence over IAM policy

• Example
  • IAM policy grants access to an object
  • S3 bucket policy denies access to that object
  • No S3 ACL
  • —> access denied (S3 bucket policy win)
• Example
  • 1 bucket policy deny all ops to all users (EXPLICIT DENY)
  • 1 bucket policy allowing one specific user to upload
  • -> EXPLICIT DENY ALWAYS OVERRIDE AN ALLOW

Encryption

Server Side

SSE-S3	AWS S3-managed Keys	AWS handles key management/protection Each object is encrypted with unique key Object key encrypted with master key
SSE-KMS	AWS KMS-managed Keys	AWS handles key management/protection but you manage the keys Separate permissions for using the master key Provides auditing
SSE-C	Customer-provided Keys	AWS only performs encryption/decryption of the object You send the plaintext encryption key in the request
DSSE-KMS	Dual-layer encryption with keys stored in KMS	Applies two layers of encryption to objects when they are uploaded to an S3 bucket

Client Side

CSE-KMS	AWS KMS-managed Customer Master Key
CSE-C	Client-side Master Key

Monitoring

• CloudTrail by default records bucket-level actions
• Can enable CloudTrail logging of object-level actions by setting that property on a bucket in S3 (can choose read/write)
• Server access logging: separate audit log, configured per-bucket, that stores events in a bucket
• A Guide to S3 Logging

Specific Setups

Force TLS

Add bucket policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowRequests",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::your-s3-bucket/*"
        },
        {
            "Sid": "ForceSSLRequests",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::your-s3-bucket/*",
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        }
    ]
}

Pre-signed URLs

• Give temporarily access to a specific object (since by default buckets are private)
• Anyone who has access to a pre-signed URL will have access to the object
• The illustrated guide to S3 pre-signed URLs
• Must provide
  • Security credentials (must have the right permissions to access the S3 object)
  • Bucket name
  • Object key
  • Expiration (default 3600s, can be up to 7d)

$ aws s3 presign s3://<bucket>/file.txt --expires-in 300

Block Public Access

Applied to specific buckets, or all buckets in an account

BlockPublicAcls	Can't create new public bucket or object ACLs
IgnorePublicAcls	Existing (and new) public ACLs are ignored
BlockPublicPolicy	Can't create public bucket policies
RestrictPublicBuckets	Blocks all anonymous and cross-account access to a bucket

Cross region replication

• Replicates objects from one region to another in an asynchronous fashion
• Features
  • You can replicate objects from a source bucket to only one destination
  • After S3 replicates an object, the object cannot be replicated again
  • Enabling cross-region replication does not copy existing objects by default → need to manually copy existing objects
  • Uses SSL by default (no need for policy)
• Conditions
  • Versioning must be enabled in both source and destination
  • The replication role gives all the necessary permissions to replicate and access the objects on the source bucket
  • If the destination bucket is owned by a different AWS account, the account owner of the destination must grant the required permission to store the replicated objects
    • It is also possible to change object ownership during replication to the destination bucket
  • If the owner of the source bucket does not have permissions on the stored objects, the read and read_acp permissions should be granted (via ACL)
  • Bucket owner must have permissions to access objects (ACLs) to perform the replication
• What replicated
  • New objects created after the replication is activated
    • Old objects will not be replicated unless they are changed after the replication was activated
  • Unencrypted objects
  • Objects encrypted using SSE-S3 and SSE-KMS (no SSE-C)
  • Object metadata, ACL updates, existing tags, and lock retention information
  • Only object in the source bucket for which the bucket owner has permission to read objects and read ACLs
  • If you perform a delete operation to mark the object as deleted, this marker will be replicated
• Not replicated
  • Objects created before the CRR was enabled
  • Objects created with SSE-C, or customer-provided encryption keys
  • Objects deleted through a delete operation with a version ID specified (particular object version) → prevents malicious deletion of data

---

Glacier

Use cases

• Files stored as Archives, Archives stored in Vaults
• Data Retrieval requires job initiation then getting the output from the job

Characteristics

Data Storage

• Automatically encrypts data (AES-256)
• Regular data integrity checks
• Only your account can access data
• Can use IAM to specify which users within the account have rights to operate on a given Vault

Archive

• Stores data <40TB
• Each assigned a unique ID
• Automatically encrypted
• Immutable (after creation can't be modified)

Vault

• Container for Archives (can store infinite number)
• Max 1000 Vaults per region

Vault Lock Policies

• A vault access policy that can be locked to prevent changes to it
  • configure and enforce compliance controls for individual Glacier Vaults (~ IAM policy)
• Use cases
  • configure WORM (write once read many) archives
  • create data retention policies
• 2 steps to configure
  • initiate the lock by attaching a vault lock policy to the vault (sets the lock to an "in-progress" state)
  • you then have 24h to validate the lock policy
  • once validated, lock policies are immutable

---

1. S3 Encryption ↩

2. IAM Policies and Bucket Policies and ACLs! Oh, My! (Controlling Access to S3 Resources) ↩