S3 & Glacier
S3¶
General Info¶
Object storage
Features
Feature | Description |
---|---|
Regionality |
|
Data Consistency |
|
MFA Delete |
|
Versioning |
|
S3 Transfer Acceleration | Speeds up transfers to S3 |
Limits |
|
Naming
Use Case | Naming | Example |
---|---|---|
Direct bucket access | bucket name after FQDN (path-style) | https://s3.us-east-2.amazonaws.com/testBucket/images |
Website hosting | bucket name part of FQDN (virtual-hosted-style) | https://testBucket.s3.us-east-2.amazonaws.com/images |
Storage Classes¶
Class | Description | Availability | Durability |
---|---|---|---|
S3 Standard | Frequently accessed data | 99.99 | 11 9s |
S3 Intelligent-Tiering |
| 99.99 | 11 9s |
S3 Standard-IA |
| 99.9 | 11 9s |
S3 One Zone-IA |
| 99.5 | 11 9s |
Glacier |
| 99.99 | 11 9s |
Glacier Deep Archive | 12-48 hours latency | 99.99 | 11 9s |
Access Control¶
The account (not user) that creates objects/buckets owns them, even if the bucket is hosted by a different account
Contexts
Context | Description |
---|---|
User needs to have permission | Using identity policies (or user is the root of an account) |
For bucket operations: bucket needs to have permission |
|
For object operations: User has to have permission (or be root) |
|
Grant Access
Option | Description |
---|---|
IAM Policies |
|
S3 ACLs |
|
S3 Bucket policies |
|
Query String Auth |
|
Conditions |
|
Policy Precedence (IAM Policy / S3 Bucket Policy / S3 ACL)
- Decisions ALWAYS default to
DENY
- If no method specifies an
ALLOW
—>DENY
by default
- If no method specifies an
- An EXPLICIT
DENY
ALWAYS override anALLOW
- Only if no method specify a
DENY
&& 1+ methods specify anALLOW
—>ALLOW
- S3 bucket policies has precedence over IAM policy
- Example
- IAM policy grants access to an object
- S3 bucket policy denies access to that object
- No S3 ACL
- —> access denied (S3 bucket policy win)
- Example
- 1 bucket policy deny all ops to all users (EXPLICIT DENY)
- 1 bucket policy allowing one specific user to upload
- -> EXPLICIT DENY ALWAYS OVERRIDE AN ALLOW
Encryption¶
Server Side
SSE-S3 | AWS S3-managed Keys |
|
SSE-KMS | AWS KMS-managed Keys |
|
SSE-C | Customer-provided Keys |
|
Client Side
CSE-KMS | AWS KMS-managed Customer Master Key |
CSE-C | Client-side Master Key |
Monitoring¶
- CloudTrail by default records bucket-level actions
- Can enable CloudTrail logging of object-level actions by setting that property on a bucket in S3 (can choose read/write)
- Server access logging: separate audit log, configured per-bucket, that stores events in a bucket
Specific Setups¶
- Force TLS
- Add bucket policy
"Condition": { "Bool": { { "aws:SecureTransport": false } }}
- Pre-signed URLs
- Give access (temporarily) to a specific object since by default buckets are private
$ aws s3 presign s3://<bucket>/file.txt —expires-in 300 # 300 seconds (default is 1h)
- Block Public Access
-
Applied to specific buckets, or all buckets in an account
BlockPublicAcls
Can't create new public bucket or object ACLs IgnorePublicAcls
Existing (and new) public ACLs are ignored BlockPublicPolicy
Can't create public bucket policies RestrictPublicBuckets
Blocks all anonymous and cross-account access to a bucket - Cross region replication
-
- Features
- Uses SSL by default (no need for policy)
- You can replicate objects from a source bucket to only one destination
- After S3 replicates an object, the object cannot be replicated again
- Enabling cross-region replication does not copy existing objects by default → need to manually copy existing objects
- Conditions
- Object owner must grant the bucket owner the
READ
andREAD_ACP
permissions via the object ACL - The owner of the destination bucket must grant the owner of the source bucket permissions to replicate objects with a bucket policy
- Versioning must be enabled
- Object owner must grant the bucket owner the
- What replicated
- Unencrypted objects
- Objects encrypted using
SSE-S3
andSSE-KMS
(noSSE-C
) - Object metadata
- Any object ACL updates
- Any object tags
- Only object in the source bucket for which the bucket owner has permission to read objects and read ACLs
- NOT REPLICATED deletes to a particular VERSION of an object (security mechanism)
- Features
Glacier¶
- Use cases
-
- Files stored as
Archives
, Archives stored inVaults
- Data Retrieval requires job initiation then getting the output from the job
- Files stored as
- Characteristics
-
- Data Storage
-
- Automatically encrypts data (
AES-256
) - Regular data integrity checks
- Only your account can access data
- Can use IAM to specify which users within the account have rights to operate on a given
Vault
- Automatically encrypts data (
- Archive
-
- Stores data <40TB
- Each assigned a unique ID
- Automatically encrypted
- Immutable (after creation can't be modified)
- Vault
-
- Container for
Archives
(can storeinfinite
number) - Max 1000
Vaults
per region
- Container for
- Vault Lock Policies
-
- A vault access policy that can be locked to prevent changes to it
- configure and enforce compliance controls for individual Glacier Vaults (~ IAM policy)
- Use cases
- configure
WORM
(write once read many) archives - create data retention policies
- configure
- 2 steps to configure
- initiate the lock by attaching a vault lock policy to the vault (sets the lock to an "in-progress" state)
- you then have 24h to validate the lock policy
- once validated, lock policies are immutable
- A vault access policy that can be locked to prevent changes to it