Data Manipulation
Kinesis
Streaming data platform consisting of 3 services
Use Cases
| Use Case | Description |
| DATA INGESTION | Ensure data is accepted reliably & successfully stored in AWS |
| REALTIME PROCESSING of massive data streams | Act on knowledge gleaned from a big data stream right away |
| NOT for BATCH Jobs | Not appropriate for batch jobs (ETL) |
Services
| Service | Description |
| FIREHOSE | - STORE: Load massive volumes of STREAMING data into AWS
- Receives stream data & stores it in S3/Redshift/Elastic
- Just create a Delivery Stream & configure destination for data
 |
| STREAMS | - PROCESS: Collect and process large streams of data records in realtime
- Can create a Kinesis STREAMS App that processes data as it moves through the stream
- Can scale by distributing incoming data across shards
- Processing executed on consumers which read data from the shards & run the Kinesis Streams App
|
| ANALYTICS | - Analyze streaming data realtime with SQL
|
Elastic MapReduce (EMR)
- Description
-
- Fully managed on-demand HADOOP framework
- Use Cases
- Log processing
- Clickstream analysis
- Genomics and life sciences
Types of Storage
| Storage Type | Description |
HDFS | - Hadoop Distributed FS (standard FS that comes with Hadoop)
- All data replicated across multiple instances to ensure durability
- Can use
- EC2 Instance Storage → lose data if shut down
- EBS
|
EMRFS | - Implementation of HDFS that stores data on S3
- Preserve data if cluster is shut down
|
Types of Clusters
| Cluster Type | Description |
| Persistent | - Runs 24x7 (when continuous analysis is run on data)
- Better with HDFS
|
| Transient | - Stopped when not in use
- Use EMRFS
|
Access Control
| Control | Description |
| Security Groups | - 2 SGs setup when launching job flows → both don't allow access from external
- 1 for
PRIMARY node: 1 port open for comm with service, 1 port for SSH - 1 for
SECONDARY nodes: only allows interaction with primary
|
| IAM | - Can set permissions that allow users other than the default
hadoop user to submit jobs - If IAM user launches a cluster → cluster hidden from other IAM users
|
Data Pipeline
- Description
-
- Process and MOVE data between different AWS storage/services/on-prem, at specific intervals
- Use Cases
- BATCH MODE ETL process
- NOT for continuous data streams (use Kinesis)
Components
| Component | Description |
| PIPELINE | Schedules and runs tasks according to the pipeline definition & interacts with data stored in data nodes |
| DATA NODES | Locations where the pipeline reads input data or writes output data (can be AWS or on-prem) |
| ACTIVITIES | - Executed by the pipeline
- Represent common scenarios (e.g., moving location)
- May require additional resources to run (e.g., EMR, EC2)
- Supports PRECONDITIONS: conditional statements that must be
true before an activity can run - If it fails, retry is automatic
|
Import/Export
- Description
-
- Accelerates transferring large amount of data IN and OUT of AWS using PHYSICAL STORAGE APPLIANCES (bypassing the Internet)
- Data is copied to a device at the source (AWS or data center), shipped, and then copied to destination
- Use Cases
- Storage migration
- Application migration
Options
| Option | Description |
| SNOWBALL | - Uses Amazon-provided shippable storage appliances
- Each Snowball is protected by KMS & physically ragged
- Snowball Edge offers data processing at the edge before that data is returned to AWS
- Sizes: 50TB, 80TB
- Features:
- Import/Export: on premise ↔ S3
- Encryption enforced
- Manage jobs via its console
|
| IMPORT/EXPORT DISK | - Supports transfers data directly onto and off of storage devices you own using Amazon high-speed internal network
- Features:
- Import data in S3/Glacier/EBS
- Export data from S3
- Encryption is OPTIONAL
- You buy and maintain your HW devices
- Can't manage jobs
- Upper limit of 16TB
|