Engineering Decisions

Documentation¶

Design Docs, RFCs, PRDs¶

Documentation Types

Design DocsArchitecture Documents

Design Docs (or RFCs) are a great way to get higher-level feedback on an approach, before starting the work
Written to share:
- context
- suggested approach
- bird's eye view of requirements and general architecture of the project
- tradeoffs
- and to invite feedback

Architecture Decision Records (ADRs) document implementation decisions
Written to document decisions, and less for getting feedback on these decisions
Usually live in the same repo (living docs)

Link	Notes
Design Docs at Google	Anatomy of a good design doc
PRD Template
Gergely Orosz Scaling Engineering Teams via RFCs: Writing Things Down	The power of writing things down, and spreading knowledge across the organization
Gergely Orosz Engineering Planning with RFCs, Design Documents and ADRs	What are some successful planning approaches engineering teams use as they grow?
Architecture decision record (ADR)	An architectural decision record (ADR) is a document that captures an important architectural decision made along with its context and consequences
Markdown Any Decision Records	A lean template to capture any decisions in a structured way
Docs as code is a broken promise	The pitfalls and risks of implementing a docs as code workflow, and a consideration for the future of writing docs with this workflow

Writing¶

Link	Notes
Technical Writing Courses for Engineers from Google	Collection of courses and learning resources aims to improve your technical documentation Refers to the Google Developer Documentation Style Guide
S.P.A.D.E. Toolkit: How to implement Square's famous decision-making framework	A decision-making framework, alternative to consensus built on accountability and clarity, where the person responsible for executing the decision is the one who decides
How to write SMART goals

Infrastructure¶

Architectures¶

Link	Notes
Cloud design patterns	Design patterns for building reliable, scalable, secure applications in the cloud by walking through examples based on Microsoft Azure
monday.com’s Multi-Regional Architecture: A Deep Dive	When making a decision to go multi-region, one needs to understand the primary motivation, as the work will vary greatly between performance-first, resilience-first and privacy-first designs
Uber Crane: Uber’s Next-Gen Infrastructure Stack	Post examining the original motivation and some key features behind Uber's been multi-year journey to reimagine their infrastructure stack for a hybrid, multi-cloud world
Building ClickHouse Cloud From Scratch in a Year	Have you ever wondered what it takes to build a serverless SaaS offering in under a year? Planning process, design and architecture decisions, security and compliance considerations, global scalability and reliability in the cloud, and some of the lessons learned
Cruise Container Platform Security at Cruise	Identity, Authentication, Authorization, Secrets, Encryption
Cruise Building a Container Platform at Cruise	Environments & Tenants, Infrastructure Boundaries, Platform Boundaries
Building the Next Evolution of Cloud Networks at Slack	How Slack has gone through an evolution of their AWS infrastructure from running a few hand-built EC2, all the way to provisioning thousands of them across multiple AWS regions
Decentralized GitOps over multiple environments	How SAP Artificial Intelligence implements GitOps in their large-scale project spanning multiple environments
Hashnode's Overall Architecture	How Hashnode runs on a scale (almost) completely serverless
Securing Research Infrastructure for Advanced AI	A snapshot of the security architecture OpenAI designed to protect their research infrastructure and model training
(Almost) Every infrastructure decision I endorse or regret after 4 years running infrastructure at a startup	The author reviews infrastructure decisions over four years at a startup, covering successful choices and regrets, emphasizing AWS, EKS, managed services, automation, and the value of early adoption and GitOps

Cloud Implementations¶

Link	Notes
Multicloud failover is almost always a terrible idea	Multicloud failover is complex and costly to the point of nearly almost always being impractical, and it's not an especially effective way to address cloud resilience risks
AWS Cloud Encryption is worthless! Click here to see why...	When evaluating your cloud security posture priorities, encryption should be at the bottom of your list. First, get your IAM house in order
AWS App-Layer Encryption in AWS
AWS Network access for private clusters	Very interesting article going into the problem of providing network connectivity between Kubernetes clusters and other internal tools (like deployment pipelines)
AWS Square Adopting AWS VPC Endpoints at Square	Secure communication between data centers and the cloud
AWS Square Providing mTLS Identities to Lambdas	Writeup on how Square added support for mutual TLS calls from AWS Lambda into their data center
AWS Square Expanding Secrets Infrastructure to AWS Lambda	How Square extended their datacenter-based secrets infrastructure to enable a cloud migration supporting Lambda
AWS Square Connecting Block Business Units with AWS API Gateway	How Block enables backend services to securely connect across business unit boundaries using AWS API Gateway
GCP Container Platform Networking at Cruise	GCP Hybrid Connectivity Options

Kubernetes & Microservices¶

Link	Notes
Container technologies at Coinbase: Why Kubernetes is not part of our stack	Container technologies also create a large set of challenges that must be overcome to prevent failures
Uber Why We Leverage Multi-tenancy in Uber's Microservice Architecture	Testing in production: make the current production stack multi-tenant and allow both test as well as production traffic to flow through it Canary deployments: a canary can be treated as yet another tenant in a multi-tenant architecture Capture/replay and shadow traffic: replaying previously captured live traffic or replaying a shadow copy of live production traffic in a hermetically safe environment is another use case of multi-tenancy
Uber Introducing Domain-Oriented Microservice Architecture	This piece explains DOMA, the concerns that led to the adoption of this architecture for Uber, its benefits for platform and product teams, and, finally, some advice for teams who want to adopt this architecture
A Kubernetes engineer's guide to mTLS	What mTLS is, how it relates to ordinary TLS, and why it's relevant to Kubernetes
Design Considerations at the Edge of the ServiceMesh	Set of design patterns around inbound and outbound traffic to and from a service mesh
How we use HashiCorp Nomad	Reliability model of services running in our more than 200 edge cities worldwide

Development Environments & CI¶

Link	Notes
AWS Setup	My CI/CD pipeline is my release captain: How Amazon continuously release changes to production by practicing trunk-based development, by using CI/CD pipelines to manage deployment artifacts and coordinate releases across multiple production environments, and by practicing proactive and automatic rollbacks Automating safe, hands-off deployments: The principles of continuous delivery, the use of blue-green deployments, and how to implement automated canary deployments. It also provides best practices for monitoring and rolling back deployments
GCP Setup	Google Cloud's approach to change: How Google Cloud introduces code changes into its products by following a rigorous change management process
Automating Our Infrastructure to Empower Engineers	Syncing Dev Environments Mirroring Dev and Prod Environments Developing Locally Deploying to Production
Balancing Safety and Velocity in CI/CD at Slack	Pre-merge pipeline: high criticality end-to-end tests execute and must pass to merge a user's PR to mainline Post-merge pipeline: After merging to mainline, a larger subset (medium-criticality) of end-to-end tests execute against each commit, and these tests give signal to the CD pipeline before deploy Regression pipeline: After merging to mainline, the remainder of end-to-end tests batch execute against batches of commits. These tests give signal to broken low-criticality features on each deploy, and run on a bi-hourly cadence
Lyft Scaling productivity on microservices at Lyft	History of development and test environments
Devpod: Improving Developer Productivity at Uber with Remote Development	How Uber improved the daily edit-build-run developer experience using DevPods
How Discord Moved Engineering To Cloud Development Environments	How Discord moved to Coder
Stripe's monorepo developer environment - Made of Bugs
Uber's Continuous deployment for large monorepos	How Uber reimagined continuous deployment of microservices to improve deployment automation and the user experience of managing microservices
Superuser Gateway: Guardrails for Privileged Command Execution	Uber's "Superuser Gateway" replaces direct superuser CLI access with a Git-backed, peer-reviewed workflow
Orchestrating AI Code Review at scale	Cloudflare built a CI-native, plugin-based AI code review system using OpenCode, orchestrating up to 7 specialised agents (security, performance, code quality, etc.) per merge request

Software Engineering¶

Link	Notes
Why is it so hard to decide to buy?	If it isn't related to your core business No one else has an offering or one that is any good for you to use You shouldn't build a better solution, you should ask yourself why no one else needs to solve this problem
Common Mistakes in Modularisation	Design decisions have consequences on other design decisions. This is an important aspect of how and when to decide things Modules can be used to hide design decisions so they no longer affect subsequent decisions
The top 10 fallacies in platform engineering	The prioritization fallacy The visualization fallacy The "wars you cannot win" fallacy The "everything and everybody at once" fallacy The "the new setup isn’t better" fallacy The abstraction fallacy The "loudest voice" fallacy The freedom fallacy The "Google/Facebook/Netflix" fallacy The "compete with AWS" fallacy
The slop cannons in your engineering org	"Slop cannons" are engineers/designers who over-rely on AI agents, shipping high volumes of low-quality, poorly-understood PRs. The article identifies the pattern, its causes, and offers mitigation strategies for managers and practitioners

Other¶

Link	Notes
Ledger: Stripe’s system for tracking and validating money movement	Technical details on how Stripe built a state-of-the-art money movement tracking system
What Happens on GitLab When You do git push?	Ever wondered how Git and GitLab operate under the hood?