mySystemDesign

BEGINNER TO ADVANCED

Embedding-Based Retrieval for Airbnb Search

How Airbnb improved page performance using HTTP Streaming

Airbnb's Data Framework for faster and more reliable read-heavy workloads

Avoiding Double Payments in a Distributed Payments System

Training code generation models to debug their own outputs

The technology behind Amazon's GenAI-powered shopping assistant, Rufus

Ensuring that customers don't miss out on trending products

From structured search to learning-to-rank-and-retrieve

Invalidating robotic ad clicks in real time

Using large language models (LLMs) to synthesize training data

Lessons learned from 10 years of DynamoDB

Using graph neural networks to recommend related products

How Atlassian Scaled and Enhanced Throughput in the Jira Export Service

How one of Atlassian's critical services consistently gets above 99.9999% of availability

How Atlassian made Git push over HTTPS faster for Bitbucket Cloud

How Atlassian Revamped Confluence Cloud Search

Caching JQL search in Jira Cloud

Scaling, rearchitecting, and decomposing Confluence Cloud

Scaling Bitbucket's Database

Atlassian's journey scaling low latency, multi-region services on AWS

Anomaly Detection in Time Series Using Statistical Analysis

How Booking Cut 20% of the Cloud Cost with a Single Code Change

The Engineering Behind Booking.com's High-Performance Ranking Platform

How Booking.com Leverage graph technology for real-time Fraud Detection and Prevention

How Booking.com Predicts cancellations with survival modeling

Canva's continuous data platform

How Canva's drawing tool works

How Canva collects 25 billion events per day

Canva's scalable and reliable content usage counting service

How Canva saves millions annually in Amazon S3 costs

How Canva scaled media uploads from Zero to 50 Million per day

Canva's fast and scalable reverse image search

How Canva enables real-time collaboration with RSocket

How Coinbase Optimizes Network Requests

Accelerating Deep Learning Adoption at Coinbase

Lessons from launching Enterprise-grade GenAI solutions at Coinbase

How Coinbase Uses ML to Predict Traffic and Scale Databases

Detecting Fraudulent Transactions at Coinbase

Building a notification platform at Coinbase

How Discord Reduced Websocket Traffic by 40%

How Discord Stores Trillions of Messages

Pushing Discord's Limits with a Million+ Online Users in a Single Server

How Discord uses ML to Build a Delightful Notification Experience

How Discord Creates Insights from Trillions of Data Points

How DoorDash Uses LLMs to transcribe restaurant menu photos

How DoorDash leverages LLMs for better search retrieval

Building DoorDash's product knowledge graph with large language models

DoorDash's in-house search engine

DoorDash's write-heavy scalable and reliable inventory platform

Doordash's scalable real time event processing with Kafka and Flink

DoorDash's Lessons on Improving Performance on High-Traffic Web Pages

How DoorDash Applied Client-Side Caching to Improve Feature Store Performance by 70%

Building a Unified Chat Experience at DoorDash

How Dropbox evolved its infrastructure through the messaging system model

Dropbox's scalable, consistent, metadata caching solution

Bringing AI-powered answers and summaries to file previews on the web

Dropbox's ML-powered file organization

How Dropbox uses ML to identify date formats in file names

How Dropbox optimizes payments with machine learning

How ebay Exports Billion-Scale Graphs on Transactional Graph Databases

eBay's Personalized User-Based Ranking Model for Recommendations

How Multimodal Embeddings Elevate eBay's Product Recommendations

eBay's Blazingly Fast Billion-Scale Vector Similarity Engine

How eBay Handles Real-Time Push Notifications at Scale

Building a Deep Learning Based Retrieval System for Personalized Recommendations

How eBay Loads and Updates Over Ten-Billion-Vertex Graphs

eBay's real-time and performant index service for its large-scale, in-house database platform

Indexing code at scale with Glean - Meta's open source system

Inside Facebook's video delivery system

Meta's Sequence learning Model for personalized ads recommendations

How Meta animates AI-generated images at scale

How Meta trains large language models at scale

Building Meta's GenAI Infrastructure

RoCE networks for distributed AI training at scale

How Meta built the infrastructure for Threads

Building end-to-end security for Messenger

Modernizing Meta's data platform

How Precision Time Protocol is being deployed at Meta

Scaling data ingestion for machine learning training at Meta

Meta's cloud gaming infrastructure

Cache made consistent - How Meta handles cache invalidation

A highly available, strongly consistent storage service using chain replication

Making a distributed priority queue disaster-ready

How we built a general purpose key value store for Facebook with ZippyDB

Fully Sharded Data Parallel: faster AI training with fewer GPUs

How Facebook encodes your videos

Scaling a distributed priority queue at Meta

How machine learning powers Facebook's News Feed ranking algorithm

How Meta scaled Live streaming for millions of viewers simultaneously

The infrastructure behind AI search in Figma

Speeding up file load times at Figma

Figma's LiveGraph: a real-time data system at scale

How Figma horizontally scaled Postgres to unlock nearly infinite scalability

How Figma improved performance and load time with incremental frame loading

How Figma reduced potential instability by scaling to multiple databases

The hidden challenges of autosave

Figma's deep search to find the right files even faster

Flipkart's MySQL Highly Available Setup

Running a multi-region Zookeeper at Flipkart

Memory Tuning a High Throughput Microservice

Building Flipkart's Personalized Search Autosuggestion

Predicting your next query even before you type!

How Flipkart Adapted Search to Indian Phonetics

How we improved push processing on GitHub

How GitHub uses merge queue to ship hundreds of changes every day

How GitHub Docs' new search works

The technology behind GitHub's new code search

Scaling Git's garbage collection

Improve Git monorepo performance with a file system monitor

Partitioning GitHub's relational databases to handle scale

Load balancing with random job arrivals

Transformers in music recommendation

Scaling multimodal understanding to long videos

Answering billions of reporting queries each day with low latency

Grammar checking at Google Search scale

World scale inverse reinforcement learning in Google Maps

Resolving code review comments with ML

Scaling the Instagram Explore recommendations system

Reducing Instagram's basic video compute time by 94 percent

Improving Instagram notification management with machine learning and causal inference

Building text animations for Instagram Stories

Pushing the limits of compression in Facebook's mobile apps

How Instagram suggests new content

Real-time Fraud Detection with Yoda and ClickHouse

How Instacart Uses ML to Suggest Replacements for Out-of-Stock Products

Sequence models for Contextual Recommendations at Instacart

Supercharging Discovery in Search with LLMs

Optimizing search relevance at Instacart using hybrid retrieval

Instacart's Item Availability Architecture: Solving for scale and consistency

Instacart's one Deep Learning model for multiple surfaces

Distributed Machine Learning at Instacart

How Instacart Uses Embeddings to Improve Search Relevance

The Journey to Real-Time Machine Learning at Instacart

How Instacart Uses ML-Driven Autocomplete to Help People Fill Their Carts

How Instacart optimized its Logistics engine using ML

A simple search query correction heuristic for the resource-constrained

Predicting the real-time availability of 200 million grocery items

How Instacart delivers on time

Scalable federated learning at LinkedIn

Building a resilient DNS client for web-scale infrastructure

Journey of next generation control plane for data systems

Candidate Generation in a Large Scale Graph Recommendation System

Accelerating LinkedIn's My Network tab by reducing latency and improving UX

Tuning Java for high-performance services

LinkedIn OpenHouse for Big Data Management

How LinkedIn Adopted A GraphQL Architecture for Product Development

How LinkedIn Is Using Embeddings to Up Its Match Game for Job Seekers

Building the Infrastructure for Delivering Labor Market Insights from LinkedIn Data

Upscaling LinkedIn's Profile Datastore While Reducing Costs

Unifying Messaging Experiences across LinkedIn

Applying multitask learning to AI models at LinkedIn

Building a mutable dataset in data lake

Completing a member knowledge graph with Graph Neural Networks

Homepage feed multi-task learning using TensorFlow

Evolving LinkedIn's analytics tech stack

Scaling LinkedIn's Hadoop YARN cluster beyond 10,000 nodes

HTTP/2 in infrastructure: Ambry network stack refactoring

Building a heterogeneous social network recommendation system

From Big Data to Better Data: Ensuring Data Quality with Verity

Building Real-time Machine Learning Foundations at Lyft

The Recommendation System at Lyft

lyft2vec — Embeddings at Lyft

Powering Millions of Real-Time Decisions with LyftLearn Serving

Pricing at Lyft

ML Model Training Infrastructure built on Kubernetes

Elasticsearch Optimizations at Lyft

How Lyft discovered OpenStreetMap is the Freshest Map for Rideshare

Using Client-Side Map Data to Improve Real-Time Positioning

How Lyft predicts a rider's destination for better in-app experience

A New Real-Time Map-Matching Algorithm at Lyft

Foundation Model for Personalized Recommendation

How Netflix processes billions of impressions daily

Netflix's Distributed Counter Abstraction

Evolving Netflix's WebSocket proxy for the future

Netflix's Key-Value Data Abstraction Layer

Netflix's TimeSeries Data Abstraction Layer

Recommending for Long-Term Member Satisfaction at Netflix

Maestro: Data/ML Workflow Orchestrator at Netflix

Reverse Searching Netflix's Federated Graph

Supporting Diverse ML Systems at Netflix

Rebuilding Netflix Video Processing Pipeline with Microservices

Building In-Video Search

Streaming SQL in Data Mesh

Migrating Netflix to GraphQL Safely

Scaling Media Machine Learning at Netflix

Building a Media Understanding Platform for ML Innovations

Finding Cuts with Smooth Visual Transitions Using Machine Learning

Machine Learning for Fraud Detection in Streaming Services

Netflix's High-Throughput, Low-Latency Priority Queueing System

Rapid Event Notification System at Netflix

Building Netflix's Distributed Tracing Infrastructure

Building and scaling Notion's data lake

How we sped up Notion in the browser with WASM SQLite

The Great Re-shard: adding Postgres capacity (again) with zero downtime

Creating the Notion API

The data model behind Notion's flexibility

Lessons learned from sharding Postgres at Notion

Scaling PayPal's AI Capabilities with PayPal Cosmos.AI Platform

Scaling Kafka to Support PayPal's Data Growth

JunoDB: PayPal's Key-Value Store

Scaling Kubernetes to Over 4k Nodes and 200k Pods

GraphQL at PayPal: An Adoption Story

How PayPal Uses Real-time Graph Database and Graph Analysis to Fight Fraud

Next-Gen Data Movement Platform at PayPal

Deploying Large-scale Fraud Detection Machine Learning Models at PayPal

How Pinterest improved Search Relevance using LLMs

How Pinterest built it's Text-to-SQL feature

Change Data Capture at Pinterest

Real Time Anomaly Detection at Pinterest

Improving Distributed Caching Performance and Efficiency at Pinterest

How Pinterest Leverages Realtime User Actions to Boost Homefeed Engagement Volume

How Pinterest scaled the size of it's ad corpus by 60x

The machine learning behind delivering relevant ads

Building Embedding Search at Quora

Migrating a decade of Redshift usages to Trino at Quora

Trino at Quora Scale: Cost, Speed, and Reliability

MySQL sharding at Quora

Razorpay's Authentication Revamp

The Making of Razorpay Developer-Console

How Razorpay Reduced Data Platform Cost by $2M

Reducing Kubernetes cost by $300,000 at Razorpay

How does Razorpay Capital Detect Duplicate or Fraud Merchants?

Razorpay's Real-Time Denormalized Data Streaming Platform

How Razorpay's Notification Service Handles Increasing Load

How Trino and Alluxio power analytics at Razorpay

Handling Burst Traffic During IPL

Evolving Reddit's Media Infrastructure

Scaling our Apache Flink powered real-time ad event validation pipeline

Scaling Reddit's ad-serving system

Product Candidate Generation for Reddit Dynamic Product Ads

Scaling Ads Pacing: from Singleton to Sharded

Introducing a Global Retrieval Ranking Model in the Ads Funnel

Building an Experiment-Based Routing Service

The Reddit Media Metadata Store

Scaling Real-Time Search to 30 Billion Queries with Sub-Second Latency and 0% Downtime

Scaling Agentic AI Powering 2 Billion Predictions Monthly

How Agentforce Data Library Powers RAG with 99.99% Uptime

Secrets for Managing 100,000 Training and Metadata Requests Per Minute

Inside the Brain of Agentforce

How Salesforce Supports Millions of Users Seamlessly for GenAI

Inside Salesforce's Scalable Time Series Forecasting AI Platform

How Salesforce's Data Cloud Handles 250 Trillion Transactions Weekly

How Shopify improved consumer search intent with real-time ML

Horizontally scaling the Rails backend of Shop app with Vitess

Improving Shopify App's Performance

Building a ShopifyQL Code Editor

Creating a Flexible Order Routing System with Shopify Functions

Using Server Sent Events to Simplify Real-time Streaming at Scale

Capturing Every Change From Shopify's Sharded Monolith

How Slack Optimizes its E2E Pipeline

How Slack built enterprise search to be secure and private

Advancing Our Chef Infrastructure

How We Re-Architected Slack for Our Largest Customers

How Slack automatically detects stolen session cookies

How a request flows — from a Slack's user perspective

Slack's Migration to a Cellular Architecture

Real-time Messaging at Slack

How Slack traces the flow of notifications across systems

Slack's Unified end-to-end machine learning infrastructure to generate recommendations

How We Design Our APIs at Slack

How we built an eventually-consistent data model to predict Slack Connect invites

Migrating Millions of Concurrent Websockets to Envoy

Scaling Datastores at Slack with Vitess

Bento - Snap's ML Platform

Snap's Embedding-based Retrieval for its video recommendation system

How Snap Speed Up Feature Engineering for Recommendation Systems

How Snap leverages synthetic data to boost the development of ML models

Training Large-Scale Recommendation Models with TPUs

Machine Learning for Snapchat Ad Ranking

How Spotify Generated Millions of Content Annotations

Spotify's Data Platform

The What, Why, and How of Mastering App Size

How Spotify Automated Content Marketing to Acquire Users at Scale

How We Built Infrastructure to Run User Forecasts at Spotify

Stripe's system for tracking and validating money movement

How Stripe Processed $1 Trillion in Payments with Zero Downtime

How Stripe built its fraud prevention system

How Stripe builds interactive docs with Markdoc

Stripe's payments APIs: The first 10 years

Swiggy's Text-to-SQL Solution

Optimising the picking process for faster Instamart deliveries

Improving search relevance using small language models

Predicting Food Delivery Time at Cart

Contextual Bandits for Ads Recommendations

Using deep learning to detect dissonance between address text and location

Designing Resilient Microservices at Swiggy

Designing the Serviceability Platform at Swiggy for High Scale

A brief introduction to Engineering challenges at Swiggy

Re-Architecting Swiggy's logistics systems

Using Deep Learning for Ranking in Dish Search

Learning to Predict Two-Wheeler Travel Distance

Learning To Rank Restaurants

Running Geo Queries At Scale

Deploying deep learning models at scale at Swiggy

Tinder API Style Guide

Building Obsidian, Tinder's Design System

How Tinder built its API Gateway

Scaling out Tinder Android Payment Flow using State Machine

Ingesting Live Video Streams at Global Scale

Breaking the Monolith at Twitch

Using Machine Learning to Review Emotes

Defense, threat modeling and High Availability at Twitch

Twitter's Recommendation Algorithm

How we scaled Reads On the Twitter Users Database

Powering real-time data analytics with Druid at Twitter

How we built Twitter's highly reliable ads pacing service

Storing and retrieving millions of ad impressions per second

Processing billions of events in real time at Twitter

Logging at Twitter

Twitter's ads serving platform

Migrating Uber's Compute Platform to Kubernetes

MySQL At Uber

How Uber Uses Ray to Optimize the Rides Business

How Uber Optimizes LLM Training

Natural Language to SQL Using Gen AI

Lucene: Uber's Search Platform

Uber's implementation of Live Activity on iOS

Odin: Uber's Stateful Platform

Kafka Tiered Storage at Uber

Modernizing Logging at Uber with CLP

How Uber ensures Apache Cassandra's tolerance for single-zone failure

How LedgerStore Supports Trillions of Indexes at Uber

Balancing HDFS DataNodes in the Uber DataLake

How Uber Serves Over 40 Million Reads Per Second from Online Storage Using an Integrated Cache

How Uber Optimized Cassandra Operations At Scale

How Uber Optimizes the Timing of Push Notifications using ML and Linear Programming

Deduping and Storing Images at Uber Eats

Uber's Next Gen Push Platform on gRPC

Uber's Highly Scalable and Distributed Shuffle as a Service

How Uber Predicts Arrival Times Using Deep Learning

Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot

Walmart's Cassandra CDC Solution

Scaling the Walmart Inventory Reservations API for Peak Traffic

A Markov Chain Formulation for the Grocery Item Picking Process

How we rebuilt the Walmart Autocomplete Backend

Building a Notification Framework for Microservice-based Application

Building a cost-effective logging platform for petabyte scale

How Zomato Handles 100 Million Daily Search Queries

How Zomato Powers restaurant ads using ML

How Zomato uses embeddings to identify and cluster unique addresses

How Zomato predicts your order's Food preparation time

How Zomato locates its users

The Deep Tech Behind Estimating Food Preparation Time