“Make the API faster.”

This is the vaguest ticket a developer can receive. Does “faster” mean it handles more users (Throughput)? Or that it responds quicker to one user (Latency)?

Mixing these up is why you add 10 more servers but your database queries still take 2 seconds.

This is the Mastery Guide to System Performance. We’ll ditch the textbook definitions and use the only analogy that works: Plumbing.

Part 1: Foundations (The Mental Model)

The Water Pipe

Imagine a data connection is a water pipe moving water from A to B.

1. Bandwidth = Pipe Width How wide is the pipe?

   • Unit: Mbps (Megabits per second).
   • Meaning: Theoretical max capacity. You can’t fit a cruise ship through a garden hose.

2. Latency = Travel Time How fast does a single drop of water travel from A to B?

   • Unit: Milliseconds (ms).
   • Meaning: Speed. If the pipe is 100km long, even a wide pipe has high latency.

3. Throughput = Flow Rate How much water is actually coming out of the end right now?

   • Unit: Requests per Second (RPS) or TPS.
   • Meaning: Reality. Throughput is always <= Bandwidth.

Crucial Insight:

• Adding Servers increases Throughput (Widening the highway).
• Optimizing Code/DB decreases Latency (Increasing the speed limit).

The “Traffic Jam” Paradox

You can have massive Throughput but terrible Latency.

• Example: A traffic jam on a 10-lane highway.
• Throughput: High (Thousands of cars passing per hour).
• Latency: High (It takes 2 hours to get home).

Part 2: The Investigation (Metrics that Matter)

1. The Trap of “Average” Latency

Never looks at the Average (Mean). If 9 requests take 100ms and 1 request takes 10,000ms (timeout):

• Average: ~1,090ms.
• This number lies. It tells you everyone is slightly slow, but actually 90% are fast and 1 person is dead.

2. Use Percentiles (P95, P99)

• P50 (Median): The “Normal” experience. 50% of users are faster than this.
• P95: The “Bad” experience. 5% of users (1 in 20) are slower than this.
• P99: The “Worst Case”. 1 in 100 users are suffering.

Rule: Optimize for P95. If P95 is good, almost everyone is happy.

3. Measuring Tools

• Ping: Measures Network Latency (The empty pipe).

  1 2	ping google.com # time=14.2 ms <-- Just the travel time

• Load Test (k6 / JMeter): Measures Throughput & Latency under load.

Part 3: The Diagnosis (Bottleneck Hunt)

System slow? Check the symptoms.

Part 4: The Resolution (Action Plan)

1. Fix Latency (Make it Quicker)

Latency is usually a Depth problem. You are digging too deep.

• Caching (Redis): Stop calculating; just remember. (Reduces DB trips).
• CDN (Cloudflare): Move the content physically closer to the user. (Reduces travel distance).
• Database Indexing: Find the needle without searching the haystack.

2. Fix Throughput (Make it Wider)

Throughput is a Width problem. The pipe is too narrow.

• Horizontal Scaling: Add more servers behind a Load Balancer.
• Async Processing (Kafka/RabbitMQ): Don’t do it now; do it later. Acknowledge the request immediately, process in background.

Final Mental Model

1
2
3
4
5
6

Latency    -> Speed of a car. (Fix by buying a Ferrari / Optimizing Engine).
Throughput -> Width of the road. (Fix by adding lanes / Load Balancing).
Bandwidth  -> The laws of physics limit. (Fix by upgrading fiber optics).

"The site is slow" usually means Latency.
"The site is down" usually means Throughput limits exceeded.

Next time a manager asks for “better performance,” ask them: “Do you want the Ferrari (Latency) or the Highway (Throughput)?”