Reliable, Scalable and Maintainable Applications, Part 2 of 2

Median means the 50th percentile. It is abbreviated as p50. If median is 200 ms, half the requests finish in less than 200 ms. The other half take longer than 200 ms.

That sounds fine for one request. It fails for a session. A page often needs many requests. The chance that at least one request exceeds the median grows fast with request count.

Illustrative calculation: assume 10 independent requests. Assume each has 50 percent chance to exceed the median. Then chance at least one exceeds it is 1 minus 0.5 to the power 10. That equals about 99.9 percent. This answers why a 10 request page is almost certain to hit a slow request.

Outliers need higher percentiles. Common choices are p95, p99, and p999. A p95 of 1.5 s means 95 percent of requests are faster than 1.5 s. It means 5 percent are 1.5 s or slower.

High percentiles track real user harm. Amazon sets internal targets at p999. The reason is that slow requests often belong to heavy buyers. Those are the most valuable customers. Amazon will not chase p9999. Random events outside your control dominate that far out. Benefit shrinks while cost grows.

Think of a percentile as a line in the sand. Sort all response times from fast to slow. Walk to the 95 percent mark. Read the value. That is p95.

Averages blend fast and slow together. Percentiles keep the slow visible. SLO means service level objective. SLA means service level agreement. Both contracts use percentiles. Example terms are R1, R2, and R3 above.

Do not average percentiles across time. Do not average percentiles across machines. That math is meaningless. Add the underlying histograms instead. Then compute the percentile from the combined histogram. Adding histograms preserves the count of slow requests. Averaging p99 values hides that count.

Two effects create long tails.

First, head-of-line blocking. In plain words it means slow work at the front blocks fast work behind it. A server processes only a small number of things in parallel. The limit can be CPU cores. A few slow requests then hold up later requests. Later requests may be fast to process. The client still sees a slow total because of queue wait.

So measure on the client side. Server time starts when work starts. Server time misses queue wait. Client time includes queue wait. Client time shows user pain.

Second, parallel fan out. Backend services are often called many times for one user request. Even parallel calls must wait for the slowest call. One slow backend call makes the whole user request slow.

Chapter Figure 1-5 gives concrete numbers:

• Backend 1: 92 ms
• Backend 2: 76 ms
• Backend 3: 103 ms
• Backend 4: 143 ms
• Backend 5: 86 ms
• Backend 6: 487 ms
• Backend 7: 133 ms
• End user request: 487 ms

Six calls look normal. Backend 6 is slow. The user waits for Backend 6. Illustrative failure case: add one more slow backend. Then the user still waits for the slower of the two. Illustrative fix case: remove Backend 6. Then the user time drops to 143 ms. That is the next slowest.

Load testing has one trap. A closed loop client waits for a reply before sending the next request. That keeps queues artificially short. It skews results to look better than reality. A correct generator sends requests independently of response time.

Production monitoring has a standard pattern. Keep a rolling window of response times from the last ten minutes. Every minute compute median over that window. Every minute compute chosen percentiles over that window. Plot them.

Naive code keeps a list. It sorts it each minute. That can be too heavy. Named options for cheap approximation are forward decay, t-digest, and HdrHistogram. The chapter names them without endorsing one. For combining hosts, add histograms. For combining time buckets, add histograms. Do not average p99 values.

Once load and speed are defined, the scaling question is direct. How do we keep performance good when load numbers grow.

A design for one load level rarely survives ten times that load. Fast growing services often rethink architecture on each order of magnitude increase.

Two broad moves exist. Scaling up means vertical scaling to a more powerful machine. Scaling out means horizontal scaling across many smaller machines. Load spread across many machines with no shared state is called a shared nothing architecture.

Scaling up is often simpler. A single machine avoids distribution work. High end machines get very expensive. Very heavy work then must scale out. Real designs mix both. Several fairly powerful machines can beat many tiny virtual machines on simplicity. They can beat many tiny virtual machines on cost.

Elastic means the system adds resources automatically on load rise. Manual scaling means a human reviews capacity. Manual scaling means a human adds machines. Elastic helps with unpredictable load. Manual is simpler. Manual has fewer operational surprises.

State changes the price. Stateless services spread across machines with little pain. Moving a stateful data system from one node to distributed adds much complexity. Common wisdom was to keep the database on one node. Cost forced distribution. High availability forced distribution. Better tools may change that default. The book will revisit distributed data systems for scalability. It will revisit them for ease of use. It will revisit them for maintainability.

There is no generic magic scaling sauce. Each large design fits its application. Relevant load shape includes read volume. It includes write volume. It includes stored data volume. It includes data complexity. It includes response time needs. It includes access patterns. Usually it is a mix.

Example from text: 100,000 small requests per second looks nothing like 3 huge requests per minute. Both move similar bytes. Architectures differ completely. Good scaling bets on which operations will be common. Good scaling bets on which operations will be rare. Those bets are the load parameters. Wrong bets waste effort at best. They hurt at worst. Early unproven products should favor fast iteration. Hypothetical future load should wait. Reusable blocks still apply. Familiar patterns still apply.

Most software cost is maintenance, not initial build. Maintenance includes bug fixes. It includes operations. It includes failure analysis. It includes platform moves. It includes new use cases. It includes tech debt. It includes new features. Legacy pain is hard to generalize. We can avoid creating new legacy by designing for three goals.

Numbered goals with lesson shorthand:

• M1: operability. Make life easy for operations.
• M2: simplicity. Make the system easy for new engineers to understand.
• M3: evolvability. Make future change easy.

M1 work: good operations can work around bad software. Good software cannot survive bad operations. Some operations can be automated. Humans still build that automation. Humans still check that automation. Typical ops duties include monitoring health. They include restoring service. They include finding causes of failure or slowness. They include patching software and platforms. They include watching cross system effects. They include planning capacity. They include building deployment and config tools. They include moving platforms. They include guarding security. They include defining stable processes. They include preserving knowledge as people leave.

Good operability makes routine tasks easy. That frees ops for high value work. Helpful traits are visibility with good monitoring. They include support for automation and standard tools. They include no dependence on single machines. They include clear docs and operational model. They include good defaults with override freedom. They include self healing where apt. They include manual control when needed. They include predictable behavior with few surprises. A miss here is lesson breaks (M1). Symptom is toil. Symptom is slow recovery. Example complaint: pager fires nightly for a full disk. No dashboard shows disk growth. Restart needs a single fragile host.

M2 work: small projects start clear. Large projects drift toward a big ball of mud. Symptoms include state explosion. They include tight coupling. They include tangled dependencies. They include inconsistent naming. They include performance hacks. They include special cases for distant bugs. Complex code slows everyone. It busts budgets and schedules. It raises bug risk on each change. Hidden assumptions are missed.

Fix is to remove accidental complexity. That means complexity not inherent in the user problem. It comes only from implementation. The main tool is abstraction. A good abstraction hides detail behind a clean front. It also gets reused widely. Reuse spreads quality fixes to all users. Examples are high level languages hiding machine code. They hide CPU registers. They hide syscalls. SQL hides on disk structures. It hides in memory structures. It hides concurrent clients. It hides post crash inconsistency. Good distributed abstractions are still hard. The book will hunt for them. A miss here is lesson breaks (M2). Symptom is fear of change. Example complaint: a newcomer cannot trace a request. A small edit breaks distant modules.

M3 work: requirements never freeze. Facts change. Use cases emerge. Priorities shift. Platforms turn over. Laws change. Growth forces redesign. Agile process helps adapt. Local tools include test driven development. They include refactoring. The book seeks agility at data system scale. Example question is how to refactor Twitter timelines from approach 1 to approach 2. Simple systems with clean abstractions evolve more easily. That system level agility is called evolvability. It is also called extensibility. It is also called modifiability. It is also called plasticity. A miss here is lesson breaks (M3). Symptom is product requests that require rewrites. Example complaint: a normal feature needs a full rewrite. The data layout blocks the change.

This runnable Python uses only chapter numbers. It computes p50, p95, and p99 from the Figure 1-5 backend times. It shows correct histogram add versus wrong percentile average. Read the comments for lines that matter.

Note: concatenation in code only illustrates the add histograms principle. Real systems add histogram buckets. They do not ship raw lists.

# Figure 1-5 backend times in ms, from the chapter text
times_a = [92, 76, 103, 143, 86, 487, 133]

def percentile(data, p):
    # Line that matters: sort, then pick rank. This is the threshold model.
    # p is 0 to 100. 50 gives median. 95 gives p95.
    if not data:
        return None
    s = sorted(data)
    # Line that matters: rank uses ceiling logic, capped at last index.
    # This keeps the slow tail visible instead of blending it.
    import math
    k = math.ceil(p / 100 * len(s)) - 1
    k = max(0, min(k, len(s) - 1))
    return s[k]

print("sorted:", sorted(times_a))
print("p50:", percentile(times_a, 50))
print("p95:", percentile(times_a, 95))
print("p99:", percentile(times_a, 99))
print("user waits for max:", max(times_a))

# Two hosts, each with its own measurements.
# Correct way per text: add histograms.
# Wrong way per text: average the two p99 values.
host1 = [76, 86, 92]
host2 = [103, 133, 143, 487]
print("host1 p99:", percentile(host1, 99))
print("host2 p99:", percentile(host2, 99))
wrong = (percentile(host1, 99) + percentile(host2, 99)) / 2
print("wrong avg of p99s (illustrative):", wrong)
combined = host1 + host2  # illustrative only, stands in for adding histograms
print("correct combined p99:", percentile(combined, 99))

# SLA check with R1 and R2 from the lesson header.
# R1: median < 200 ms. R2: p99 < 1000 ms.
# breaks (1) and breaks (2) are lesson shorthand, not chapter terms.
median = percentile(combined, 50)
p99 = percentile(combined, 99)
print("breaks (1)?", not median < 200)
print("breaks (2)?", not p99 < 1000)

Expected output

sorted: [76, 86, 92, 103, 133, 143, 487]
p50: 103
p95: 487
p99: 487
user waits for max: 487
host1 p99: 92
host2 p99: 487
wrong avg of p99s (illustrative): 289.5
correct combined p99: 487
breaks (1)? False
breaks (2)? False

• Sort then rank shows the threshold idea. No averaging hides the 487 ms value.
• Concatenation stands in for adding histograms. That is only an illustration of the chapter approved combine step.
• Averaging p99 values is shown only to mark it wrong. It understates the tail.
• Final checks map directly to lesson breaks (1) and lesson breaks (2).

• Amazon internal services: target p999 because 1 in 1,000 still hits valuable heavy buyers.Reject p9999 because random outside events dominate and benefit fades.
• Amazon store: treats 100 ms extra as 1 percent sales loss.Reason to fund tail work up to a point.
• Web search and content sites cited: treat 1 second extra as 16 percent satisfaction loss.Reason to put tail latency in SLOs.
• Twitter home timelines: Part 1 recap for describing load.Excerpt use here is the evolvability test case. That test is refactor from approach 1 to approach 2.
• Rolling dashboard owners: use ten minute window with per minute percentile calc.Use forward decay, t-digest, or HdrHistogram if sort each minute is too heavy. Reason is ongoing visibility without excess CPU. Reason is ongoing visibility without excess memory.

• Q: Why not just report average response time.A: Average blends. One 487 ms call among six fast calls barely moves the mean. User still waits 487 ms. Percentiles expose that. Use p50 for typical. Use p95 or higher for outliers.
• Q: Why does p99 get worse when you add more backend calls.A: User waits for the slowest parallel call. More calls means more chances to hit a slow one. Even low per call slow rate yields high per user slow rate.
• Q: Why measure on the client.A: Queues add wait before server work starts. Server timing misses that wait. Client timing includes it. Head of line blocking makes this gap large at high percentiles.
• Q: What is wrong with this load test where the client sends next request only after prior reply.A: It is closed loop. It keeps queues short. Real load arrives independently. Results look faster than production. Send without waiting for prior response.
• Q: Scale up or scale out.A: Ask load shape first. Same bytes can need opposite designs. 100,000 by 1 kB needs request rate handling. 3 by 2 GB needs bulk transfer handling. Start simple, often scale up. Scale out when cost forces it. Scale out when availability forces it. Expect a mix. Revisit each order of magnitude.
• Q: Elastic or manual.A: Elastic fits unpredictable spikes. Manual fits stable load with desire for fewer surprises. State makes either harder.
• Q: How do operability, simplicity, and evolvability differ.A: Operability helps ops run it. Lesson breaks (M1) shows as toil. Simplicity helps newcomers grasp it. Lesson breaks (M2) shows as fear to touch code. Evolvability helps future change. Lesson breaks (M3) shows as rewrites for normal requests.

Chapter 1 frames data intensive work as three non functional goals plus function. Functional means what it does. It stores data. It retrieves data. It searches data. It processes data. Non functional covers how well it does that. This chapter details reliability, scalability, and maintainability.

Reliability means correct operation despite faults. Faults span hardware. They span software. They span humans. Fault tolerance hides some faults from users.

Scalability means good performance under growing load. It needs load numbers. It needs speed numbers. Twitter timelines show load description. Percentiles show speed measurement. A scalable system adds capacity. Added capacity keeps it reliable under high load.

Maintainability means better life for engineering and operations. Good abstractions cut complexity. Good abstractions ease change. Good operability gives visibility. Good operability gives control. No quick fix makes a system reliable. No quick fix makes a system scalable. No quick fix makes a system maintainable. Later chapters supply building blocks. Part III joins components as in Figure 1-1.