How to Think Like a Systems Engineer: What Cloudflare’s 100 TB RAM Optimization Can Teach Every Developer

Most developers learn optimization in a fairly predictable way.

The application is slow, so we look at the database.

The server is using too much memory, so we check for memory leaks.

The API is taking too long, so we add caching.

The traffic increases, so we add another server.

All of these are reasonable responses.

But sometimes the biggest optimization doesn't come from adding a faster database, buying a bigger server, or rewriting an application.

Sometimes it comes from asking a much simpler question:

Why are we storing this much data in the first place?

Cloudflare recently published a fascinating engineering story about doing exactly that.

Its team found that one of its systems was using significantly more memory than expected. After digging into the data structures, the mathematics behind the algorithm, and the actual requirements of the system, they changed how the system represented and generated its data.

The result?

More than 100 TB of RAM reclaimed across Cloudflare's infrastructure. (Cloudflare Blog)

The interesting part isn't the number.

The interesting part is the thought process that produced it.

And that thought process is useful even if you're building a small Node.js application on a single VPS.

---

The 100 TB problem

Cloudflare operates thousands of servers across the world, with enormous amounts of memory and computing capacity.

At that scale, an optimization that saves a few bytes doesn't sound particularly exciting.

Until you multiply it.

Imagine that something consumes an extra 4 bytes.

For one object:

4 bytes.

For one million objects:

4 MB.

For one billion objects:

4 GB.

Now imagine that the same thing exists across thousands of machines.

Suddenly, a tiny inefficiency isn't tiny anymore.

This is one of the most important ideas in large-scale engineering:

The cost of an inefficiency is not determined only by how expensive one instance is. It is determined by how many times that inefficiency is repeated.

Cloudflare's engineers were dealing with exactly this kind of problem.

The system involved was called Pingora Backend Router, or PBR, and it used a technique called consistent hashing to distribute cacheable requests across servers. (Cloudflare Blog)

To understand why the optimization worked, we first need to understand what the system was doing.

---

A simple way to understand consistent hashing

Imagine three servers:

Server A
Server B
Server C

And imagine that every request gets converted into a number using a hash function.

You could put those numbers on a circular range:

                 A

          ┌───────────────┐
       C  │               │  B
          │               │
          └───────────────┘

The exact implementation is more technical than this, but the basic idea is simple.

A request gets hashed to a position, and the system finds the appropriate server associated with that part of the hash space.

The problem is that hashes are effectively random.

So the servers don't necessarily get equal portions of the space.

You might end up with something like:

A → 50%
B → 30%
C → 20%

even though you wanted something closer to:

A → 33%
B → 33%
C → 33%

That's a problem for a load-balancing system.

One way to make the distribution more even is to give every server multiple positions on the hash ring.

Instead of:

A → 1 position
B → 1 position
C → 1 position

you might have:

A → 160 positions
B → 160 positions
C → 160 positions

The more independent points you use, the more the random differences tend to average out.

This is essentially the same intuition behind the law of large numbers.

One random measurement can be very different from another.

A large collection of random measurements tends to become more predictable.

So far, everything makes sense.

But then comes the question that changed the entire optimization:

How many positions do we actually need?

---

More isn't always better

This is where engineering becomes more interesting than simply "make it bigger."

Suppose 100 hash points give you a certain level of distribution.

Would 1,000 be better?

Probably.

Would 10,000 be better?

Probably.

Would 100,000 be better?

Technically, perhaps.

But there's another question:

How much better?

And more importantly:

What does each additional hash cost us?

Cloudflare's engineers calculated how the variation in distribution changes as the number of hashes increases.

What they found was a classic case of diminishing returns.

The first increase in hash points provides a significant improvement.

The next increase provides another improvement.

Eventually, however, you are spending dramatically more memory for a very small reduction in error.

In the example discussed by Cloudflare, a configuration with around 100,000 hashes per server was getting very little additional benefit from the last 90,000 hashes. The team determined that the number of hashes could be reduced by about 90% without introducing appreciable error. (Cloudflare Blog)

That is a very different way of thinking about optimization.

Instead of asking:

"How can we make this more accurate?"

they asked:

"What level of accuracy is actually necessary?"

That question is incredibly useful outside Cloudflare.

---

The hidden cost of "just in case"

Developers do this all the time.

We keep extra data "just in case."

We request more database columns than we need.

We keep caches longer than necessary.

We store verbose objects.

We use 64-bit values when a smaller representation would be sufficient.

We load an entire dataset when we only need ten records.

We keep multiple copies of the same information.

We add infrastructure "for future scale."

None of these decisions are necessarily wrong.

The problem is when nobody knows why they're necessary anymore.

Over time, "just in case" becomes part of the architecture.

And architecture has a cost.

---

A six-byte lesson

One of the smaller but more beautiful optimizations in Cloudflare's work came from looking at a simple data structure.

The original structure contained two values:

struct Point {
    hash: u32,
    index: u32,
}

That's 8 bytes.

The hash needed 32 bits.

But the index was pointing to a server, and the system wasn't realistically going to coordinate more than about 65,000 servers at once.

So a 16-bit integer was sufficient for the index.

That suggests:

struct Point {
    hash: u32,
    index: u16,
}

You might expect that to consume 6 bytes.

But programming languages have rules around memory alignment and padding.

So the structure could still occupy 8 bytes in memory.

The engineers changed the representation to store the values in a six-byte array and access them through helper methods.

That reduced the memory used by those structures by 25%. (Cloudflare Blog)

Twenty-five percent sounds impressive.

But the real lesson is not:

"Everyone should use byte arrays."

The lesson is:

Know what your data actually costs in memory.

A data structure that looks small in source code isn't necessarily small in memory.

And a four-byte difference doesn't matter much until that structure exists millions or billions of times.

---

This is where "thinking like a systems engineer" begins

You don't have to work at Cloudflare to think this way.

A systems engineer develops the habit of looking beyond the immediate code.

Instead of seeing:

const users = [...]

they start asking:

• How many users?

• How large is each object?

• How many copies exist?

• Is this data duplicated?

• How long does it need to stay in memory?

• Does every request need all of it?

• Could the database return only what we need?

• Could this be compressed?

• Could we calculate it instead of storing it?

• What happens when the dataset becomes 10× larger?

The code hasn't changed.

The questions have.

And those questions often lead to the optimization.

---

Small × huge = enormous

This is probably the most transferable lesson from Cloudflare's story.

Suppose your application stores an object that is 2 KB larger than necessary.

At 100 records:

200 KB

Nobody cares.

At 100,000 records:

~200 MB

Now you start caring.

At 10 million records:

~20 GB

Now it's an architectural problem.

This is why scale changes the meaning of "small."

A developer working on a prototype can reasonably ignore a 2 KB inefficiency.

A developer working on a system with 100 million records cannot.

The trick is knowing when the multiplication starts to matter.

---

You can use the same thinking in a Node.js application

Imagine you're building an API that returns user profiles.

The database contains:

id
name
email
avatar
bio
created_at
updated_at
preferences
activity
settings
...

Your endpoint only needs:

id
name
avatar

But your query returns everything.

At 100 users, you probably won't notice.

At 100,000 requests per minute, you might.

You're moving data:

Database
   ↓
Node.js
   ↓
Serialization
   ↓
Network
   ↓
Browser

Every unnecessary field travels through that pipeline.

The optimization isn't necessarily:

"Buy a faster server."

It might simply be:

"Don't move data we don't need."

That's systems thinking.

---

The same principle applies to databases

Consider a table with millions of rows.

You need:

SELECT id, name
FROM users
WHERE organization_id = ?;

But your application does:

SELECT *
FROM users
WHERE organization_id = ?;

The second query is convenient.

The first query communicates the actual requirement.

At small scale, the difference may be invisible.

At larger scale, it affects:

• disk reads

• memory

• network transfer

• serialization

• application memory

• database cache

• query performance

One small assumption can propagate through an entire system.

---

What about Redis?

Redis is another great example.

Developers often think:

"Redis is fast, so I'll cache everything."

But Redis is still memory.

Suppose you cache:

user:123

and the cached value is 20 KB.

Now multiply it:

20 KB
× 1 million users
≈ 20 GB

And that's before accounting for Redis' own data structure overhead.

The better question isn't:

"Can Redis store this?"

It's:

"Does this information need to be cached, and does the cached representation need to be this large?"

Maybe the answer is yes.

Maybe it isn't.

The point is to ask.

---

Sometimes computation is cheaper than storage

This is another powerful systems idea hiding inside the Cloudflare example.

There are two ways to deal with information:

Store it.

Or:

Calculate it when needed.

Neither is automatically better.

Suppose you have a value that takes:

1 ms

to calculate and occupies:

10 MB

in memory.

Depending on how often it's needed, calculating it might be cheaper.

On the other hand, if it's requested thousands of times per second, storing or caching it may make more sense.

That's the trade-off:

          STORAGE
             ↕
        COMPUTATION

Good engineering is often about finding the right point on that curve.

---

This applies to frontend development too

Systems thinking isn't limited to backend infrastructure.

Consider a web application loading:

12 JavaScript libraries
8 icon libraries
3 analytics scripts
5 font files
large images
unused CSS

Each individual resource may seem harmless.

Together:

more bytes
   ↓
more network work
   ↓
more parsing
   ↓
more JavaScript execution
   ↓
more memory
   ↓
slower page

This is the same pattern again.

Small inefficiencies multiply.

Instead of asking:

"Which framework is fastest?"

sometimes the better question is:

"Why are we sending all of this to the browser?"

---

But don't optimize everything

There's an important warning here.

The lesson from Cloudflare isn't:

Make everything as small as possible.

That would be a terrible interpretation.

Optimization has a cost.

A six-byte representation may be harder to read than a normal struct.

A highly optimized database query may be more complicated to maintain.

A custom cache can introduce bugs.

A compressed representation can increase CPU usage.

A clever algorithm can make a system harder for the next developer to understand.

So the goal isn't:

Maximum optimization.

The goal is:

The right trade-off for the actual problem.

Cloudflare didn't simply start packing every integer into the smallest possible number of bytes.

They found a system where the memory cost was significant enough to justify the complexity.

That's an important distinction.

---

The 10× question

Here's a habit worth developing.

Whenever you build something, ask:

What happens if this becomes 10× bigger?

Not 1,000×.

Just 10×.

If you have:

1,000 users

ask what happens at:

10,000

If you're processing:

100 requests/second

ask what happens at:

1,000 requests/second

If your database has:

100,000 rows

ask what happens at:

1 million

This doesn't mean you need to build for that scale today.

It means you should know where the next wall is.

---

A small story from everyday development

Imagine a developer building a notification system.

The first version stores every notification as a large JSON document:

{
  "id": "...",
  "user": "...",
  "type": "message",
  "message": "...",
  "sender": {
    "id": "...",
    "name": "...",
    "avatar": "..."
  },
  "metadata": {...},
  "createdAt": "...",
  "read": false
}

It works.

A few months later, the application has millions of notifications.

Redis is consuming a huge amount of memory.

The developer's first thought is:

"We need a larger Redis instance."

But another developer asks:

"Why are we storing the sender's name and avatar inside every notification?"

The sender already exists in the user database.

So they change the notification to:

{
  "id": "...",
  "userId": "...",
  "senderId": "...",
  "type": "message",
  "createdAt": "...",
  "read": false
}

Suddenly the object is dramatically smaller.

Nothing about the feature changed.

The user still receives the same notification.

The system simply stopped storing information it didn't need to store repeatedly.

That's the kind of thinking Cloudflare's story encourages.

---

Start measuring the thing you're afraid of

Another lesson from the Cloudflare work is the importance of measurement.

A lot of performance work starts with intuition:

"This must be the bottleneck."

Sometimes we're right.

Often we're not.

A better process looks like this:

Problem
   ↓
Measure
   ↓
Find the expensive part
   ↓
Understand why
   ↓
Change one thing
   ↓
Measure again

For memory:

What objects consume the most memory?

For PostgreSQL:

Which queries consume the most time?

For Redis:

Which keys consume the most memory?

For an API:

Where is the latency actually coming from?

For a frontend:

What is actually contributing to page load?

The profiler is often more valuable than your intuition.

---

The mathematics doesn't have to be scary

One thing I particularly like about Cloudflare's article is that the engineers didn't treat mathematics as something separate from software engineering.

They used probability to answer a practical question:

How many hash points are enough?

That's the useful side of mathematics in engineering.

You don't need to derive every formula from scratch.

You need to understand enough mathematics to recognize questions like:

• How likely is this collision?

• How much variation should I expect?

• How much does increasing the sample size help?

• What happens as the dataset grows?

• When does the improvement become negligible?

• What is the expected cost?

For a developer, mathematics becomes useful when it helps replace:

"I think..."

with:

"The numbers suggest..."

---

The birthday paradox shows up in real systems too

Cloudflare also had to consider hash collisions.

A 32-bit hash has:

2³²

possible values.

That's more than four billion possible values.

That sounds huge.

But collisions become relevant much sooner than most people intuitively expect.

This is the famous birthday paradox: you don't need hundreds of people in a room before two people are likely to share a birthday.

The same general intuition applies to hash values.

As the number of generated hashes increases, the probability of collisions rises.

And that created another interesting result for Cloudflare:

Adding more hash points wasn't simply consuming more memory.

At some point, it could also introduce more collisions and undermine some of the theoretical benefit.

So the engineers had to consider both:

More hashes
     ↓
Better distribution

and:

More hashes
     ↓
More memory
     +
More collisions

The optimal point isn't necessarily the maximum point.

It's the point where the overall trade-off makes sense. (Cloudflare Blog)

---

This is how you should think about caching

Caching is another area where this mindset pays off immediately.

The usual question is:

"What can we cache?"

A better set of questions is:

"What should we cache?"

Then:

"For how long?"

Then:

"How large is each cached object?"

Then:

"How frequently is it accessed?"

Then:

"What happens when the cache is full?"

A cache isn't free.

You're exchanging one resource for another.

Usually:

more memory
     ↓
less computation / database work

The right cache strategy depends on the cost of both sides.

That's systems thinking.

---

Don't confuse complexity with engineering quality

There's another lesson worth taking from this story.

The Cloudflare engineers went deep into statistics, memory representation, hash collisions, migration strategy, and rollout controls.

But that doesn't mean every application needs that level of complexity.

If you're building a small internal dashboard with 200 users, spending three days shaving a few bytes from a data structure is probably not good engineering.

If you're running infrastructure where the same structure exists at enormous scale, it can be an excellent investment.

So always ask:

What is the scale of the problem?

The same optimization can be brilliant in one environment and pointless in another.

---

Where you can apply this thinking every day

You can use this mindset in almost every part of software development.

APIs

Ask:

• Am I returning data the client doesn't need?

• Am I serializing unnecessarily large objects?

• Am I making five requests where one would work?

Databases

Ask:

• Do I need all these columns?

• Is this index actually useful?

• Am I fetching thousands of rows to use ten?

• Is duplicated data creating unnecessary storage?

Redis

Ask:

• Does this really need to be cached?

• How large is each key?

• How many copies exist?

• What's the cache hit rate?

Frontend

Ask:

• Why am I shipping this JavaScript?

• Are these dependencies actually needed?

• Are these images larger than necessary?

• Is some CSS or JavaScript never used?

Servers

Ask:

• What's consuming memory?

• What's consuming CPU?

• Are connections being held unnecessarily?

• Is the application doing work that could be avoided?

Logs

Ask:

• Do I need every log at this verbosity?

• How much storage do these logs consume?

• How long do I actually need to retain them?

Architecture

Ask:

• Do I need another service?

• Is this complexity solving a real problem?

• Could the existing system handle it?

• What happens at 10× scale?

---

Five questions worth putting beside your monitor

The next time you find yourself optimizing something, ask:

1. What am I actually storing?

Don't look at the variable name.

Look at the actual data.

2. How many times does it exist?

One copy and one billion copies are completely different problems.

3. What does one copy really cost?

Include memory, storage, network, CPU, database overhead, and operational cost where relevant.

4. How much accuracy or performance do I actually need?

More isn't automatically better.

5. What happens when this becomes 10× larger?

You don't need to solve the future problem today.

But you should know where it might appear.

---

The difference between a coder and a systems thinker

Writing code is often about making something work.

Engineering is also about understanding what that solution costs.

A developer might look at a data structure and see:

hash
index

A systems engineer sees:

hash
+
index
+
alignment
+
memory
+
number of instances
+
access pattern
+
CPU cost
+
distribution
+
failure modes

A developer might see a cache and think:

"It's fast."

A systems thinker asks:

"Fast compared with what, at what memory cost, with what hit rate, and what happens when the cache fills?"

A developer might see a database query and think:

"It works."

A systems thinker asks:

"What happens when this query runs 10,000 times per second?"

Neither mindset is about being smarter.

They're simply looking at different parts of the same system.

---

You don't need Cloudflare's scale to think like Cloudflare

This may be the most important takeaway.

You probably aren't managing thousands of servers.

You probably don't have 100 TB of RAM sitting around waiting to be optimized.

You don't need to.

The useful part of Cloudflare's story isn't the scale.

It's the habit.

They found a resource problem.

They didn't immediately throw more resources at it.

They investigated.

They understood the algorithm.

They looked at the mathematical behavior.

They examined the memory representation.

They questioned whether all the work was necessary.

They measured the trade-offs.

They changed the implementation.

And they rolled it out carefully.

That sequence can be applied to a tiny application running on a $5 server just as easily as it can be applied to a global network.

---

The next time your application feels expensive, ask a different question

When your server needs more RAM, don't immediately ask:

"Which bigger server should I buy?"

Ask:

"What exactly is using the RAM?"

When your API is slow, don't immediately ask:

"Which framework should I switch to?"

Ask:

"Where is the time actually going?"

When your database is getting expensive, don't immediately ask:

"Which database should I migrate to?"

Ask:

"What are we storing, reading, and doing unnecessarily?"

And when your application grows, don't just ask:

"How do I make it handle more?"

Ask:

"What assumptions did I make when the system was small that no longer make sense now?"

That is the real lesson behind Cloudflare's 100 TB optimization.

The impressive part wasn't finding a magical trick that suddenly created 100 TB of memory.

It was realizing that a system had been doing more work and storing more information than it actually needed.

Good engineering often starts by removing what you don't need.

And sometimes, the smallest change in a system becomes enormous when you multiply it enough times.