What Building Humble Taught Me About Realtime Systems, Scaling, and Backend Engineering

Humble is a campus platform built around real student communities, conversations, events, discovery, and other parts of campus life.

When I started building its realtime features, I initially thought the problem was fairly straightforward:

User sends a message → Socket.IO sends it → another user receives it.

That works surprisingly well when an application is small.

Then I started asking different questions.

What happens when a user opens multiple tabs?

What happens when the realtime server restarts?

What happens when Redis stops responding?

What happens when a message is successfully saved but the realtime broadcast fails?

What happens when one server is no longer enough?

And perhaps the most important question:

What should happen when something goes wrong?

Those questions changed how I approached Humble's backend.

Instead of treating realtime communication as the system itself, I started treating it as one layer of a larger system.

This post is about that journey, the decisions behind it, and some of the things I still haven't solved perfectly.

1. The Problem Was Bigger Than "Add WebSockets"

Humble uses a hybrid architecture.

The main application handles the product itself: authentication, business logic, APIs, database operations, notifications and other application features.

The realtime layer is a separate Socket.IO service.

At a high level, the system looks like this:

                    ┌─────────────────┐
                    │     Browser     │
                    └────────┬────────┘
                             │
                  ┌──────────┴──────────┐
                  │                     │
                HTTPS                 WebSocket
                  │                     │
          ┌───────▼────────┐   ┌────────▼────────┐
          │   Main App     │   │ Socket Service  │
          │  REST / API    │   │   Realtime      │
          └───────┬────────┘   └────────┬────────┘
                  │                     │
                  ▼                     ▼
            ┌───────────┐         ┌───────────┐
            │ PostgreSQL│         │   Redis   │
            │  Durable  │         │ Cache /   │
            │   Data    │         │ Presence  │
            └───────────┘         │ Pub/Sub   │
                                  └───────────┘

The important part isn't the number of technologies.

It is why each technology exists.

The main application is the source of truth. PostgreSQL stores the durable state. Redis supports caching, presence, distributed coordination and realtime scaling. Socket.IO is primarily responsible for delivering events to connected clients.

That separation became one of the most important design decisions in Humble.

2. The Database Should Know About the Message Before the Socket Does

One of the biggest architectural changes I made was separating persistence from delivery.

A naive realtime architecture can look like:

Client
   ↓
Socket.IO
   ↓
Broadcast message

But then an uncomfortable question appears:

What if the socket server broadcasts something that never gets persisted?

The client may see a message that doesn't actually exist in the database.

Or the opposite:

What if the database succeeds but the socket broadcast fails?

That shouldn't mean the message disappears.

So Humble follows a different model:

Client
   │
   ▼
REST API
   │
   ▼
PostgreSQL
   │
   │ message is now durable
   ▼
Realtime broadcast
   │
   ▼
Socket.IO
   │
   ▼
Recipients

The message is first written to PostgreSQL.

Only after that does the application trigger the realtime broadcast.

The broadcast is deliberately non-blocking. If the realtime layer has a problem, the message still exists in the database and can be retrieved again by the client.

This led to a simple rule:

The database is the source of truth. Realtime delivery is an optimization for experience, not the system of record.

That distinction made several other decisions much easier.

3. Why I Didn't Make Socket.IO Responsible for Everything

Once persistence and delivery were separated, Socket.IO became much simpler to reason about.

The realtime service is responsible for things such as:

• delivering new messages

• presence

• typing indicators

• realtime notifications

• joining chat rooms

• communicating with connected clients

The application API remains responsible for:

• authentication

• authorization

• business rules

• database writes

• durable messages

• notifications as stored records

• application state

This means a temporary realtime failure doesn't necessarily become an application failure.

That was an important lesson for me:

A feature can be realtime without making realtime the only way that feature can function.

4. Then Redis Became More Than a Cache

Initially, Redis is easy to think about:

Database → slow
Redis → fast

But as the system evolved, Redis became useful for several different jobs:

The important part was realizing that these responsibilities have different failure consequences.

A cache disappearing shouldn't destroy the application.

A realtime coordination service disappearing shouldn't corrupt the database.

A rate limiter disappearing shouldn't necessarily make the whole application unavailable.

That pushed me toward graceful degradation.

5. What Happens When Redis Goes Down?

This became one of my favorite engineering problems in Humble.

Suppose the Socket.IO service is using Redis and Redis suddenly becomes unavailable.

A simplistic implementation might do this:

Redis fails
   ↓
Socket.IO throws errors
   ↓
Server crashes
   ↓
Everyone disconnects

I didn't want that.

Instead, the realtime layer can fall back to an in-memory mode when Redis is unavailable.

                 Redis available
                       │
                       ▼
              ┌─────────────────┐
              │   Redis Adapter │
              └─────────────────┘
                       │
                   broadcasts
                       │

Redis fails ────────────┤
                       ▼
              ┌─────────────────┐
              │ In-memory mode  │
              └─────────────────┘
                       │
                local realtime

The implementation catches Redis-related failures, disables the Redis adapter, and continues operating in single-server mode rather than allowing the dependency failure to crash the realtime process. When Redis becomes stable again, the system can recover back to Redis-backed operation.

There is an important limitation here.

If there are multiple realtime servers and Redis disappears, those servers can continue serving their own connected clients, but they lose the cross-server coordination that Redis provides.

So graceful degradation does not mean:

"Nothing changes when Redis fails."

It means:

"The system loses non-essential capabilities before it loses the entire service."

That distinction is something I understand much better now.

6. Presence Sounds Easy Until You Have Multiple Tabs

"Online" sounds like a boolean.

online = true
offline = false

But a user might actually have:

Prashant
 ├── Chrome tab
 ├── Firefox tab
 ├── Mobile browser
 └── Another laptop

If one connection disappears, should the user become offline?

Obviously not.

So Humble treats presence as a connection count rather than simply a boolean.

Conceptually:

Tab 1 connects
connections = 1
→ user becomes online

Tab 2 connects
connections = 2
→ user remains online

Tab 1 closes
connections = 1
→ user remains online

Tab 2 closes
connections = 0
→ user becomes offline

The implementation uses Redis-backed reference counting, connection expiry and heartbeats. A heartbeat periodically refreshes the connection's lifetime so abandoned connections don't remain online forever.

This was one of those features that looked trivial from the UI but required much more thought on the backend.

7. Reconnection Is Part of the Feature

A realtime connection will eventually disappear.

The server can restart.

The network can change.

A laptop can wake up from sleep.

A reverse proxy can terminate a connection.

A user can temporarily lose connectivity.

So I didn't want the client to simply say:

Disconnected.
Good luck.

The client uses reconnection with increasing delays and jitter.

Conceptually:

Disconnect
    ↓
Retry
    ↓
1s
    ↓
2s
    ↓
4s
    ↓
...
    ↓
30s maximum

The randomization is important because if a server goes down while thousands of clients are connected, you don't want every client reconnecting at exactly the same moment.

That creates what's commonly called a thundering herd.

The client therefore uses exponential backoff with jitter and then re-syncs application state through the API after reconnecting.

The lesson for me was:

A realtime system isn't just about establishing a connection. It's also about recovering from losing one.

8. Database Design Started Becoming More Important

Once the application became more feature-rich, the database stopped being "just storage."

Humble's schema covers multiple domains including:

• users and profiles

• chats

• chat members

• messages

• posts

• comments

• notifications

• dating-related data

• moderation

• reports

The current design contains 23 models and a substantial number of indexes and unique constraints.

But the important part isn't simply having many indexes.

The important part is connecting an index to a real query.

For example:

Chat
  ↓
Messages
  ↓
createdAt

Chat history is commonly retrieved by chat and time, so the database has a composite index around:

(chatId, createdAt)

Other examples include indexes supporting:

These aren't indexes added because "production applications need indexes."

They exist because the application repeatedly asks those questions.

That changed how I think about database optimization.

Don't start with "Which indexes should I add?" Start with "Which queries will this feature execute repeatedly?"

9. Cursor Pagination Was Another Small Problem With a Big Lesson

Imagine a chat has thousands of messages.

You don't want to load everything.

So you paginate.

A simple approach might use:

OFFSET 100
LIMIT 50

But as datasets grow, offset-based pagination becomes less attractive for continuously changing data.

Humble uses cursor-based pagination for message history.

The interesting detail is that the cursor isn't based only on the timestamp.

It uses:

createdAt + id

Why?

Because two messages can have the same timestamp.

If I only compare timestamps:

Message A → 10:30:01.123
Message B → 10:30:01.123

the pagination boundary can become ambiguous.

Adding the ID as a tie-breaker creates a stable ordering.

This was a good example of something I learned by actually building the feature:

Correctness often lives in the edge cases, not the happy path.

10. I Started Looking for N+1 Problems

Another lesson was learning not to blindly query the database inside loops.

For example, if a chat has many members, this pattern can become expensive:

Get members

for each member:
    query something

Instead, Humble batches related reads and reuses the results where possible.

The message API combines membership, member-list and chat-state checks, then reuses the member information for subsequent cache invalidation rather than querying the same information again.

The general lesson is simple:

A database query that looks cheap once can become expensive when multiplied by users, messages and requests.

11. Caching Isn't Just "Put Everything in Redis"

The caching layer evolved into several strategies.

For example:

GET recent messages
        │
        ▼
   Cache exists?
     /       \
   yes        no
   │           │
   ▼           ▼
return      PostgreSQL
              │
              ▼
            cache

For writes, the system can update relevant cached data instead of always throwing the entire cache away.

The cache layer also has TTLs for different categories of data, and Redis being unavailable should result in a database fallback rather than a completely broken feature.

This led me to a broader rule:

A cache should improve the system, not become the only thing keeping it alive.

12. Rate Limiting Needs More Than One Layer

Realtime applications are especially vulnerable to event spam.

Things like:

typing:start
chat:join
message events

can happen much more frequently than normal HTTP requests.

So Humble has rate limiting at both the application/API level and the realtime event level.

The important design decision is that the two layers have different priorities.

The HTTP layer protects operations that affect durable application state.

The realtime layer protects event-heavy operations.

The realtime layer can also fail open when Redis is unavailable because, in this case, preserving availability is more important than completely enforcing the realtime event limit during a dependency outage.

This is another lesson I didn't appreciate initially:

Security and reliability aren't always about choosing "deny." Sometimes the correct decision depends on what you're protecting.

13. Notifications Taught Me About Abstraction

As Humble grew, different features started needing notifications.

Likes.

Comments.

Chat events.

Group changes.

Dating events.

Administrative actions.

I didn't want every API route to construct notification records differently.

So I separated notification creation from notification presentation.

Conceptually:

Feature
   │
   ▼
Notification Service
   │
   ├── recipient
   ├── actor
   ├── notification type
   ├── entity
   └── metadata
        │
        ▼
     Database
        │
        ▼
   Realtime delivery

New notification types are added through a shared notification service and templates rather than duplicating notification logic throughout the application. The documentation also includes rules such as avoiding self-notifications, preserving entity references and handling notification failures without breaking the main operation.

This isn't a revolutionary architecture.

But that's exactly why I like it.

Good abstractions don't always look impressive.

Sometimes they're simply the thing that prevents the tenth feature from becoming harder than the first.

14. Database Changes Need Their Own Discipline

Another part of building Humble that changed my engineering habits was database migration safety.

I documented a simple rule:

The Prisma schema and migration history are the source of truth.

The idea is to avoid casually modifying production tables and instead make schema changes through migrations.

The project documentation explicitly separates development migration workflows from production deployment workflows and emphasizes preserving migration history rather than resetting databases when drift is detected.

This may sound boring compared with Socket.IO and Redis.

But production engineering has a lot of boring things that become extremely important when something goes wrong.

15. Designing for Horizontal Scaling Doesn't Mean I Have Scaled Horizontally

This is probably the most important disclaimer in this article.

I designed Humble's realtime architecture so that multiple Socket.IO instances can eventually work together through Redis and a load-balancing layer.

The documented architecture looks roughly like:

                 Load Balancer
                 /     |     \
                /      |      \
        Socket 1   Socket 2   Socket 3
                \      |      /
                     Redis
                       │
                   PostgreSQL

The load-balancer documentation describes multiple socket instances sharing Redis for cross-instance communication.

But the current committed deployment is still a single-instance deployment.

The horizontal architecture is designed and documented; it is not something I want to pretend has been proven by a production load test.

That distinction matters.

There is a huge difference between:

"I designed a system that can scale horizontally."

and:

"I have demonstrated this system handling a specific amount of concurrent production traffic."

I currently have stronger evidence for the first statement than the second.

And I think being honest about that is part of engineering maturity.

16. What I Got Wrong or Still Need to Improve

I don't want this article to become a "look how perfect my architecture is" post.

It isn't perfect.

There are real gaps.

Automated testing

This is probably the biggest one.

The architecture contains failure paths for:

• Redis failure

• Redis recovery

• presence counting

• rate limiting

• realtime reconnection

• degraded operation

But the project documentation still identifies automated tests as a gap.

That is something I want to improve.

If I design a recovery mechanism, I should eventually have a test that proves the recovery mechanism still works.

Production performance measurement

I have thought carefully about indexes, caching and architecture.

But design assumptions are not the same as measurements.

I still want stronger production-level visibility into things such as:

• p95 and p99 API latency

• slow database queries

• cache hit rate

• realtime connection behavior

• message throughput

• failure frequency

The current architecture exposes health and metrics information, but the documentation also identifies the lack of production query/APM measurements as a remaining gap.

Delivery guarantees

There is another interesting problem I haven't completely solved.

Suppose:

Database write succeeds
        ↓
Broadcast succeeds
        ↓
Client disconnects immediately

Did the client actually receive the event?

Not necessarily.

The current approach relies on the client re-syncing state from the API after reconnecting.

A future improvement could involve stronger delivery acknowledgements or sequence-based reconciliation.

That is an important distinction between:

"The server emitted an event."

and:

"The client has confirmed receiving and processing that event."

I don't want to pretend those are the same thing.

17. The Biggest Change Wasn't Technical

When I started, I mostly thought about:

Does the feature work?

Now I find myself asking:

What happens when it fails?

What happens when the dependency is unavailable?

What happens when the user has two tabs?

What happens when the database gets large?

What happens when the request is repeated?

What happens when the server restarts?

What happens when the network disappears?

What happens when this becomes ten times larger?

That shift in questions has probably been more valuable to me than learning any individual technology.

Redis wasn't the biggest lesson.

Socket.IO wasn't the biggest lesson.

PostgreSQL wasn't the biggest lesson.

The biggest lesson was learning to think in failure modes and trade-offs.

18. How I Documented the System

One thing I deliberately started doing while working on Humble was documenting the reasoning behind the implementation.

Not just:

Added Redis.

But:

Why do we need Redis?

What happens if Redis fails?

What state can safely disappear?

What state must survive?

How does the system recover?

What is currently deployed?

What is only planned?

What remains unfinished?

That resulted in separate documentation for the high-level architecture, low-level implementation, Redis failure handling, database migration safety, notifications, upgrade progress and load-balancing strategy.

The HLD explicitly distinguishes the current deployment from the planned horizontal-scaling architecture, while the LLD documents actual entities, indexes, state machines and algorithms from the codebase.

For me, documentation became part of the engineering process rather than something written after the engineering was finished.

19. The Way I Think About Engineering Now

If I had to reduce what I learned from Humble into a few principles, they would be:

20. What I Want to Build Next

The next stage isn't simply adding more infrastructure.

I want to make the existing engineering easier to prove.

That means:

1. Add automated tests around Redis failure and recovery.

2. Test reference-counted presence.

3. Add stronger realtime delivery reconciliation.

4. Run proper load tests instead of relying only on estimates.

5. Measure database and API performance under realistic workloads.

6. Improve production observability.

7. Validate the documented multi-instance architecture in an actual environment.

The goal isn't to say:

"Humble can handle X users."

The goal is to be able to answer:

"Here is how we tested it, here is what happened, here is where it failed, and here is what we changed."

That is a much more useful engineering statement.

21. What Building Humble Means to Me as a Developer

I'm still learning.

I don't consider myself someone who has already solved distributed systems.

But building Humble has given me the opportunity to encounter many of the problems that make backend engineering interesting:

• consistency

• persistence

• realtime delivery

• caching

• presence

• rate limiting

• database indexing

• failure recovery

• deployment

• observability

• scalability

• technical debt

And the more I work on it, the more I realize that writing the code is only one part of the job.

The other part is being able to explain:

Why does this exist?

What happens when it fails?

What trade-off did I make?

What evidence do I have that it works?

What would I change if the system grew?

For me, that is what Humble has become—not just a campus product, but a practical place to learn how software behaves outside the happy path.

A simple mental model I now use

                ┌─────────────────────┐
                │      User need      │
                └──────────┬──────────┘
                           │
                           ▼
                    Build the feature
                           │
                           ▼
                    Make it reliable
                           │
                           ▼
                    Think about failure
                           │
                           ▼
                    Measure the result
                           │
                           ▼
                    Document the why
                           │
                           ▼
                    Improve and repeat

Good engineering isn't only making the happy path work. It's understanding what happens when the happy path stops working.

Technical stack

Humble is being built as a real product rather than only as a technical experiment, so the architecture continues to evolve alongside the product itself. (humbleapp.in)

Project: humbleapp.in

Built and documented by Prashant Kumar