What actually breaks at 100,000 users (that never breaks in a demo)

The first time I shipped a dashboard to a hundred thousand active users, nothing I had tested was what broke. Everything that failed had passed every check I ran. It worked on my machine, it worked in the demo, it worked in the client walkthrough. Then real traffic arrived, and the same code fell over.

That gap, between "works in the demo" and "works in production," is where a lot of a senior engineer's judgment actually lives. Here's what I've learned tends to fall into it.

1. The query that was fine at fifty rows

The demo database had fifty rows. Production had two million. A page that loaded instantly on my laptop took nine seconds live, and the cause was a single Eloquent relationship lazy-loading inside a loop. One query per row. At fifty rows that's fifty queries in a few milliseconds, invisible. At two million it's a self-inflicted outage.

Nobody writes an N+1 on purpose. You write $order->customer->name inside a loop, it reads perfectly, and the framework quietly fires a query every iteration. The fix is boring, with() to eager-load. But the lesson isn't the fix. It's that the bug was there the whole time, sleeping, waiting for the data to get big enough to wake it.

2. The index you didn't know you needed

A where('email', $email) on a users table is instant, until the table has a few million rows and no index on email. Then every lookup is a full table scan, and the database does the one thing it's meant to protect you from: it reads everything to find one thing.

Indexes are invisible in development because scanning a small table is free. Add them for every column you filter or sort on in a hot path, and read your slow query log before you assume you have. The demo will never tell you.

3. The job that assumed it runs exactly once

Queues retry. That's the point of them. But it means your job can run twice, on a network blip, a timeout, a deploy mid-run. In the demo it always ran once, so "charge the customer" or "send the email" looked safe. At scale, the retry fires, and someone gets billed twice.

Design jobs to be idempotent: check whether the work is already done before doing it, key side effects to something unique, make the second run a no-op. You don't feel this at low volume because retries are rare. At high volume, rare happens every hour.

4. The cache that became the bottleneck

Caching is the first thing everyone reaches for, and at scale it grows its own failure mode. A popular key expires, and a thousand requests all miss at the same instant and all hit the database to rebuild it. The cache that was protecting your database just handed it a thousand identical queries in one second.

You solve it with staggered expiry, a lock around the rebuild, or serving the stale value while one worker refreshes. None of it matters at demo traffic, because your one test user never races themselves.

The pattern underneath all of them

Scale doesn't introduce new bugs. It takes the small inefficiencies you already shipped, the ones that were technically wrong but practically invisible, and turns the volume all the way up until they're the only thing you can hear.

So you don't design for scale by adding servers. You design for it by assuming the loop will run a million times, the query will return more than you expect, the job will run twice, and the thing that's fast today is fast only because the data is still small.

Small data hides everything. The demo is small data. That is exactly why you can't trust what it tells you.