From 4.2 seconds to 180 milliseconds: a cold-start postmortem

The number showed up on a Thursday. Our largest customer had been quietly filing tickets for two weeks about the API being "a little slow in the morning," and when we finally sat down with their request logs, the p99 cold-start latency read 4,214 milliseconds. Four seconds before a single byte of their request was processed, and that was the slow end of the distribution. We had been telling ourselves a story about how serverless cold starts were a known, bounded cost. The logs told a different story, and it was the kind of story that ends with someone rewriting the module loader on a Friday night.

This is a postmortem in the engineering sense: not a search for blame, but a patient reconstruction of how we got to four seconds, how the measurement tools hid the worst of it from us, and the sequence of changes that brought the number down by a factor of twenty-three. None of the individual fixes were clever. The lesson, as usual, was that we had been measuring the wrong thing and trusting the measurement.

01What "cold start" actually means here

A cold start is the delay between a serverless function's first invocation and the moment it is ready to handle a request. The platform has to allocate a container, load your code, run the top-level module graph, and only then call the handler. For a warm invocation — one that hits an already-running instance — none of that happens. The function is already in memory, and the request is served in milliseconds. Most traffic never sees a cold start at all, which is precisely why they are so easy to ignore.

The trouble is that "most traffic" is a statement about averages, and averages are where reliability goes to die. Our customer ran batch jobs at 9 a.m. every weekday. Those jobs scaled the function from zero to forty concurrent instances, every one of them cold, every morning, at the exact moment their traders were waiting on the result. The 4.2-second number wasn't a tail. For that customer, at that hour, it was the modal experience. A function that is fast on average and broken every morning is, for practical purposes, broken.

02Measuring without lying to yourself

The first thing we got wrong was the dashboard. We had instrumented the handler — the part of the function that runs after initialization — so our p99 chart showed a healthy 40 milliseconds and we felt good about it. That number was real, but it was the answer to a different question. The customer was asking "how long until my request is handled," and the answer included the four seconds we had decided not to measure.

We separated the two populations and started charting cold and warm latency independently, with the cold-start share of traffic as its own line. The fix to the dashboard took an afternoon. The fix to the latency took the next three weeks, but we would never have started if the chart had kept lying to us. Measuring the right boundary — from the platform's arrival time, not from when our code decided to start the clock — is the unglamorous prerequisite for every optimization that follows.

An optimization you cannot see is an optimization you will not make. The dashboard is not a reporting tool; it is the thing that decides which problems are allowed to exist.

03Four layers, four liars

Once we could see the latency, we had to find where it lived. A cold start is not one cost; it is a stack of costs, and each layer is happy to blame the one beneath it. We broke ours into four — platform provisioning, container image pull, module evaluation, and application initialization — and timed each in isolation. The results were not what any of us expected, and they disagreed, loudly, with the assumptions we had carried into the project.

Platform provisioning, the part we had blamed the loudest, turned out to be 300 milliseconds and utterly outside our control. The container image pull was 200 milliseconds, reduced to near-zero by moving to a slimmer base image. That left 3.7 seconds in the two layers we actually owned, and 3.4 of those lived in a single file the runtime was importing on every cold start.

The file was a country and currency lookup table — 48,000 lines of JSON — imported at the top of the module graph so that a handful of helper functions could reference it. No handler had ever read more than a dozen rows of it, but the runtime dutifully parsed and evaluated the entire thing on every cold start, because that is what an import asks for. We had written that import eighteen months earlier, for a feature that was since removed, and the table had been paying a four-second tax on every fresh instance ever since.

The most expensive code in our system was a file nobody read, imported by a feature nobody used, on a path the profiler refused to flag because imports "don't count" as application work. — the lesson that rewrote our onboarding checklist

04Killing the import waterfall

The fix for the lookup table was straightforward in hindsight, but it required unlearning a habit. We had always imported everything at the top of the file, because that is what linters and style guides tell you to do, and because it keeps the dependency graph explicit. That habit is fine for a long-running server that pays the import cost once and amortizes it across a million requests. It is a quiet disaster for a serverless function that pays the cost on every cold start and amortizes it across a handful of requests before being frozen.

We converted the lookup to a lazily-loaded module and moved the heavy parse behind a memoized accessor. The table now reads from disk and parses only on the first call that actually needs it, and only the slice of it that the caller requests. The change is small, but it inverts the cost model: we pay for what we use, not for everything we might use.

// before: parse 48k lines on every cold start
import { table } from './currency-table.json'

export function symbolFor(code: string) {
  return table[code]?.symbol ?? code
}

// after: parse lazily, only the rows requested
let cache = new Map()

export async function symbolFor(code: string) {
  if (!cache.has(code)) {
    const row = await readRow(code) // streams one record
    cache.set(code, row)
  }
  return cache.get(code).symbol
}

The lazy variant dropped cold-start latency from 4.2 seconds to 1.1 seconds in a single change, which is the kind of result that makes you suspicious. We profiled it three more times, on three different instance sizes, before we believed it. One deleted import, eighteen months of tax, gone in an afternoon. The lesson was not about JSON files specifically; it was about the difference between an import that is cheap to write and expensive to run, and how invisible that gap becomes in a serverless model.

05What did not work

Not every attempt paid off, and the failures are worth recording because they point at the same lesson from a different angle. We spent two days on provisioned concurrency — the platform feature that keeps instances warm by paying for idle capacity — and it worked, but it worked by spending money rather than by fixing the function. For our traffic pattern, with its sharp 9 a.m. ramp, the cost of staying warm through the night dwarfed the cost of the cold starts it eliminated. Provisioned concurrency is a valid tool, but it is a bandage, and bandages are worth questioning when the wound is an import statement.

We also tried tree-shaking the module graph with a custom bundler pass, hoping to strip dead code automatically. The bundler removed a respectable 40 percent of the transitive dependencies, and cold-start latency did not move. The dead code was never the expensive code; the expensive code was the one live import that pulled a 48,000-line file. Removing a hundred small imports to leave the one giant one is the optimization equivalent of tidying a room while the house is on fire.

The third failure was the most instructive. We wrote a start-up benchmark that imported the handler module in a loop and timed the top-level evaluation, assuming it would mirror production. It did not. The benchmark ran in a warm runtime with a hot disk cache, so it measured module resolution, not the cold parse we were paying for in production. We had built a fast, confident, and entirely wrong measurement loop, and it told us we were done two days before we actually were. Benchmarks inherit the assumptions of the environment that runs them, and a warm benchmark cannot find a cold-start cost any more than a dry run can find a leak.

06The last four hundred milliseconds

After the import fix we were at 1.1 seconds, which was no longer embarrassing but was still slow for an API that ought to feel instant. The remaining time split into three roughly equal pieces: a database connection pool that opened eagerly on cold start, a feature-flag client that fetched its entire config over the network before the first request, and a logging initialization routine that walked the filesystem synchronously to discover log handlers.

Each of these had the same shape as the JSON file: code that did its work eagerly, at module load time, on the assumption that the cost would be amortized. Each yielded to the same treatment — defer the work until it is actually needed, then do the minimum. The connection pool now opens on the first query rather than at construction. The flag client returns a local default immediately and refreshes in the background. The logger discovers handlers once and caches the result.

None of these were heroic changes, and that is the point. Once we had the right measurement and the right mental model — defer everything, pay only for what each request actually needs — the fixes were small and obvious. The hard part had been seeing the problem at all, and that had required us to stop trusting a chart that was answering a different question than the one we were asking. The last four hundred milliseconds were easy. The first four seconds were the part that demanded we change how we measured.

07What we learned

If there is a single takeaway, it is that the boundary you measure defines the problems you are allowed to see. We had a fast handler and a slow function, and for months we believed the fast part was the whole. Serverless does not forgive that confusion, because the initialization cost — the part that lives outside the handler — is exactly the part that traditional monitoring is built to ignore. The fix began the moment we drew the boundary at the platform, not at the code.

The second lesson is about defaults. Top-level imports, eager singletons, and load-time configuration are patterns we inherited from server software, where they are harmless. In a serverless runtime, those same patterns become taxes levied on every cold start, collected whether or not the work is needed. Laziness is not an optimization in this model; it is the correct default, and eagerness is the thing that should require a comment.

The last lesson is the one I keep returning to: the file that cost us the most was written by someone doing exactly the right thing for the wrong context. Good habits, carried across a context boundary without re-examination, become the most expensive bugs in the system. We did not ship a faster function by being clever. We shipped it by noticing that a habit we trusted had quietly become a four-second lie, and by being willing to measure it honestly even after the chart said we were fine.

❦