Designing high-load REST APIs: what the first production projects taught us

Over the past six months several backends have left our hands that have to serve hundreds of requests per second, and where response time directly affects the client’s business. In this post we collect what came up again and again in those projects. There is nothing revolutionary here; it is more of a checklist we wish we had written for ourselves earlier.

The database is the bottleneck, not Node.js

The first lesson: the Node.js event loop is almost never what runs out. Under NestJS, a well-written endpoint handles requests in the thousands per second in a single process, provided it is not waiting on the database. In practice, it is almost always waiting on the database.

What we consequently do in every project:

• Indexes based on real queries. Turning on pg_stat_statements is the first step in production. We review the slowest and most frequent queries with EXPLAIN (ANALYZE, BUFFERS), and we only add an index on the column combinations where the plan actually shows a sequential scan. The “index every foreign key” rule is a good starting point, but it does not replace measurement.
• Connection pooling on purpose. The number of Node.js processes multiplied by the pool size must not exceed PostgreSQL’s max_connections, and some headroom has to be left for maintenance. With multiple instances, PgBouncer goes between the processes and the database in transaction mode; at that point you have to give up session-level settings and prepared statements, and it is worth knowing that in advance.
• Short transactions. We never call an external service inside a transaction. If that is unavoidable, we close the transaction before the call and write the result back separately.

Redis: cache, rate limit and lock in one place

Redis plays three roles for us, and each has to be configured differently.

Cache. For read-heavy endpoints (lists, configuration, rarely changing master data) we store the response by key, with a short TTL. We do not try to design perfect cache invalidation: in most cases a few seconds of staleness is acceptable, and it is far simpler than event-driven eviction tied to writes. Where that is not acceptable, we simply do not cache.

Rate limiting. We keep a sliding-window counter per client (by API key or IP), and return the X-RateLimit-* headers in the response. The limit is checked in a global NestJS guard rather than inside each endpoint, so it cannot be forgotten.

Distributed lock. If more than one instance can modify the same resource (processing an external payment callback, for example), we use a short-lived SET NX lock. Not Redlock, just a single instance; fault tolerance comes from idempotent processing, not from the lock.

Idempotency and pagination

Two things that are easy to neglect and hard to retrofit.

An idempotency key is mandatory for us on every mutating endpoint that an external system or a mobile app can call. The client sends an Idempotency-Key header, the server stores the response associated with that key for a while, and returns the same response on a repeat call. This is what saves the system from duplicate orders and messages caused by network retries.

For pagination, the OFFSET-based approach slows down noticeably above tens of thousands of rows, and with a moving data set it skips or repeats items. We switched to cursor-based pagination: the client gets back the sort key of the last item it saw and asks for the next page with it. This is less convenient for “jump to page 47” interfaces, but for APIs it is almost always better.

Observability: we cannot fix what we do not measure

The most important lesson from the first weeks in production is that log files alone are not enough. What we build into every system:

• structured (JSON) logging with a request ID that can be followed from the incoming request down to the database query,
• Prometheus metrics per endpoint: request count, error rate, response-time percentiles (p50, p95, p99),
• the state of the database pool (number of waiters, busy connections) as a metric,
• a health check endpoint that verifies both the database and Redis, and that both Docker and Nginx watch.

The p99 response time is the one worth watching. The average is almost always pretty; p99 shows what one percent of users actually experience.

Load testing, before going live

Last but not least: we do not believe our own estimates. Before every launch we run a scenario with k6 or a similar tool that simulates a multiple of the expected peak load, against a database filled with a realistic amount of data. We run the test on staging, which runs in Docker with the same configuration as production.

What these tests typically surface: a missing index, a pool that is too small, an N+1 query the ORM hid, or an endpoint that returns several megabytes of JSON for a single request. Every one of them is cheaper to find in a staging test than on the first day of a campaign.

In the next post we write about multi-tenant SaaS architectures, where these problems gain an extra dimension: tenants affecting each other.