13 October 2026, 05:50 PM
[attachment=9631]
What Is Performance Testing? Types, Process, Benefits, and Best Practices
A system can be perfectly correct and still be broken.
Picture a checkout page. One tester clicks through it and everything works: the totals are right, the confirmation email arrives, the order lands in the database. Green across the board. Then a promotion goes out, a few hundred people hit the same page in the same minute, and the page takes forty seconds to respond. Nothing is wrong with the logic. Every request would eventually succeed. Customers leave anyway.
That gap between "works" and "works when it matters" is exactly what performance testing exists to measure. It asks how a system behaves under load: how fast it responds, how many requests it can handle, how it fails, and how quickly it recovers. Functional testing tells you the answer is right. Performance testing tells you whether the answer arrives in time, for everyone who asked at once.
I'm going to organize this article around the questions you'd actually ask, because in my experience the biggest mistake teams make isn't running the wrong tool. It's running the wrong test for the question they have.
[attachment=9632]
"Will it hold up on a normal busy day?"
That's a load test. You apply the traffic you realistically expect at peak, using the mix of actions real users perform, and watch response times, error rates and resource use.
It's the foundation, and the one most teams run first. The key word is realistic. A load test that hammers one endpoint with identical requests tells you about that endpoint. A test that mixes browsing, searching, logging in and paying in the proportions you see in production tells you about the system.
"Where does it break, and how?"
That's a stress test. You push beyond expected peak, past the point where things degrade, to find the limit and to see what failure looks like.
The interesting result isn't the number. It's the behavior. Does the system slow down gracefully and shed load, or does it collapse, corrupt data, and need a manual restart? A system that fails safely at twice your peak is in better shape than one that survives 20 percent more but falls over in a messy way.
"What if traffic jumps in seconds?"
That's a spike test. Instead of a steady ramp, load leaps suddenly and then drops. This exposes things a gentle ramp hides: autoscaling that takes minutes to react, cold caches, database connection storms, and queues that fill faster than they drain. If your business has flash sales, push notifications to your whole user base, or scheduled events, this one earns its place.
"Does it get worse the longer it runs?"
That's a soak test, sometimes called an endurance test. Moderate load held for hours, or longer. It finds slow problems: memory leaks, connection pools that never fully release, logs that fill disks, caches that grow without bound. These bugs are invisible in a twenty-minute test and obvious on day three in production.
"Will adding servers actually help?"
That's a scalability test. You increase resources and load together and measure whether capacity grows in proportion. It's how you discover that the shared database, the single message broker or a global lock is the real ceiling, no matter how many application servers you add. Doing this before a big infrastructure spend can save a lot of money.
"How does it behave with years of data?"
That's a volume test, or data-scale testing. Many systems feel fast with a few thousand rows and slow with a hundred million. Queries that used a good index at small scale pick a bad plan at large scale. Reports that took seconds now time out. Test with production-sized data, or the closest realistic equivalent you can create.
The Process, Told as a Story
Here's how a sensible engagement unfolds. I'll tell it as a sequence because the order matters.
It starts with a goal, not a tool. Before anyone writes a script, someone has to say what "good enough" means. Not "fast." Something like: ninety-five percent of checkout requests complete within a stated time at expected peak, with an error rate under an agreed ceiling. These are your service level objectives, and they come from the business and from user expectations, not from whatever the system currently does.
Next comes the workload model. Use production analytics to learn what people actually do, in what proportions, and how fast. Include think time, since real users pause. Decide whether your load is best modeled as a fixed number of users, or as a fixed arrival rate of requests. This distinction matters more than most beginners expect. Real-world traffic usually behaves like an arrival rate: new people keep arriving whether or not earlier ones have been served. If you model it as a fixed pool of users who wait for each response, a slowing system automatically receives less load, which flatters it.
Then the environment and data. A test environment that's a tenth the size of production tells you about a tenth-size system, and scaling results up isn't reliable. Aim for production-like architecture, the same configuration, and realistic data volumes. Where that's not affordable, be explicit about what the differences mean for your conclusions.
Then scripts and a baseline. Script the journeys, validate that they behave like real users (correct responses, not just fast ones), and run a small baseline first. A baseline gives you a reference to compare every later change against.
Here's a short example using k6. It models a steady arrival rate and fails the run if the targets are missed. The numbers are placeholders to show the shape, not recommendations.
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
checkout: {
executor: 'ramping-arrival-rate',
startRate: 10,
timeUnit: '1s',
preAllocatedVUs: 100,
maxVUs: 500,
stages: [
{ target: 50, duration: '5m' }, // ramp up
{ target: 50, duration: '20m' }, // hold at expected peak
],
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(95)<800', 'p(99)<1500'],
},
};
export default function () {
const res = http.post('https://staging.example.com/api/checkout', JSON.stringify({ cartId: 'test-cart' }),
{ headers: { 'Content-Type': 'application/json' } });
check(res, { 'status is 200': ® => r.status === 200 });
}
JMeter, Gatling, Locust and others can express the same ideas. Pick the tool your team can maintain, not the one with the best demo.
Then execution and observation. Run the test while watching the system, not just the client. Response times tell you that something slowed down. CPU, memory, thread and connection pools, queue depth, garbage collection and database metrics tell you why. A performance test without server-side observability produces a graph and an argument.
Then analysis. Look at percentiles, not averages. An average of 300 milliseconds can hide the fact that one request in twenty takes five seconds. Look for the point where latency bends upward as load increases: that "knee" is your practical capacity, usually well before anything crashes. Check that errors didn't quietly rise, and that the load generator itself wasn't the bottleneck.
Finally, tune, retest, and keep the test. Fix the biggest bottleneck, rerun, and repeat. Fix one thing at a time or you won't know what helped. Then put a smaller version of the test into your delivery pipeline so a regression is caught the week it's introduced.
A Piece of Arithmetic Worth Knowing
There's a simple relationship, often called Little's Law, that explains a lot of mysterious bottlenecks: the average number of requests in the system equals the arrival rate multiplied by the time each request spends in it.
Here's an illustration with made-up numbers. If requests arrive at 200 per second and each takes 250 milliseconds, then about 50 requests are in flight at any moment. If your service has a pool of 40 workers, requests will queue even though the CPU looks idle. Then queueing increases response time, which increases the number of requests in flight, which makes the queue longer. That feedback loop is how a small slowdown turns into an outage. A few minutes with this formula and your real numbers can explain a result before you've even run a test.
What You Actually Gain
Teams often ask me to justify the cost. The honest benefits are these.
Fewer surprises at the worst moment. You discover limits on your schedule instead of your customers'.
Better capacity decisions. You can say how much headroom you have and what the next bottleneck is, which turns infrastructure spending into an informed choice instead of a guess.
Cheaper fixes. A query that needs an index is far cheaper to find in a test than during an incident.
Evidence for business events. Before a launch or a seasonal peak, you can show what the system can handle.
Regression protection. Performance quietly erodes with each release. Automated checks catch the slide early.
I'd be cautious about promising a specific return on investment. The value shows up as incidents that never happened, which are hard to count. Frame it as risk reduction for the journeys that matter most to revenue and trust.
Best Practices That Survive Contact With Reality
Where Teams Go Wrong
Testing too late. Discovering at the end of a project that the architecture can't scale is the most expensive way to learn it. Test early, even with small experiments.
Averages as the headline. They hide the customers having the worst experience.
Only testing the happy path. Real traffic includes failed logins, retries, empty searches and abandoned carts, and these often stress the system in different ways.
Closed-loop models that flatter the system. As described above, a fixed pool of waiting users reduces load exactly when the system struggles.
Ignoring the database. Many performance problems are query problems. Include data volume and realistic access patterns.
Chasing a number instead of a goal. "10,000 users" means nothing without a workload model and a target. Start from what the business needs.
One big test before launch. Performance is a property that changes with every release. A single test is a snapshot.
When not to run a heavy performance test: for low-risk internal tools with a handful of users, a quick load check is plenty. Save the full treatment for customer-facing, revenue-critical and shared-infrastructure systems.
Where AI Fits In, Including the Systems Built on It
AI tooling helps in several places. It can analyze production logs to suggest realistic workload mixes, detect anomalies in latency and error series that a human would miss on a dashboard, cluster failures to point at likely causes, and forecast capacity from trends. Those are useful accelerators.
It shouldn't be the judge. Whether a system meets its objective is a numerical comparison against criteria that people have agreed on.
Keep that explicit and reviewable.
If your product includes an LLM or retrieval-based feature, add a few dimensions to your performance thinking. Measure time to first token and tokens per second, not only total response time, because users perceive the first as responsiveness. Test behavior when the model provider rate-limits or slows down. Test how latency changes with prompt length and with the number of retrieved documents. Watch cost under load, since a traffic spike can become a budget spike. And apply the usual rule: a slow dependency you don't control needs timeouts, fallbacks and graceful degradation, which you should verify under load.
A Closing Thought
Here's an exercise for your next planning meeting. Ask each person to write down the single moment their system is most likely to be under stress in the next six months, and what they believe will break first. Then compare the answers. The disagreement is your test plan.
I expect performance work to shift from an event before big launches toward a constant background signal: small, automated checks on every build, regular capacity rehearsals tied to the business calendar, and observability that makes production itself part of the experiment. If you'd like experienced help designing that, our Performance Testing team works with engineering and product leaders on exactly this kind of readiness.
What Is Performance Testing? Types, Process, Benefits, and Best Practices
A system can be perfectly correct and still be broken.
Picture a checkout page. One tester clicks through it and everything works: the totals are right, the confirmation email arrives, the order lands in the database. Green across the board. Then a promotion goes out, a few hundred people hit the same page in the same minute, and the page takes forty seconds to respond. Nothing is wrong with the logic. Every request would eventually succeed. Customers leave anyway.
That gap between "works" and "works when it matters" is exactly what performance testing exists to measure. It asks how a system behaves under load: how fast it responds, how many requests it can handle, how it fails, and how quickly it recovers. Functional testing tells you the answer is right. Performance testing tells you whether the answer arrives in time, for everyone who asked at once.
I'm going to organize this article around the questions you'd actually ask, because in my experience the biggest mistake teams make isn't running the wrong tool. It's running the wrong test for the question they have.
[attachment=9632]
"Will it hold up on a normal busy day?"
That's a load test. You apply the traffic you realistically expect at peak, using the mix of actions real users perform, and watch response times, error rates and resource use.
It's the foundation, and the one most teams run first. The key word is realistic. A load test that hammers one endpoint with identical requests tells you about that endpoint. A test that mixes browsing, searching, logging in and paying in the proportions you see in production tells you about the system.
"Where does it break, and how?"
That's a stress test. You push beyond expected peak, past the point where things degrade, to find the limit and to see what failure looks like.
The interesting result isn't the number. It's the behavior. Does the system slow down gracefully and shed load, or does it collapse, corrupt data, and need a manual restart? A system that fails safely at twice your peak is in better shape than one that survives 20 percent more but falls over in a messy way.
"What if traffic jumps in seconds?"
That's a spike test. Instead of a steady ramp, load leaps suddenly and then drops. This exposes things a gentle ramp hides: autoscaling that takes minutes to react, cold caches, database connection storms, and queues that fill faster than they drain. If your business has flash sales, push notifications to your whole user base, or scheduled events, this one earns its place.
"Does it get worse the longer it runs?"
That's a soak test, sometimes called an endurance test. Moderate load held for hours, or longer. It finds slow problems: memory leaks, connection pools that never fully release, logs that fill disks, caches that grow without bound. These bugs are invisible in a twenty-minute test and obvious on day three in production.
"Will adding servers actually help?"
That's a scalability test. You increase resources and load together and measure whether capacity grows in proportion. It's how you discover that the shared database, the single message broker or a global lock is the real ceiling, no matter how many application servers you add. Doing this before a big infrastructure spend can save a lot of money.
"How does it behave with years of data?"
That's a volume test, or data-scale testing. Many systems feel fast with a few thousand rows and slow with a hundred million. Queries that used a good index at small scale pick a bad plan at large scale. Reports that took seconds now time out. Test with production-sized data, or the closest realistic equivalent you can create.
The Process, Told as a Story
Here's how a sensible engagement unfolds. I'll tell it as a sequence because the order matters.
It starts with a goal, not a tool. Before anyone writes a script, someone has to say what "good enough" means. Not "fast." Something like: ninety-five percent of checkout requests complete within a stated time at expected peak, with an error rate under an agreed ceiling. These are your service level objectives, and they come from the business and from user expectations, not from whatever the system currently does.
Next comes the workload model. Use production analytics to learn what people actually do, in what proportions, and how fast. Include think time, since real users pause. Decide whether your load is best modeled as a fixed number of users, or as a fixed arrival rate of requests. This distinction matters more than most beginners expect. Real-world traffic usually behaves like an arrival rate: new people keep arriving whether or not earlier ones have been served. If you model it as a fixed pool of users who wait for each response, a slowing system automatically receives less load, which flatters it.
Then the environment and data. A test environment that's a tenth the size of production tells you about a tenth-size system, and scaling results up isn't reliable. Aim for production-like architecture, the same configuration, and realistic data volumes. Where that's not affordable, be explicit about what the differences mean for your conclusions.
Then scripts and a baseline. Script the journeys, validate that they behave like real users (correct responses, not just fast ones), and run a small baseline first. A baseline gives you a reference to compare every later change against.
Here's a short example using k6. It models a steady arrival rate and fails the run if the targets are missed. The numbers are placeholders to show the shape, not recommendations.
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
checkout: {
executor: 'ramping-arrival-rate',
startRate: 10,
timeUnit: '1s',
preAllocatedVUs: 100,
maxVUs: 500,
stages: [
{ target: 50, duration: '5m' }, // ramp up
{ target: 50, duration: '20m' }, // hold at expected peak
],
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(95)<800', 'p(99)<1500'],
},
};
export default function () {
const res = http.post('https://staging.example.com/api/checkout', JSON.stringify({ cartId: 'test-cart' }),
{ headers: { 'Content-Type': 'application/json' } });
check(res, { 'status is 200': ® => r.status === 200 });
}
JMeter, Gatling, Locust and others can express the same ideas. Pick the tool your team can maintain, not the one with the best demo.
Then execution and observation. Run the test while watching the system, not just the client. Response times tell you that something slowed down. CPU, memory, thread and connection pools, queue depth, garbage collection and database metrics tell you why. A performance test without server-side observability produces a graph and an argument.
Then analysis. Look at percentiles, not averages. An average of 300 milliseconds can hide the fact that one request in twenty takes five seconds. Look for the point where latency bends upward as load increases: that "knee" is your practical capacity, usually well before anything crashes. Check that errors didn't quietly rise, and that the load generator itself wasn't the bottleneck.
Finally, tune, retest, and keep the test. Fix the biggest bottleneck, rerun, and repeat. Fix one thing at a time or you won't know what helped. Then put a smaller version of the test into your delivery pipeline so a regression is caught the week it's introduced.
A Piece of Arithmetic Worth Knowing
There's a simple relationship, often called Little's Law, that explains a lot of mysterious bottlenecks: the average number of requests in the system equals the arrival rate multiplied by the time each request spends in it.
Here's an illustration with made-up numbers. If requests arrive at 200 per second and each takes 250 milliseconds, then about 50 requests are in flight at any moment. If your service has a pool of 40 workers, requests will queue even though the CPU looks idle. Then queueing increases response time, which increases the number of requests in flight, which makes the queue longer. That feedback loop is how a small slowdown turns into an outage. A few minutes with this formula and your real numbers can explain a result before you've even run a test.
What You Actually Gain
Teams often ask me to justify the cost. The honest benefits are these.
Fewer surprises at the worst moment. You discover limits on your schedule instead of your customers'.
Better capacity decisions. You can say how much headroom you have and what the next bottleneck is, which turns infrastructure spending into an informed choice instead of a guess.
Cheaper fixes. A query that needs an index is far cheaper to find in a test than during an incident.
Evidence for business events. Before a launch or a seasonal peak, you can show what the system can handle.
Regression protection. Performance quietly erodes with each release. Automated checks catch the slide early.
I'd be cautious about promising a specific return on investment. The value shows up as incidents that never happened, which are hard to count. Frame it as risk reduction for the journeys that matter most to revenue and trust.
Best Practices That Survive Contact With Reality
- Test the journeys that matter, not every endpoint. Prioritize by business impact and by traffic.
- Use percentiles and error rates as pass criteria, and agree on them before the run, so nobody adjusts the target after seeing the result.
- Keep data realistic. Tiny, uniform test data hides caching effects and bad query plans.
- Isolate the test environment so other teams' activity doesn't contaminate results, and so your test doesn't affect real users.
- Warm up first. Cold caches and just-in-time compilation skew early measurements.
- Check the load generator's health. If it runs out of CPU or network, you're measuring the generator.
- Test the dependencies you can't control with stubs that mimic their latency and failures, and verify against the real thing carefully and sparingly.
- Repeat runs. One run can be an outlier. Look for consistency before trusting a number.
- Treat results as evidence, not proof. No test environment matches production exactly. Record the differences.
Where Teams Go Wrong
Testing too late. Discovering at the end of a project that the architecture can't scale is the most expensive way to learn it. Test early, even with small experiments.
Averages as the headline. They hide the customers having the worst experience.
Only testing the happy path. Real traffic includes failed logins, retries, empty searches and abandoned carts, and these often stress the system in different ways.
Closed-loop models that flatter the system. As described above, a fixed pool of waiting users reduces load exactly when the system struggles.
Ignoring the database. Many performance problems are query problems. Include data volume and realistic access patterns.
Chasing a number instead of a goal. "10,000 users" means nothing without a workload model and a target. Start from what the business needs.
One big test before launch. Performance is a property that changes with every release. A single test is a snapshot.
When not to run a heavy performance test: for low-risk internal tools with a handful of users, a quick load check is plenty. Save the full treatment for customer-facing, revenue-critical and shared-infrastructure systems.
Where AI Fits In, Including the Systems Built on It
AI tooling helps in several places. It can analyze production logs to suggest realistic workload mixes, detect anomalies in latency and error series that a human would miss on a dashboard, cluster failures to point at likely causes, and forecast capacity from trends. Those are useful accelerators.
It shouldn't be the judge. Whether a system meets its objective is a numerical comparison against criteria that people have agreed on.
Keep that explicit and reviewable.
If your product includes an LLM or retrieval-based feature, add a few dimensions to your performance thinking. Measure time to first token and tokens per second, not only total response time, because users perceive the first as responsiveness. Test behavior when the model provider rate-limits or slows down. Test how latency changes with prompt length and with the number of retrieved documents. Watch cost under load, since a traffic spike can become a budget spike. And apply the usual rule: a slow dependency you don't control needs timeouts, fallbacks and graceful degradation, which you should verify under load.
A Closing Thought
Here's an exercise for your next planning meeting. Ask each person to write down the single moment their system is most likely to be under stress in the next six months, and what they believe will break first. Then compare the answers. The disagreement is your test plan.
I expect performance work to shift from an event before big launches toward a constant background signal: small, automated checks on every build, regular capacity rehearsals tied to the business calendar, and observability that makes production itself part of the experiment. If you'd like experienced help designing that, our Performance Testing team works with engineering and product leaders on exactly this kind of readiness.