19 October 2026, 07:37 PM
[attachment=9703]
"We Passed the Pen Test" and Other Sentences That Make Me Nervous in Banking QA
I've heard that sentence in a lot of kickoff calls. Usually it's said with relief, sometimes with pride, and almost always about a report that's four months old.
The pen test was real. The findings were fixed. But since then the team shipped eleven releases, swapped an identity provider, added a mobile endpoint, and moved a service behind a new gateway. The report describes a system that no longer exists.
Banking software is where this gap hurts most, because the three things you're asked to prove are tangled together. Security, performance and reliability aren't three checklists. They pull against each other, and most real incidents happen in the tension between them.
So let's take the common beliefs one at a time.
Myth 1: "Security Testing Is a Gate You Pass"
Reality: Security is a property you keep re-proving.
A pen test is a snapshot. It's valuable, and you should still do it, but it can't tell you whether the release you shipped on Tuesday broke an authorization rule. For that you need security checks inside the delivery pipeline: static and dependency scanning on every build, dynamic scanning against a deployed test environment, and a small set of security regression tests that run every time.
What belongs in that regression set for a bank? In my experience, these earn their place:
One thing I'd add for newer products: if the bank has an AI assistant, treat it as an attack surface. Test for prompt injection that tries to pull another customer's data, and for responses that leak internal instructions or policy text. The same authorization rules apply to a chatbot as to an API, and they're easier to forget.
When this isn't enough: automated tests find known patterns. Business logic abuse, like exploiting the order of approvals or chaining two harmless features, still needs a human thinking like an attacker. Schedule that, and run it after major architecture changes, not only on an annual calendar.
Myth 2: "Performance Testing Means Running a Load Test Before Launch"
Reality: Banks don't have average days. They have payday, month end, tax deadlines, festival shopping and the morning a rumor starts about a rival bank.
A single load test at expected volume is better than nothing, but it answers a narrow question. The questions that actually protect you are different.
What does the system do at the knee? Every system has a point where response times stop growing gently and start climbing steeply. Find it deliberately with a stress test, and know how far it is from your busiest realistic day.
What does a spike look like, versus a ramp? A thousand users arriving over ten minutes is a different test from a thousand arriving in ten seconds. Connection pools, caches and autoscaling all react differently.
What happens over time? Soak tests, running for hours at steady load, catch memory leaks, connection exhaustion and slow log growth. These are the failures that show up on day three, not hour one.
What does the batch window do to online traffic? Many banks still run end-of-day jobs, statement generation or reconciliation on the same databases that serve customers. Test them together. I've seen systems that handled both beautifully in isolation and fell over when they overlapped.
Which percentile are you measuring? Averages hide pain. A p50 of 300 milliseconds with a p99 of nine seconds means some customers are staring at a spinner while their payment is in doubt. Set targets on p95 and p99 for the journeys that matter: login, balance, transfer, bill pay.
A quick note on honesty here. Performance results are only as good as the environment. If your test environment has a tenth of production's data volume or a mocked core system that answers instantly, say so in the report. Numbers from a friendly environment are directional, not a guarantee.
Myth 3: "Reliability Means High Uptime"
Reality: Uptime tells you the front door was open. It doesn't tell you the money went where it should.
A banking system can be up and wrong. It can accept a transfer, lose the confirmation, and leave the customer unsure whether to try again. For banks, reliability has two halves: staying available, and staying correct while things fail.
Availability and recovery. Know your recovery targets, the maximum time you can be down and the maximum data you can afford to lose, and then prove them. A disaster recovery plan that has never been executed is a document, not a capability. Run failover drills, time them, and write down what surprised you.
Correctness under failure. Kill a service mid-transaction. Drop the network between the payments service and the core. Return a slow, malformed or empty response from a partner. Then check the state of the money. Was the debit reversed? Was the retry safe? This is where idempotency matters: repeating a request must not repeat its effect.
Graceful degradation. When a dependency is down, what does the customer see? A clear message and a safe fallback is a good outcome. A hung screen followed by a double submission is how duplicate transfers happen.
Observability. You can't recover from what you can't see. Test that alerts fire, that dashboards show the failing component, and that a support agent can trace one customer's transaction across systems. If tracing a payment takes an hour of manual searching, your reliability has a hole in it, whatever the uptime says.
Where the Three Collide
Here's the part that single-topic articles skip. These properties trade against each other, and the failures hide in the trade.
[attachment=9702]
That second row deserves emphasis. A retry policy that looks sensible in a design review can turn a ten-minute partner slowdown into an hour-long outage, because every client piles in at once. Test your timeouts and backoff, don't just configure them.
A Simple Way to Decide Where to Spend Effort
You'll never have time to test everything at full depth, so rank by consequence.
Mistakes I'd Warn You About
Testing with security turned off. The numbers look wonderful and describe nothing real.
Treating the sandbox as production. Partner and core-system test environments are usually faster and more forgiving than live ones.
Never testing the unhappy recovery. Teams drill failover once, early, and never again as the architecture changes.
One owner per silo. Security, performance and reliability handled by three separate teams who first meet during an incident.
Reporting averages. Leadership sees a comfortable number while the tail latency quietly fails customers.
Where AI Fits
Used carefully, AI helps at the edges: anomaly detection on production metrics, clustering similar failures from logs, generating realistic traffic patterns, and prioritizing which regression tests to run after a change. It shouldn't be the judge of whether a transfer is safe or a limit is enforced. Those checks stay deterministic and explicit. And any AI inside the product, such as fraud scoring or a customer assistant, needs its own testing for drift, bias, leakage and guardrails.
Closing Thought
Ask your team one question this week: if a payment partner slowed down by a factor of ten right now, during your busiest hour, what would happen to the money in flight and what would the customer see? If the honest answer is "we think it would be fine," that's the test to write.
The strongest banking systems I've seen aren't the ones that never fail. They're the ones whose failures are small, visible and reversible, because someone made a habit of breaking them on purpose first. For teams that want that habit built into delivery, our FinTech Software Testing Services practice works across security, performance and resilience together, so the trade-offs get tested before customers find them.
"We Passed the Pen Test" and Other Sentences That Make Me Nervous in Banking QA
I've heard that sentence in a lot of kickoff calls. Usually it's said with relief, sometimes with pride, and almost always about a report that's four months old.
The pen test was real. The findings were fixed. But since then the team shipped eleven releases, swapped an identity provider, added a mobile endpoint, and moved a service behind a new gateway. The report describes a system that no longer exists.
Banking software is where this gap hurts most, because the three things you're asked to prove are tangled together. Security, performance and reliability aren't three checklists. They pull against each other, and most real incidents happen in the tension between them.
So let's take the common beliefs one at a time.
Myth 1: "Security Testing Is a Gate You Pass"
Reality: Security is a property you keep re-proving.
A pen test is a snapshot. It's valuable, and you should still do it, but it can't tell you whether the release you shipped on Tuesday broke an authorization rule. For that you need security checks inside the delivery pipeline: static and dependency scanning on every build, dynamic scanning against a deployed test environment, and a small set of security regression tests that run every time.
What belongs in that regression set for a bank? In my experience, these earn their place:
- Object-level authorization. Log in as customer A, then request customer B's account, statement or beneficiary list by changing the ID. This is one of the most common serious API flaws, and automated scanners often miss it because they don't understand who should own what.
- Function-level authorization. Can a teller role call an endpoint meant for a branch manager? Can a customer call an internal admin route if they know the URL?
- Authentication edges. Expired tokens, reused tokens, tokens from a different environment, session behavior after a password reset, and step-up verification that can be skipped by calling a later step directly.
- Abuse resistance. Credential stuffing, enumeration of valid usernames through different error messages, and brute-force protection that quietly resets when the client changes an IP header.
- Sensitive data exposure. Account numbers and personal data in logs, URLs, error messages, caches and analytics events.
One thing I'd add for newer products: if the bank has an AI assistant, treat it as an attack surface. Test for prompt injection that tries to pull another customer's data, and for responses that leak internal instructions or policy text. The same authorization rules apply to a chatbot as to an API, and they're easier to forget.
When this isn't enough: automated tests find known patterns. Business logic abuse, like exploiting the order of approvals or chaining two harmless features, still needs a human thinking like an attacker. Schedule that, and run it after major architecture changes, not only on an annual calendar.
Myth 2: "Performance Testing Means Running a Load Test Before Launch"
Reality: Banks don't have average days. They have payday, month end, tax deadlines, festival shopping and the morning a rumor starts about a rival bank.
A single load test at expected volume is better than nothing, but it answers a narrow question. The questions that actually protect you are different.
What does the system do at the knee? Every system has a point where response times stop growing gently and start climbing steeply. Find it deliberately with a stress test, and know how far it is from your busiest realistic day.
What does a spike look like, versus a ramp? A thousand users arriving over ten minutes is a different test from a thousand arriving in ten seconds. Connection pools, caches and autoscaling all react differently.
What happens over time? Soak tests, running for hours at steady load, catch memory leaks, connection exhaustion and slow log growth. These are the failures that show up on day three, not hour one.
What does the batch window do to online traffic? Many banks still run end-of-day jobs, statement generation or reconciliation on the same databases that serve customers. Test them together. I've seen systems that handled both beautifully in isolation and fell over when they overlapped.
Which percentile are you measuring? Averages hide pain. A p50 of 300 milliseconds with a p99 of nine seconds means some customers are staring at a spinner while their payment is in doubt. Set targets on p95 and p99 for the journeys that matter: login, balance, transfer, bill pay.
A quick note on honesty here. Performance results are only as good as the environment. If your test environment has a tenth of production's data volume or a mocked core system that answers instantly, say so in the report. Numbers from a friendly environment are directional, not a guarantee.
Myth 3: "Reliability Means High Uptime"
Reality: Uptime tells you the front door was open. It doesn't tell you the money went where it should.
A banking system can be up and wrong. It can accept a transfer, lose the confirmation, and leave the customer unsure whether to try again. For banks, reliability has two halves: staying available, and staying correct while things fail.
Availability and recovery. Know your recovery targets, the maximum time you can be down and the maximum data you can afford to lose, and then prove them. A disaster recovery plan that has never been executed is a document, not a capability. Run failover drills, time them, and write down what surprised you.
Correctness under failure. Kill a service mid-transaction. Drop the network between the payments service and the core. Return a slow, malformed or empty response from a partner. Then check the state of the money. Was the debit reversed? Was the retry safe? This is where idempotency matters: repeating a request must not repeat its effect.
Graceful degradation. When a dependency is down, what does the customer see? A clear message and a safe fallback is a good outcome. A hung screen followed by a double submission is how duplicate transfers happen.
Observability. You can't recover from what you can't see. Test that alerts fire, that dashboards show the failing component, and that a support agent can trace one customer's transaction across systems. If tracing a payment takes an hour of manual searching, your reliability has a hole in it, whatever the uptime says.
Where the Three Collide
Here's the part that single-topic articles skip. These properties trade against each other, and the failures hide in the trade.
[attachment=9702]
That second row deserves emphasis. A retry policy that looks sensible in a design review can turn a ten-minute partner slowdown into an hour-long outage, because every client piles in at once. Test your timeouts and backoff, don't just configure them.
A Simple Way to Decide Where to Spend Effort
You'll never have time to test everything at full depth, so rank by consequence.
- List your customer-facing journeys and your internal critical paths.
- For each, ask three questions: what's the worst thing that happens if it's breached, if it's slow, if it's wrong?
- Put your deepest testing on the journeys where the answer to any of those is severe, which in banking usually means money movement, authentication and anything touching regulated customer data.
- For lower-risk journeys, a lighter automated baseline is a reasonable tradeoff. Say that out loud so it's a decision, not an accident.
Mistakes I'd Warn You About
Testing with security turned off. The numbers look wonderful and describe nothing real.
Treating the sandbox as production. Partner and core-system test environments are usually faster and more forgiving than live ones.
Never testing the unhappy recovery. Teams drill failover once, early, and never again as the architecture changes.
One owner per silo. Security, performance and reliability handled by three separate teams who first meet during an incident.
Reporting averages. Leadership sees a comfortable number while the tail latency quietly fails customers.
Where AI Fits
Used carefully, AI helps at the edges: anomaly detection on production metrics, clustering similar failures from logs, generating realistic traffic patterns, and prioritizing which regression tests to run after a change. It shouldn't be the judge of whether a transfer is safe or a limit is enforced. Those checks stay deterministic and explicit. And any AI inside the product, such as fraud scoring or a customer assistant, needs its own testing for drift, bias, leakage and guardrails.
Closing Thought
Ask your team one question this week: if a payment partner slowed down by a factor of ten right now, during your busiest hour, what would happen to the money in flight and what would the customer see? If the honest answer is "we think it would be fine," that's the test to write.
The strongest banking systems I've seen aren't the ones that never fail. They're the ones whose failures are small, visible and reversible, because someone made a habit of breaking them on purpose first. For teams that want that habit built into delivery, our FinTech Software Testing Services practice works across security, performance and resilience together, so the trade-offs get tested before customers find them.