Vulnerability scan, pen test, red team or continuous testing: which do you need?
Security testing is sold under many names, and buying the wrong one is common. A plain guide to what each type of test does, what it costs you, and which one your auditors and customers actually expect.
By Cyber Ally ·
A customer asks for your latest penetration test. Your auditor wants evidence of independent security testing. You search for a provider and find vulnerability scans, penetration tests, red teams, “pen testing as a service” and bug bounties, at prices from a few hundred dollars to six figures.
They are not interchangeable. Here is what each one is, and how to choose.
Vulnerability scanning
What it is. An automated tool checks your systems against a database of known weaknesses: missing patches, outdated software, exposed services and common misconfigurations.
What it is good for. Catching known issues quickly and often. Scans should run continuously or at least monthly as part of normal operations.
What it is not. A scan cannot understand your application. It will not notice that one customer can view another customer’s invoices by changing a number in the URL, which is exactly the kind of flaw that causes real breaches. A scan report sold as a “penetration test” is one of the most common ways companies end up with evidence their auditor or customer will not accept.
Penetration testing
What it is. An experienced tester attacks your systems the way a real attacker would, using tooling to cover ground and manual work to find what tools miss: broken access control, business logic flaws, and chains of small issues that add up to a serious one.
What it is good for. Finding what is genuinely exploitable, in a defined scope, within a fixed time. It is what SOC 2 auditors, ISO 27001 certification bodies and enterprise customers generally mean when they ask for independent security testing.
Common scopes:
- Web application and API. Usually the most important test for a SaaS company, because the application is where your customers’ data lives.
- External network. Everything you expose to the internet.
- Internal network. What an attacker could reach after getting a foothold inside.
- Cloud configuration. Identity and access settings, storage and logging in AWS, Azure or Google Cloud.
- Mobile application. The app on the device and the APIs behind it.
- AI and LLM applications. Prompt injection, data leakage, and agents that can be manipulated into taking actions they should not. This is new territory for many testers, so ask specifically.
Black, grey or white box. Black box testing starts with no inside knowledge. Grey box gives the tester user accounts and some documentation. White box adds code or configuration access. For most SaaS products, grey box is the best balance: it reflects a realistic attacker, such as a malicious customer, without spending budget on reconnaissance.
Red teaming
What it is. A goal-based simulation of a real adversary over weeks, often combining technical attacks, phishing and physical access, designed to test whether your people and processes detect and respond.
What it is good for. Mature organisations with a security operations team that want to test detection and response, not just find vulnerabilities.
What it is not. A substitute for a penetration test, or a sensible first step. If you do not yet have a team watching for attacks, a red team mostly confirms what you already know. Most growing companies get far more value from a well-scoped penetration test.
Continuous testing and bug bounties
Continuous testing, sometimes sold as pen testing as a service, spreads testing across the year, often combining automated scanning with periodic manual testing. It suits teams that release frequently and want issues found close to when they are introduced.
Bug bounties pay outside researchers for issues they find. They can surface creative findings, but coverage is unpredictable, and they work best once the basics are already solid. They do not usually satisfy an auditor on their own.
So which do you need?
- Preparing for SOC 2 or ISO 27001, or a customer has asked for a pen test: a manual penetration test of your application and APIs, plus your external network, at least once a year and after major changes.
- Running a cloud-native product: add a cloud configuration review.
- Shipping AI features or agents: add AI and LLM application testing.
- Every company: run vulnerability scanning continuously in the background, as hygiene.
- A mature security team testing its detection: that is when a red team makes sense.
Whatever you buy, check that it includes manual testing, a report your engineers can act on, and a retest once you have fixed the findings.
How we help
We provide penetration testing for web applications, APIs, external and internal networks, cloud, mobile and AI applications. We do not offer red teaming, and for most of the companies we work with we would not recommend it yet. See our penetration testing service, or talk to us and we will help you scope the right test.