When Does Your Software Need a Penetration Test?
Software penetration testing should be scheduled before meaningful exposure and after changes that affect who can access the application, what sensitive data it holds or which systems it trusts. Book the test early enough to investigate findings, fix important weaknesses and have those fixes retested before the relevant launch. Use a recurring review alongside these triggers, with its timing based on your product's risk and any applicable obligations.
What is a penetration test, and how is it different from a scan?
A penetration test is an authorised assessment in which testers investigate whether weaknesses can be exploited within agreed boundaries. An automated vulnerability scan checks targets for detectable weaknesses; a penetration test adds human investigation, including attempts to connect weaknesses and examine application-specific behaviour.
When buying a test, ask what the engagement includes. Do not assume that a service labelled “penetration testing” includes manual examination of account permissions, business workflows or interactions between features.
For example, ask a tester to examine whether a customer can access another customer's records, rather than requesting only a list of outdated components. Both questions matter, but they need different investigation.
Treat scanning, code review and penetration testing as separate parts of your quality plan. For the broader picture, see software testing and quality assurance.
Which changes should trigger a test?
Use a decision table during release planning. Focus on changes to access, sensitive information and the boundaries between trusted and untrusted users or systems.
| Change or milestone | What to include in the assessment |
|---|---|
| First public launch | Public routes, account creation, recovery and protected functions |
| New customer roles or shared workspaces | Permissions within and between customer accounts |
| Payments or sensitive records | Access to records, changes to transactions and relevant integrations |
| New API or external integration | Authentication, permissions and handling of incoming data |
| Major hosting or architecture change | Changed entry points and access controls |
| Significant security fix | The original weakness and related paths that might remain exposed |
A wording change or colour adjustment does not automatically justify a new full test. A change to how users are identified or authorised deserves closer review, even if the screen looks identical.
After a suspected compromise, handle containment and investigation first. Commission further testing against a defined recovery question, rather than treating a routine penetration test as the incident response plan.
How early should testing happen?
Agree the scope while planning the release. Schedule the main assessment when the relevant workflows are complete and the test environment is stable enough to represent the intended release.
Avoid making the report delivery date the launch date. Instead, work backwards through four checkpoints:
- The tester delivers findings with enough evidence to investigate them.
- Your development partner implements and verifies fixes.
- The tester retests the affected paths.
- The product owner decides whether any remaining risk is acceptable.
Agree how urgent findings will be reported during the engagement, so the team can begin investigating before the final report arrives.
For a product already in use, review the testing schedule after meaningful changes. If you also set a recurring interval, document why it suits the product and revisit that reasoning as exposure changes. Do not present a particular interval as a universal requirement.
What should the scope include?
Define scope in terms of user journeys and access boundaries, alongside the technical targets. “Test the website” leaves too much room for different interpretations.
Write down the applications, APIs and interfaces included. Identify the roles to examine, the sensitive actions they can perform and the separation expected between customers.
For example, scope might include registration, password recovery, invitations, file uploads, record exports and administrator actions. Include mobile or background-service paths where they support those workflows.
Also document exclusions. If an external provider is outside your authority, agree how the integration can be assessed without testing that provider directly. Obtain written authorisation for every target and permitted activity.
Set rules for disruptive actions, data handling, testing hours, emergency contacts and stopping the test. Make clear who can approve a scope change.
For related planning questions, read data security in modern applications.
What access should testers receive?
Choose access according to the questions the test must answer. A test beginning with no account asks a different question from one examining what a signed-in customer or administrator can do.
Request the access needed to cover the agreed roles. That may include separate customer accounts, accounts belonging to different organisations, administrator access and documentation for relevant interfaces.
Provide a way to exercise invitations, account recovery and other flows that require messages or state changes. Confirm that test accounts have realistic permissions, rather than special restrictions that hide normal behaviour.
Ask the proposal to explain what each access level enables and what remains untested. If you want both an external-attacker assessment and a detailed permissions review, name both objectives.
Give credentials through an agreed secure channel and arrange their removal or rotation after testing.
Should the test run in a test environment or production?
Prefer a representative test environment for activities that could change records, interrupt service or create unwanted transactions. Use synthetic data and safe substitutes for actions such as sending messages or taking payments.
Before testing starts, compare that environment with the intended production release. Record differences in authentication, permissions, configuration and connected services.
A result applies to the targets, conditions and scope assessed; it does not establish that an untested environment has the same security properties.
Where production-specific checks are necessary, agree them separately. Define permitted actions, monitoring, stop conditions and recovery arrangements. Confirm whether connected service providers require additional permission.
Keep a record of the tested release so later changes do not silently invalidate the comparison.
What should the deliverables contain?
Ask for a report that supports decisions and fixes, with a retest included in the agreement.
The report should identify:
- Coverage: targets, roles, workflows, exclusions and testing limitations.
- Evidence: enough detail for the development team to reproduce each finding.
- Impact: what an attacker could do under the demonstrated conditions.
- Priority: the reasoning behind urgency, including required access and affected data.
- Remediation: practical guidance that addresses the cause.
- Retest status: which fixes were checked and which findings remain unresolved.
Request a plain-language summary for the product owner alongside the technical detail. Restrict access to the detailed report and agree how test data and evidence will be retained or deleted.
What common mistakes should you avoid?
Buying a certificate instead of defining coverage. Ask what was tested, with which access, and what was excluded.
Testing too late. Reserve development capacity and a retest window before committing to launch.
Providing only one account. Where customer separation matters, include accounts that let testers examine that boundary.
Changing the target throughout the assessment. Agree how releases during testing will be handled and which version the report covers.
Closing findings after a code change alone. Require evidence that the affected behaviour was checked again.
Treating the report as a permanent guarantee. Use it as evidence about the assessment performed, and review later changes against its scope.
What are the frequently asked questions?
Does an early product need penetration testing?
Base the decision on exposure and potential harm. Before opening access or collecting sensitive information, review whether independent testing is needed and which workflows deserve priority.
Must every release receive a full test?
Use a change review to decide. A small presentation change may need ordinary quality checks, while a change to permissions may justify focused security testing and broader regression checks.
Can our development partner test its own work?
Include the partner's checks, but ask who provides independent challenge, how coverage is documented and who verifies fixes. Define those responsibilities before commissioning the assessment.
What if findings remain before launch?
Make an explicit release decision. Record the affected behaviour, potential harm, available restrictions, accountable owner and resolution plan. Do not let a launch deadline serve as acceptance of unresolved risk.