Written by Jaiz Anuar

Security Products Should Be Tested Against The Outcome, Not The Alert

Detection is evidence, but protection is an outcome. The strongest security testing follows the complete attack chain to determine whether controls actually change what the attacker can achieve.

For years, cybersecurity products have been measured by what they can detect.

How many attack techniques can the product identify? How much telemetry does it generate? How many MITRE ATT&CK techniques does it map? How quickly does it raise an alert?

Those questions are useful, but they do not tell the whole story.

A product may detect malicious activity and still allow the attacker to continue moving through the environment. It may generate an alert while privilege escalation continues. It may identify suspicious behaviour while the attacker still reaches a critical system.

That creates a more important question:

Did the security control actually change the outcome of the attack?

Detection Is Evidence. Protection Is Outcome.

The recent launch of SE Labs’ PIVOT testing programme is interesting because it takes a broader view of security testing. Rather than stopping at whether a product detected malicious activity, PIVOT follows the attack chain and looks at how far the attacker was able to progress, what defenders could see, and whether the security product actually interrupted the attack before meaningful harm occurred. Infosecurity Magazine covered the programme here.

That is a useful shift in thinking.

Detection is important. But detection alone is not the final objective.

The real objective is reducing impact.

If the attacker is detected at the first stage but still steals credentials, escalates privileges and moves laterally, the organisation has visibility.

It may not yet have protection.

The Alert Is Only The Beginning

Security teams often celebrate detection metrics.

An alert was generated within thirty seconds.

The EDR identified the behaviour.

The SIEM correlated the events.

The NDR saw the unusual connection.

All of these are positive signals.

But the next question should always be:

What happened after the alert?

Did the endpoint get isolated?

Was the credential revoked?

Was the session terminated?

Was lateral movement blocked?

Did the attacker reach the crown jewels before containment happened?

A good security product should help the organisation move from detection to action.

If the product produces excellent telemetry but the attacker can still complete the objective, the architecture still has a problem.

Security Controls Should Be Tested As Part Of An Attack Chain

Real attackers do not attack one control at a time.

They move through environments.

Initial access leads to credential access. Credential access leads to privilege escalation. Privilege escalation leads to lateral movement. Lateral movement eventually reaches the target.

That is why testing individual controls in isolation can create false confidence.

An EDR may perform well.

A firewall may perform well.

An identity platform may perform well.

But what happens when those controls are tested together against a complete attack path?

The weakness may exist between them.

The attacker may exploit the gap between endpoint detection and identity response, or between network segmentation and privileged access.

Architecture lives in those gaps.

Buyers Need To Understand What “Good” Actually Means

Cybersecurity products are difficult to compare.

Marketing material naturally highlights strengths. Vendor reports often emphasise detection coverage, threat intelligence and technical capability.

Independent testing helps.

But even independent testing needs to answer questions that matter to the organisation.

A product detecting 98% of techniques may sound better than one detecting 95%.

But what if the second product stopped the attack earlier?

What if it gave analysts enough context to contain the incident faster?

What if the first product detected more individual techniques but still allowed the attacker to reach a privileged account?

Percentages are useful.

Outcomes are more useful.

Architecture Should Measure How Far The Attacker Gets

From a security architecture perspective, one of the strongest measures of defensive effectiveness is simple:

How far can the attacker move after the first control fails?

If one compromised endpoint immediately leads to domain-level access, the architecture is weak regardless of how quickly the first alert appeared.

If the attacker must cross multiple trust boundaries, obtain additional identities, overcome segmentation and trigger multiple independent controls, the architecture is doing something valuable.

The objective is not necessarily to prevent every initial compromise.

That is unrealistic.

The objective is to prevent one compromise from becoming a business-impacting incident.

This is where security products should support architecture rather than become substitutes for it.

The SOC Experience Matters Too

There is another aspect that product testing sometimes overlooks.

What does the analyst actually see?

A security product may detect an attack accurately but provide so little context that the SOC spends another thirty minutes trying to understand what happened.

That delay matters.

An alert should help answer:

What happened? Which identity was involved? Which system was affected? What did the attacker attempt next? What should be contained first?

Visibility without context can become another form of noise.

The effectiveness of a security product therefore depends not only on what it detects, but also on whether the organisation can understand and act on that detection quickly enough.

Final Thoughts

Security testing should continue to measure detection.

But detection should not become the destination.

The more important measure is whether the control changes the attack outcome.

Did it slow the attacker?

Did it prevent privilege escalation?

Did it block lateral movement?

Did it protect the critical asset?

Did it give defenders enough information to contain the incident before business impact occurred?

A security product can generate the perfect alert and still lose the attack.

That is why the next generation of security testing should increasingly focus on what happens after detection.

Because organisations do not buy cybersecurity products to generate alerts.

They buy them to prevent attacks from becoming incidents.

Security products should be tested against the outcome, not the alert.

Question assumptions. Share knowledge. Build trust.