NIST and CISA on context-based phishing tests

For CISOs, IT, and compliance teams in LATAM, a phishing test becomes useful when it stops being a single aggregated click rate and becomes an internal, comparable, segmented, and auditable test. According to NIST, CISA, and the Brazilian cases in the source material, the real value comes from combining role or group, interaction, clicks, data submission, reporting, repeat offenders, lure difficulty, and response speed to decide whether to strengthen training, adjust instruction, or make a technical change.

What the sources show together

None of the sources in the material makes this full comparison on its own. Read side by side, they show a clear progression: ISO/IEC 27001:2022 A.6.3 sets the general requirement for security awareness based on job role, NIST adds segmentation and difficulty context, CISA turns that into reproducible metrics, and the Brazilian cases show which behaviors appear when you measure more than clicks.

What does a useful phishing test actually measure?

According to NIST SP 800-50r1, the exercise should check whether users detect the attempt or are induced to click a malicious site or open an infected attachment. According to CISA, phishing vulnerability scanning evaluates levels of vulnerability across defined groups and can show how many users opened the message and how many clicked to estimate potential impact.

Whalemate's reading is that a useful test does not measure susceptibility alone. It measures observable behavior to decide something concrete: who needs reinforcement, where the reporting channel needs adjustment, and which part of the awareness control can be defended with evidence.

Why is a standalone click rate no longer enough?

NIST warns in its 2023 awareness blog that clicks alone do not provide a complete view of employee behavior, even though organizations use them to measure how many people click or report. In tension with that, a Centro Paula Souza monograph used click rate as a behavioral risk indicator in monthly campaigns run between 2020 and 2024 at a power sector company.

Whalemate's reading is that clicks are still useful, but only as a partial signal. If they become the only indicator, they lose explanatory power and can lead to the wrong response, because they do not distinguish between someone who opened the email, someone who clicked, someone who submitted data, someone who reported it, and someone who repeated the behavior.

What changes when you segment by role or group?

NIST SP 800-50r1 recommends first determining current click rates by specific group or job role to decide whether more targeted training is needed. ISO/IEC 27001:2022 A.6.3, according to SIROS ID Compliance and Securitm, requires awareness, education, training, and regular updates that are relevant to the job role.

Whalemate's reading is that segmentation is not optional refinement. It is the point where a simulation stops being an aggregate number and becomes a decision tool, because it links the test result to specific populations and to evidence requirements by role or function.

How does lure difficulty affect interpretation?

The NIST Phish Scale, according to NIST, exists to contextualize click rates by assessing the human difficulty of detecting the email. A study published on arXiv found that click rates rose from 7.0% for easy lures to 15.0% for difficult lures based on difficulty measured by the NIST Phish Scale, with a statistically significant effect reported as F(2, 12086) = 41.415, p < 0.001, η² = 0.007, according to the same study.

Whalemate's reading is straightforward: two campaigns with the same click rate do not mean the same thing if the lure difficulty was different. Reporting clicks without qualifying difficulty mixes incomparable populations, messages, and contexts.

What does it reveal to measure views, clicks, data submission, and reporting?

The campaign by Brazil's Ministério da Justiça e Segurança Pública showed 2,307 participating users, of whom 1,144 viewed the email, 401 clicked, 131 submitted their data, and 53 reported it, according to the official meeting record. In that same campaign, according to the same agency, 50.41% did not view the email, 49.6% viewed it, and among those who viewed it, 35% clicked and 11.4% submitted their data.

Whalemate's reading is that these layers change the decision. A group that opens but does not interact is not the same as one that clicks but does not submit data, and neither is the same as a group that also fails to report. Each behavior enables a different response in training, control design, and escalation channels.

Is one simple metric enough for audit purposes?

CISA's FISMA assessment guide requires monitoring and analyzing qualitative and quantitative measures of the effectiveness of awareness policies, procedures, and practices, and it also requires that the data be obtained in an exact, consistent, and reproducible way. In tension with that, FUNSERV Sorocaba presents the simulation result as the interaction rate with the simulated email, including clicks or responses.

Whalemate's reading is that a simple metric can work as a local operational dashboard, but it falls short as strong evidence if the organization needs to defend methodological consistency, traceability, and comparison over time. CISA's bar pushes organizations to design a measurement framework, not just a campaign.

Does ISO require phishing simulations?

ISO/IEC 27001:2022 Annex A 6.3 establishes, according to SIROS ID Compliance and Securitm, that personnel and relevant interested parties must receive appropriate awareness, education, and training, with regular updates, based on job role. Qalea clarifies that the control does not literally prescribe phishing simulations.

In tension with that, CISA describes support for anti-phishing programs as a combination of awareness and training, simulated attacks, and results analysis to change training and mitigate risk. Whalemate's reading is that ISO requires the control outcome, while CISA describes a mature way to demonstrate it. Simulation can be strong evidence, but it does not replace the continuous program required by the control.

How much weight does reporting carry versus clicking?

CISA's FISMA metric 2.19.2 records how many users identified and reported the exercise to the appropriate cybersecurity resource. The ICO also recommends training people to report suspicious attempts and says a reporting culture helps review existing technical measures and improve employee awareness.

Verizon reported in awareness exercise data contributed by partners during 2023 that 20% of users reported the simulated phishing and that 11% of those who clicked also reported it. Whalemate's reading is that reporting does not compete with clicking as a metric, it corrects it. It separates the person who made a mistake but triggered a defense from the person who interacted and left the organization with no early warning.

What problem does response speed add?

The Verizon 2024 DBIR reported a median of 21 seconds from opening the malicious email to clicking and another 28 seconds before entering data, with a total median under 60 seconds to fall for phishing, according to its finance snapshot. The University of Delaware says that, in a study cited in its material, users who clicked spent between 0 and 19 seconds reviewing the simulation landing page.

Whalemate's reading is that reaction time makes any program that looks only at the final result weeks later insufficient. If compromise can be completed in less than a minute, the organization also needs to measure reporting speed and the response capacity of the official channel, not just historical susceptibility.

How do you keep a simulation from becoming a ritual with no effect?

The ICO warns that phishing exercises that are too frequent can create complacency or disinterest among staff and recommends considering other combined measures to raise awareness. CISA, for its part, proposes a mix of awareness and training, simulated attacks, and results analysis to change training and mitigate risk.

Whalemate's reading is that an isolated test loses value when it turns into repetition without learning. The simulation only matters if it feeds concrete interventions, whether that means tailored training, group-level reinforcement, improvement of the reporting channel, or technical tuning.

What decision does this enable tomorrow for CISOs, IT, and compliance in LATAM?

First, stop reporting only aggregated clicks. According to NIST, comparison by specific group or job role is needed, and according to the NIST Phish Scale, lure difficulty has to be put in context. Second, include at minimum interaction, click, data submission, reporting, and repeat behavior, because NIST mentions reporting rates and repeat clickers, and the Ministry of Justice and Public Security of Brazil case shows how much the interpretation changes when views, data submission, and reporting are added.

Third, design reproducible evidence. According to CISA's FISMA guide, data must be exact, consistent, and reproducible, and the assessor must obtain evidence and results from phishing exercises or other methods used to measure effectiveness. Fourth, use the result to decide focused reinforcement for groups that interact or do not report, adjust the official reporting channel, and read the results in light of lure difficulty and response time. For compliance, this also aligns better with ISO/IEC 27001:2022 A.6.3, which requires awareness relevant to the role and regular updates, and with PCI DSS 12.6, which in this material is presented as a requirement for a security awareness program, not a one-off course.

What to read next?

• Awareness on the platform
• LATAM Awareness Platforms: What to Evaluate
• NIST and CISA on social engineering