NIST, PCI and Brazil on social engineering

For a CISO, IT, or compliance team, social engineering is not a dictionary decision but a measurement decision. Across NIST, CIS, PCI, CISA, NCSC, and Brazil's guidance, one uncomfortable point stands out: meeting an annual or periodic training frequency may satisfy compliance, but it does not prove effectiveness. The defensible choice is different: measure observable behavior, exercises, reports, and response capability against risks such as exposed credentials, unauthorized access, fraud, money transfers, or malware installation.

Placed side by side, the sources do not debate whether training is needed. They debate what counts as evidence that the program works. That is where the difference appears that none of the sources in the material makes on its own, between minimum frequency, exercises, metrics, and reporting channels.

What does social engineering really mean for a CISO?

CISA defines social engineering as the attempt to trick someone into revealing information or taking an action that could compromise systems or networks, and NIST adds that the deception can lead to unauthorized access or fraud. BSI adds concrete effects: sharing confidential information, bypassing controls, sending money, or installing malware. Whalemate's reading is that social engineering, in a risk discussion, does not describe an academic concept but an observable class of loss.

That changes the internal conversation. If the risk is credential disclosure, fraud, or malware, then the program cannot be measured only by training attendance. According to the joint guidance from CISA, NSA, FBI, and MS-ISAC, two common phishing tactics are stealing access credentials and delivering malware, which makes any approach focused only on formal awareness insufficient.

Does the annual frequency required by CIS and PCI prove effectiveness?

CIS Controls v8 requires training at onboarding and at least annually, and content must be updated at least once a year or when there are significant changes. PCI DSS 12.6.3 requires training staff at hiring and at least every 12 months, using more than one communication method, and obtaining confirmation that policies and procedures were read and understood. Whalemate's reading is that both frameworks set a compliance baseline, but neither proves risk reduction on its own.

The tension appears when that requirement is compared with empirical evidence. Researchers at UC San Diego reported that they found no correlation between the date of annual training and whether a person clicked links in simulated phishing messages. A large-scale reproduction published on arXiv also found no significant main effects on clicks, with p=0.450, or on reports, with p=0.417. Meeting the schedule, then, is not the same as proving behavior change.

What changes when NIST leaves frequency up to the organization?

NIST SP 800-172 3.2.1e requires training focused on recognizing and responding to social engineering threats, APT actors, breaches, and suspicious behavior, but leaves frequency up to the organization. It also requires updates at an organization-defined frequency or when there are significant threat changes. Whalemate's reading is that NIST pushes a risk-based decision, not a uniform minimum for everyone.

That distinction matters for LATAM teams with different attack surfaces. If an organization receives campaigns by email, SMS, calls, and direct messaging, as CISA warns, frequency should be defined by exposure patterns and observed results, not inherited cadence. In addition, according to NIST SP 800-172A, evaluation of the requirement looks at policies, materials, records, and threat information, which supports broader governance than simple attendance tracking.

Why does the NCSC focus on exercises and reports?

The UK's NCSC says the best way to test phishing readiness is through exercises rather than theory-based training alone. It also recommends measuring defensive successes, not only how many people clicked but also how many reported the message. Whalemate's reading is that this sets a tougher operational standard: measure whether the organization detects and escalates before the incident grows.

That approach corrects a common bias in awareness programs. Looking only at clicks measures individual failure, while adding reports measures distributed defensive capability. The NCSC also recommends exercising response plans before an incident, because detecting an attack without knowing where to escalate it or what to isolate still leaves an operational gap.

What does PCI DSS 12.6.3 leave out if it is treated as a complete program?

PCI DSS 12.6.3, according to the cited material, focuses on training at least every 12 months, using more than one communication method, and confirming that policies and procedures were read and understood. In that material, it does not require click metrics, report metrics, or phishing exercises. Whalemate's reading is that it works for proving documentation discipline, but it leaves the central question open: did behavior change?

That matters for compliance because a control can be formally complete and still weak as real defense. If the program only proves that people received messages and acknowledged understanding, it still does not show whether they recognized a realistic lure, used the right reporting channel, or whether the team responded in time. That gap between formal evidence and operational evidence is the most practical tension in the material.

Does Brazil raise the bar beyond the annual minimum?

Brazil's Digital Government guidance says the program should go beyond annual training with videos and phishing tests, and include frequent, updated messages on relevant topics. In addition, Brazil's ransomware prevention and response guide recommends periodic social engineering simulations, such as monthly phishing tests for all public employees, with realistic scenarios based on the organization's context. Whalemate's reading is that Brazil is describing continuous operations, not a yearly compliance checkpoint.

That position directly challenges the annual minimum logic found in CIS Controls v8 and PCI DSS 12.6.3. An organization can meet the CIS or PCI floor and still fall short of the operational standard Brazil suggests for sustaining memory, detection, and reporting in changing environments. For regional teams, this matters because the local reference model may call for more continuity than the international standard used for audits.

Does training work, or is the evidence still mixed?

The evidence in the material does not support a simple answer. The large-scale reproduction published on arXiv reported that training interventions did not produce significant main effects on clicks or reports, and it listed an overall click rate of 10.4%, with 9.8% in the control group, 10.5% in the training-only group, 10.6% in the training plus exercise group, and 10.4% in the group with training plus exercise. Whalemate's reading is that the mere existence of an intervention does not guarantee impact.

But another comparative study published by Springer reported that after 10 days it saw no significant difference between groups, while after 63 days training did produce significant improvements in click behavior. In that study, 24.5% of the trained group failed, compared with 32.08% in the feedback-only group and 47.5% in the group with neither training nor feedback. The evidence is mixed, and that is why the reasonable decision is not to abandon the program but to validate it with local data.

What metrics should matter when the goal is to reduce risk?

NIST SP 800-50 Rev. 1 describes program metrics as measures to quantify implementation, effectiveness, and efficiency, assess whether effort is sufficient, and identify improvements. The NCSC asks for defensive successes, such as reports, in addition to clicks. DomainTools, in its analysis of Verizon's 2025 DBIR data, said users with more recent training reported phishing at roughly a 21% rate, versus a 5% baseline rate, a relative difference of about four times. Whalemate's reading is that fast reporting may be a more useful indicator than chasing an impossible zero-click target.

That does not mean clicks should be ignored. It means they should be read together with time to report, exercise results, and exposure by role, as the material's findings suggest. If NIST SP 800-50 Rev. 1 also frames the program around behavior change and continuous improvement, then measuring course completion alone leaves out half the picture.

How important is the reporting channel in operational decisions?

CISA recommends defining who suspicious mail or a phishing attempt should be reported to and how, and also quickly reporting phishing incidents. The NCSC recommends helping users identify and report suspicious messages and respond quickly to incidents. Whalemate's reading is that the reporting channel is not an administrative detail; it is part of the control.

If a user recognizes the attempt but does not know where to report it, or if they report it and nobody acts, the program fails even if training was completed. CISA also warns that phishing can arrive by email, SMS, direct messages on social networks, or phone calls. That means reporting and response must be designed for multiple channels, not just corporate email.

What does the threat context show about the cost of measuring badly?

The FBI said its Internet Crime Report 2024 included 859,532 complaints of internet crime and more than $16 billion in reported losses, and that phishing and spoofing were among the top three crimes by complaint volume. The IC3 2024 report listed $16.6 billion in losses, up 33% from 2023. APWG observed 3.8 million phishing attacks during 2025, compared with 3.76 million in 2024, and in the third quarter of 2025 it recorded nearly 35% growth in SMS-based fraud and 57% growth in BEC wire-transfer attacks. Whalemate's reading is that the problem is not limited to email or to a stable vector.

Verizon adds another layer. The Verizon 2025 Data Breach Investigations Report said the human element was involved in about 60% of the breaches analyzed, and that third-party involvement doubled from 15% to 30%. In the DBIR 2025 summary, social actions were led by phishing and pretexting, and prompt bombing appeared in 14% of the incidents analyzed. Measuring badly does not only let clicks slip through; it also leaves organizations blind to fraud, third parties, and new forms of pressure on users.

What does this mean for CISO, IT, and compliance in LATAM tomorrow?

The operational decision should not be whether a yearly course is enough. It should be which observable behavior needs to improve and how that improvement will be tested. According to NIST SP 800-50 Rev. 1, the program should focus on behavior change and effectiveness metrics; according to the NCSC, clicks and reports should both be measured; according to CISA, reporting channels and quick response should be defined; and according to Brazil's guidance, frequent messages and periodic simulations should be maintained.

For a LATAM team, that supports a defensible program with five pieces. First, treat social engineering as a risk involving exposed credentials, unauthorized access, fraud, money transfers, or malware, according to CISA, NIST, and BSI. Second, measure at least clicks, reports, time to report, exercise results, and exposure by role, because the material shows that frequency alone does not prove effectiveness. Third, include simulations or exercises to test real readiness, as NIST, NCSC, and Brazil recommend. Fourth, define and rehearse reporting and response channels for email, SMS, calls, and messaging. Fifth, separate compliance from effectiveness, because CIS Controls v8 and PCI DSS 12.6.3 may be satisfied and still not demonstrate risk reduction.

What to read next?

• Awareness on the platform
• ISO, NIST, SANS, Verizon on phishing training
• NIST and CISA on social engineering