Cybersecurity governance has long relied on a single, comfortable premise: predictability.
When developing a traditional software system, we typically develop a Blueprint. We write code, and we provide explicit instructions to the machine. The machine complies with those instructions. For example, if we lock a door, it remains locked until the correct key is inserted. Since both the system and the controls we put in place are predictable, we are aware of which components of the system require protection, and how the system should operate.
However, with the advent of Artificial Intelligence (AI) into the corporate environment, this blue print is eroding. Ai learns from patterns rather than merely adheres to rules. As such, all aspects related to our ability to safeguard organizations are impacted.
The “static Castle” vs. The “living garden”
One method to illustrate why current controls are insufficient due to ai’s dynamic nature is to compare a Castle and a garden.
Traditionally, cybersecurity is similar to a Castle. Thick walls (firewalls), gates with guards (identity management), and inventory of each brick in the Castle walls. The Castle is static. When bricks fall from the Castle walls, repairs occur.
Ai represents a living garden. Seeds (data) are planted. However, there is no means to forecast how every leaf will sprout. Weather conditions, soil composition, and seasonal changes alter the growth of the garden. Therefore, the garden exists in a probabilistic state (likelihoods).
In a Castle, a “risk” is a broken lock. In a garden, a “risk” is an unplanned weed that may choke other plants in the garden ecosystem. Simply erecting a fence around a garden does not provide adequate security; managing the development of the garden is required.
Why predictable controls fail probabilistic systems
A primary issue arises when traditional governance assumes that all risks can be identified during design timem i.e., when designing the system. Many ai-based risks emerge after the system interacts with actual data.
1. The moving goalpost (data adaptation)
Unlike traditional software that does not evolve unless a human modifies the underlying code, ai evolves by adapting based upon input data. When the data consumed by ai varies, so too does its behavior.
The risk: a control implemented on Monday to prevent malicious activity may not be sufficient on Friday since the ai’s internal logic evolved due to exposure to different types of data.
2. The logic gap:
Governance traditionally employs “if/then” logic e.g., if a user incorrectly enters their password three consecutive times, then lock the account. Ai uses “most likely.” It evaluates multiple variables using complex patterns that humans cannot always identify or comprehend.
The risk: due to the fact that ai bases decisions on evolving patterns and behaviors that are difficult to define or quantify, writing a simple rule to restrict such an action (a biased/hallucinated decision) may not effectively address creating a potential security vulnerability.
3. The Visibility Problem:
Humans typically rely on system logs to trace events and actions that occurred. However, logs are not available when dealing with ai-driven applications. Rather than knowing what entered an application and what emerged from an application, the reasons behind specific actions taken by an application are often lost in millions of weighted mathematical formulas. Thus, we cannot regulate or govern something we cannot describe or explain.
Next evolution: AI Governance = cybergov
There is a natural tendency to establish an entirely new “AI Governance” department located outside of our existing cybersecurity teams. This would represent a significant error.
Rather than establishing an entirely new organization, we need to advance our current approach to cyber risk governance. We don’t need additional silos; we simply need enhanced versions of what currently exist.
How governance must advance:
- From “Point-in-time” to Continuous: annual security audits are now inadequate. We need “guard rails” that continuously monitor and evaluate the behavior of ai models — monitoring whether/how these models’ thoughts/processes change.
- From “fixed-rules” to behavioral-guardrails: instead of solely determining whether an individual system is secure, we must assess whether it operates reliably. Should an ai begin producing responses that appear suspect or diverge from its intended function, the system should automatically alert us – much like a credit card issuer alerts us to purchases made in countries where we’ve never previously shopped.
- From technical-silos to sociotechnical-systems: ai-governance necessitates combining technical evaluation with sociological assessment. We must evaluate how humans interact with ai to avoid over-reliance (Cognitive Debt) or feeding it sensitive data that could subsequently leak.
Final thought: safeguarding the Process not just the Product
Historically, we protected products i.e., the software itself – as part of our governance processes. Going forward, we will need to protect the processes i.e., data ingestion/training/feedback loops/evolution associated with ai.
Our objective is not to impede the evolutionary path of ai; instead, we aim to ensure that as it grows -it evolves within safe and trusted parameters. We are transitioning from being “architects of stone” to being “stewardship of systems”.