Building secure artificial intelligence systems: Defending against vulnerabilities in intelligent technologies

Given the increasing capability and applicability of AI systems in sensitive domains within society, we, cyber and information security specialists with a long-standing interest in critical computer systems, must extend our mission to include those systems dedicated to Artificial Intelligence. We must ensure, to the degree feasible, that AI systems function dependably and securely when deployed. After years of pushing back decades of optimism that had located AI systems beyond our field of study, a realistic attitude toward the considerable benefits and, equally, the considerable dangers that AI systems can engender has emerged. While the goal of designing such systems so that they reflect or generate intelligent behavior in a quantifiable way has regained attention, our focus here is on their security. AI systems are vulnerable to a set of attacks that differ on key dimensions from the traditional attacks against conventional computer systems. We refer to this set of attacks as the “AI Security Vulnerability Landscape.” Some of the vulnerabilities of non-AI systems are also present in AI systems, but heightened or modified. In this chapter, we summarize the kinds of vulnerabilities that we feel are most salient. We also consider some new ideas, surprisingly longstanding in some contexts, such as verification of generated behavior. Our particular focus is defensive activities (Huang et al., 2011; Goodfellow et al., 2014; Biggio & Roli, 2018).

To keep our focus limited, we restrict our attention predominantly to Machine Learning, the most visible AI activity. Most of the vulnerabilities that we would summarize for AI systems more generally are also the most relevant for Learning Systems. However, the types of intelligent systems that present other forms of weakness are somewhat broader than the kind of supervised or unsupervised learning through repetition, with a focus on generating probability distributions over symbol strings, that presently dominates in practice. For example, the increasingly popular area of Ontology-based Systems for Knowledge Representation and Generation raises different issues than those affecting Learning Systems. Other logical activities, such as planning via deriving deductions, not already covered, also require distinct emphasis (Moosavi-Dezfooli et al., 2016; Papernot et al., 2016).