Deceptive AI

Psychologists and cognitive scientists have long drawn insights and evidence from stage magic about human perceptual and attentional errors. We present a complementary analysis of conjuring tricks that seeks to understand the experience of impossibility that they produce. Our account is first motivated by insights about the constructional aspects of conjuring drawn from magicians’ instructional texts. A view is then presented of the logical nature of impossibility as an unresolvable contradiction between a perception-supported belief about a situation and a memory-supported expectation. We argue that this condition of impossibility is constructed not simply through misperceptions and misattentions, but rather it is an outcome of a trick’s whole structure of events. This structure is conceptualised as two parallel event sequences: an effect sequence that the spectator is intended to believe; and a method sequence that the magician understands as happening. We illustrate the value of this approach through an analysis of a simple close-up trick, Martin Gardner’s Turnabout. A formalism called propositional dynamic logic is used to describe some of its logical aspects. This elucidates the nature and importance of the relationship between a trick’s effect sequence and its method sequence, characterised by the careful arrangement of four evidence relationships: similarity, perceptual equivalence, structural equivalence, and congruence. The analysis further identifies two characteristics of magical apparatus that enable the construction of apparent impossibility: substitutable elements and stable occlusion.

Every model involves assumptions. While some are standard to all models that simulate intelligent decision-making (eg, discrete/continuous, static/dynamic), goal recognition is well known also to involve choices about the observed agent: is it aware of being observed? cooperative or adversarial? In this paper, we examine not only these but the many other assumptions made in the context of model-based goal recognition. By exploring their meaning, the relationships between them and the confusions that can arise, we demonstrate their importance, shed light on the way trends emerge in AI, and suggest a novel means for researchers to uncover suitable avenues for future work.

Goal recognition is the problem of determining an agent’s intent by observing its actions. In the context of AI research, the problem is tackled for two quite different purposes: to determine an agent’s most probable goal or, for human-aware planning including planned—or strategic—deception, to determine an observer’s most likely belief about that goal. Making no distinction, contemporary models tend to assume an infallible observer, deceived only while it has limited access to information or if the environment itself is only partially observable. Focusing on the second purpose, we propose an extended framework that incorporates formal definitions of confirmation bias, selective attention and memory decay. In contrast to pre-existing models, our approach combines explicit consideration of prior probabilities with a principled representation of observer confidence and distinguishes between potential observations, ie, every observable event within the observer’s frame of reference—and recalled observations which we model as a function of attention and memory. We show that when these factors are taken into consideration, false beliefs may arise and can be made to persist, even in a fully observable environment—thus providing a perceptual model readily incorporated into the ‘thinking’ of an adversarial agent for the purpose of strategic deception.

In this paper, we study the problem of deceptive reinforcement learning to preserve the privacy of a reward function. Reinforcement learning is the problem of finding a behaviour policy based on rewards received from exploratory behaviour. A key ingredient in reinforcement learning is a reward function, which determines how much reward (negative or positive) is given and when. However, in some situations, we may want to keep a reward function private; that is, to make it difficult for an observer to determine the reward function used. We define the problem of privacy-preserving reinforcement learning, and present two models for solving it. These models are based on dissimulation – a form of deception that ‘hides the truth’. We evaluate our models both computationally and via human behavioural experiments. Results show that the resulting policies are indeed deceptive, and that participants can determine the true reward function less reliably than that of an honest agent

Contemporary cost-based goal-recognition assumes rationality: that observed behaviour is more or less optimal. Probabilistic goal recognition systems, however, explicitly depend on some degree of sub-optimality to generate probability distributions. We show that, even when an observed agent is only slightly irrational (sub-optimal), state-of-the-art systems produce counter-intuitive results (though these may only become noticeable when the agent is highly irrational). We provide a definition of rationality appropriate to situations where the ground truth is unknown, define a rationality measure (RM) that quantifies an agent’s expected degree of sub-optimality, and define an innovative self-modulating probability distribution formula for goal recognition. Our formula recognises sub-optimality and adjusts its level of confidence accordingly, thereby handling irrationality—and rationality—in an intuitive, principled manner. Building on that formula, moreover, we strengthen a previously published result, showing that “single-observation” recognition in the path-planning domain achieves identical results to more computationally expensive techniques, where previously we claimed only to achieve equivalent rankings though values differed.