## New Research In

### Physical Sciences

### Social Sciences

#### Featured Portals

#### Articles by Topic

### Biological Sciences

#### Featured Portals

#### Articles by Topic

- Agricultural Sciences
- Anthropology
- Applied Biological Sciences
- Biochemistry
- Biophysics and Computational Biology
- Cell Biology
- Developmental Biology
- Ecology
- Environmental Sciences
- Evolution
- Genetics
- Immunology and Inflammation
- Medical Sciences
- Microbiology
- Neuroscience
- Pharmacology
- Physiology
- Plant Biology
- Population Biology
- Psychological and Cognitive Sciences
- Sustainability Science
- Systems Biology

# Maintaining trust when agents can engage in self-deception

Edited by Albert-László Barabási, Northeastern University, Boston, MA, and accepted by Editorial Board Member David A. Weitz July 17, 2018 (received for review February 28, 2018)

## Significance

“He who wants to kill his dog accuses it of having rabies,” the French proverb says. The fact that we alter our beliefs about others to act selfishly and, at the same time, keep a positive self-view has been widely studied by behavioral sciences. Here, we propose a mathematical description of two of these mechanisms of altering beliefs and study a simulation of a society of agents provided with these biases. We find that there are sets of parameters that make societies propagate defection actions and others that protect them from spreading malicious behavior.

## Abstract

The coexistence of cooperation and selfish instincts is a remarkable characteristic of humans. Psychological research has unveiled the cognitive mechanisms behind self-deception. Two important findings are that a higher ambiguity about others’ social preferences leads to a higher likelihood of acting selfishly and that agents acting selfishly will increase their belief that others are also selfish. In this work, we posit a mathematical model of these mechanisms and explain their impact on the undermining of a global cooperative society. We simulate the behavior of agents playing a prisoner’s dilemma game in a random network of contacts. We endow each agent with these two self-deception mechanisms which bias her toward thinking that the other agent will defect. We study behavior when a fraction of agents with the “always defect” strategy is introduced in the network. Depending on the magnitude of the biases the players could start a cascade of defection or isolate the defectors. We find that there are thresholds above which the system approaches a state of complete distrust.

Individuals often deviate from the behavior that maximizes their material reward (1, 2). For example, in the ultimatum game, people prefer to reject profitable offers that they consider unfair (3). This behavior, and other phenomena such as fairness or cooperation (2, 4), can be accounted for within a rational model that includes broader objectives or “social preferences” (altruism, fairness concerns, etc.) as part of the function which agents seek to optimize.

Naturally, agents seek to reduce the problems that arise when material rewards collide with social preferences. For example, believing that others are altruistic may make it more difficult for an agent to act selfishly which, in turn, may reduce its monetary payoff. A way of solving this tension is to develop a self-serving bias: that is, to believe that others are not altruistic to “justify” a selfish act. Cognitive dissonance theory (5, 6) aims to explain the emergence of belief with self-serving biases. The idea is that *dissonance* (contradiction) between cognitions is psychologically uncomfortable, and so it triggers mechanisms of dissonance reduction—and one way of doing so is by altering beliefs (7, 8).

Self-deception mechanisms have been broadly studied in economics (2, 9). Recently, using an experimental design called “The Corruption Game,” we demonstrated two of these principles (10):

Principle 1 (P1) Selfish action alters beliefs about others’ social-preferences.

P2. Ambiguity regarding others’ social preferences increases the likelihood of acting selfishly.

We use the term *Projection* to refer to P1, which is a trait that describes how people blame others for their actions. The notion of *Projection* by which our actions affect how we think of others (11, 12) is at the same time intuitive and paradoxical. From a rational perspective, beliefs about others should be based on what they have done, not on what we have done to them. However, it has been observed that subjects in economic games not only take into account the previous actions of other players, but also their past actions (13, 14). Additionally, people’s beliefs also depend on their own previous actions (10).

Here, we use the colloquial term *Paranoia* to refer to P2 (the idea that if there is ambiguity about how another person may act, an agent will sample the distribution biased for the worse outcomes). Closely related to P2 is the mechanism of “categorization” and “malleability” (15). For example, stealing a pen is more malleable than stealing the money needed to buy the pen. Similarly, the distribution of beliefs on the moral judgment of the malleable case (stealing the pen) is ambiguous, and hence people may use this ambiguity in their favor to act more selfishly.

The aim of this work is twofold: first, to provide a mathematical description of these self-deception mechanisms (*Paranoia* and *Projection*); and second, based on this mathematical description, to investigate the impact that they may have on the evolution of trust among the agents of a society.

## The Model

We study a set of 10^{5} interacting agents that play a modified Prisoner’s Dilemma (*SI Appendix*, section 1.1) game against each other in a static random network. The main difference from other similar approaches investigating networks and evolution of cooperation (or corruption or reputation) (16⇓⇓–19) is that here, we used a Bayesian updating rule and inference process (similar to ref. 20). This rule was necessary to generate a mathematical model of cognitive biases *Paranoia* and *Projection* and study their impact on the propagation of strategies.

For clarity, we divide the strategy of the agents into three stages (Fig. 1): observation, inference, and decision. Observation is the process of accumulation of information about other agent’s actions. The inference process uses observed information and combines it with priors to generate—using a Bayesian model—a belief about other agents’ behavior. This stage is modeled as a beta-binomial process (*SI Appendix*, section 1.3). The output of the inference process is the expected reward for each possible action. Finally, in the decision stage, the agent chooses the option that maximizes her expected reward.

Under these settings, the agents in the network will end up defecting or cooperating with each other depending on the initial conditions. Our primary goal is to investigate how incorporating the cognitive biases described in the introduction (*Paranoia* and *Projection*) affect the evolution of cooperation or defection in the network. In the next subsection, we explain how these cognitive biases can be incorporated into a Bayesian inference process.

The essential step in the inferential process in our model is the estimation of an agent’s probability of defection, θ, in a given interaction—or equivalently, the probability of cooperation

In the beta-binomial model, the *SI Appendix*, section 1.2) is used to describe the prior belief distribution of this variable. Agents use the mean value of their belief as an estimation of this parameter:

*Paranoia* and *Projection* have a different effect on the inference process. Broadly, *Projection* changes the beta distribution (as if own actions were fragments of observed actions), and *Paranoia* results in sampling unevenly (focusing on the worse outcomes) of the beta distribution.

*Projection* is a trait that describes how people blame others for their actions. Although an ideal observer constructs this distribution only from priors and observations, to model this characteristic, each time an agent defects, she modifies her beta distribution of beliefs. With *Projection* the actions of the agent impact on the resulting

Specifically, this is done by changing, whenever the agent defects, the a parameter (which measures the number of observed defections) of the *Projection* in such a way that if *Projection* determines the variation on the b parameter of the

We explore two different variations of the *Projection* bias. First, when the *Projection* affects only the a parameter if the agent defects. We call this asymmetric *Projection*. Second, when the *Projection* affects both, the a and the b, parameters after the agent defects or cooperates, correspondingly. We call this the symmetric *Projection*.

In both cases, the *Projection* bias changes the *Projection*, A will believe that B is more likely to defect. Similarly, A will believe that all other agents are more likely to defect. His own defection has changed his beliefs regarding how all of the other agents he will interact with will behave.

It is important to highlight that, even though this bias yields a wrong value for *Projection* bias (from action to beliefs).

In ref. 10, we showed that the effect of a defecting action produces an average variation of 0.2 in the belief. From this, we can have an estimate of the experimental value of *Projection* of 0.54 (see *SI Appendix*, section 1.4 for details). This result serves as an order of magnitude of realistic values of *Projection* when we inquire about the impact of this parameter on the propagation of cooperation and defection.

The other bias that we explore, *Paranoia* , acts on how *Paranoia* assumes a nonzero value (here, we only investigate positive values), the estimation is given by this implicit equation.*Paranoia* measures the total area under the probability distribution between the optimal and biased estimates *Paranoia* on the estimation of *Paranoia* on *Paranoia* is measured in units of the probability distribution (and hence has one as absolute maximum).

This bias is closely related to the models of reciprocal altruism (10, 21) where belief may be altered too. In these models, changing the belief has a cost that increases as the difference between the unbiased and the biased belief increases. The *Paranoia* bias could be thought of as step function where there is no cost in changing a belief up to some point where the cost, suddenly, approaches infinity. The explicit description of the belief as the mean of a distribution allows us to model P2.

## Results

We study how the network evolves when it is contaminated with a fraction of agents with the strategy ALLD (always defect). These agents do not learn or change their behavior in any way; they stubbornly defect independently of the history of actions.

All simulations begin with a network in which agents trust each other. That is, they believe that the expected reward of cooperating is higher than the expected reward for defecting (Fig. 5). Then, we replace a fraction of the regular agents of the network by ALLD agents. These replacements are distributed at random in the sites of the network. Specifically, the question we ask here is how the network parameters convey more resistance or vulnerability to this “infection” process. To do so, we let the network evolve under the influence of ALLD agents and study whether the defection policy extends over the network.

### Cascades.

First, we study how the system evolves when one ALLD agent is introduced in the network. Under this condition, we measure the fraction of agents that are defecting to each other, which is called the active fraction, *Projection* changes when we use the asymmetric version of the bias. There is a transition in the value *Projection* changes the vulnerability of the agents only if the value of the *Paranoia* is greater than zero. Then, a similar behavior is observed (*SI Appendix*, section 1.7).

In the case of the asymmetric bias, this value, *SI Appendix*, section 1.6).

### Percolation and Phase Transitions.

Now, we generalize this analysis to a broader situation, where not only one agent, but a fraction, *f*, of ALLD agents are present in the network. We examine the robustness of the system by analyzing its evolution as two parameters are changed: the fraction, f, and the value of the *Projection* parameter. We calculate the robustness of the network measuring

Fig 3 shows *Projection* increases. To better understand the transitions in this map, Fig. 4 shows the *Projection* for the asymmetric bias.

When *SI Appendix*, section 1.8). The analytical result yields the value

If the value of *Projection* is greater than zero, the actions of the ALLD agents change the belief of the regular agents in the network. In Fig. 4, it can be seen that when *Projection* increases, the value of

Interestingly, if the value of

If the symmetric version of the *Projection* bias is used, the results remain equivalent only if the *Paranoia* bias is set to a value higher than 0 (*SI Appendix*, section 1.7). Next, we study more generally for all models, how *Projection* and *Paranoia* can interact to affect the robustness of the network.

*Projection* and *Paranoia* Interaction.

The *Paranoia* bias not only affects the dynamics, but the initial condition of the network as well. An agent may believe that others are likely to defect because her internal distribution (based only on the observation of the actions of other agents) has shifted toward defection or because her *Paranoia* bias parameter is greater than zero and her estimation *Paranoia* and the mean of the *Paranoia* parameters in such a way that they yield the same value of *Paranoia* bias can take to keep the initial estimation *Paranoia* and not a more fine-grained set of values as we do with the *Projection* bias and the fraction of ALLD agents. But, we also explore the results using different initial values of *SI Appendix*, sections 1.9 and 1.10.

The four values of *Paranoia* depicted in Fig. 5 in combination with four values of *Projection* yield a matrix of 16 sets of parameters. As depicted in Fig. 6*A*, for the symmetric version of the *Projection* bias, the evolution of all networks could be grouped into three classes. For some parameters, marked in blue in Fig. 6*B*, the networks were highly cooperative, showing a smooth progression of *C*. This is evidence of a hidden phase transition, which is not visible in the overall activation, *A*). We refer to these three classes as (*i*) high cooperation, (*ii*) bistable, and (*iii*) high defection, respectively. These classes are also present when the initial value of *SI Appendix*, section 1.9), and then these results are robust and do not depend on this specific initial value. As in the previous simulations, a similar map with the same type of states and transitions is found if we use the symmetric version of the *Projection* bias (*SI Appendix*, section 1.7).

A specific analysis of which priors yield to different regimes of stability (Fig. 6*A*, first row) indicates that when the *Projection* parameter is set to 0, regardless of the value of paranoia, the networks belong to the high-cooperation class. This means that the society is robust under the inception of a fraction of ALLD agents. For moderate values of *Projection* (0.25) which are lower than the estimated experimentally from ref. 10, the network is in high cooperation for low values of *Paranoia* and shifts to bistability for values of *Paranoia* of 0.36. For this level of *Projection*, even for maximum values of *Paranoia* (this is very close to the strict maximum since greater values of *Paranoia* are incompatible with a network that begins in full cooperation), the network never is in the high-defection class. For higher values of *Projection* (0.75), which are slightly above experimental estimates, the network displays the three different behaviors, depending on the value of *Paranoia*. If it is zero, then the network is in high cooperation, and, as the value of *Paranoia* increases, the network behaves as a bistable system and, finally, is in a high defection state.

## Discussion

By using a Bayesian updating rule in an agent-based simulation, we were able to model how two specific cognitive biases, *Projection* and *Paranoia*, impact the decision-making process. Then, we use this to inquire how these parameters affect the propagation of defection started by a set of ALLD agents.

Our first result is that if only one ALLD agent is introduced in the network, there is a threshold in the value of *Projection* up to which the agents keep cooperating with each other. If the value of *Projection* is higher than the threshold, then a positive fraction of the agents start to defect. In the case of the asymmetric version of the *Projection* bias, the threshold could be deduced analytically and coincides with the value found in our simulations.

Then, we find two kinds of transitions when a fraction of ALLD agents is introduced. If the value of *Projection* is low enough, there is only one transition at *Projection* is >0.8, we find another transition at

Additionally, we study how the effect of *Paranoia* and *Projection* interact in the spreading of the defections produced by fraction ALLD agents. The main result is that, if the *Projection* bias is set to zero, the *Paranoia* bias does not make the network less robust to the infection of ALLD agents. Only if it is combined with the *Projection* bias does it weaken the network.

A distinguishing characteristic of our model is that it is built upon a network, and the agents can choose a different action for different contacts. In their seminal work, Nowak and May (25) use a regular lattice where each agent plays only one action in each step against their contacts. This model is suitable to study the evolution of cooperation, but we believe that, to investigate cooperation at a cultural level, we have to add the possibility of acting differently with different contacts. We used a static random network, but this is just another parameter of the model that can be modified. For example, a network whose degree distribution follows a power law, or a small-world network, could be used (26, 27). It is also possible to add dynamics to the network, allowing the interaction to change iteration after iteration. Additionally, the ALLD agents could be placed, not randomly, but rather in high-centrality nodes or in given k-shell within the network. This variation could be used to investigate the effect that highly connected people’s behavior might have on the rest of the community.

The long-term motivation for this study is to understand why different societies may converge to different policies of cooperation and defection. One particular case which we are interested in, and which was the motivation for the work on the corruption game (10), was how this might impact in different degrees of corruption. Recently, Gächter and Schulz (28) using an anonymous die-rolling experiment (29) showed that there is a correlation between individual traits and global corruption. The corruption game (10) was measured in the United States and Argentina, where there are very different indices of corruption and results were not very different. This suggests that *Projection* is not the most likely psychological bias to account for. This is consistent with our findings that a wide range of behaviors is only observed for experimentally observed values of *Projection*. Within this range, variability in *Paranoia* may explain bifurcations between societies that (with similar initial states) converge to defection or cooperation, or similarly in our interest to a high or low level of corruption. Another source of variability could be the effective impact of the ALLD agents in a society. Some governments have more efficient institutions which control the acts of the ALLD agents more effectively. Small changes in this control could lead to a different effective fraction of ALLD agents which, in a society that is in a bistable state, will lead to convergence to cooperation or defection.

Our work builds on, and links, two different fields of behavioral sciences. On the one hand, a tradition that has studied agents with simple, yet effective, strategies, and which of these strategies prevails in different contexts. Under this approach, using unbiased agents, the appearance and prevalence of corruption have been investigated (17, 18, 20). The conundrum of the emergence of cooperation has been addressed under this framework, too (30, 31). On the other hand, our work builds on a Bayesian approach to decision making that has sought to inquire how priors and evidence are used to generate beliefs and guide actions (32, 33). This work can be seen as a mixture of these two traditions, where we can then ask how low-level psychological constructs which affect the inferential process (micro) have an impact on the large-scale organization of societies (macro). To build this bridge between these traditions, we use network theory, and the process that emerged under this framework was very similar to an already-known network process: bootstrap percolation. We did not impose this process, but it was the result of the cognitive model of the agents. Under this framework, we find that small variations on cognitive biases could have a significant impact on the average behavior of all of society. According to our model, society-level phenomena, including the broken window theory (34) and the broad range in the degree of corruption among countries, is the consequence of cognitive dissonance reduction mechanisms such as *Projection* and *Paranoia*.

## Acknowledgments

We thank Nicolas E. Stier-Moses for his helpful comments. This research was supported by Consejo Nacional de Investigaciones Científicas y Técnicas and Fondo para la Investigación Científica y Tecnológica Grant PICT 2013 N°1653. M.S. is sponsored by the James McDonnell Foundation 21st Century Science Initiative in Understanding Human Cognition-Scholar Award. H.A.M. is sponsored by NIH National Institute of Biomedical Imaging and Bioengineering Grant R01EB022720 and NSF Information and Intelligent Systems Grant 1515022.

## Footnotes

↵

^{1}A.B., H.A.M., R.D., and M.S. contributed equally to this work.- ↵
^{2}To whom correspondence should be addressed. Email: ababino{at}df.uba.ar.

Author contributions: A.B., H.A.M., R.D., and M.S. designed research; A.B. performed research; A.B. analyzed data; and A.B., H.A.M., R.D., and M.S. wrote the paper.

The authors declare no conflict of interest.

This article is a PNAS Direct Submission. A.-L.B. is a guest editor invited by the Editorial Board.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1803438115/-/DCSupplemental.

Published under the PNAS license.

## References

- ↵
- ↵
- ↵
- ↵
- ↵
- Festinger L

- ↵
- ↵
- ↵
- Izuma K, et al.

- ↵
- Rabin M

*Moral Preferences, Moral Constraints, and Self-Serving Biases (Department of Economics, Univ of California, Berkeley)*. - ↵
- Di Tella R,
- Perez-Truglia R,
- Babino A,
- Sigman M

- ↵
- ↵
- ↵
- ↵
- Gracia-Lazaro C, et al.

- ↵
- ↵
- Bear A,
- Rand DG

- ↵
- ↵
- ↵
- ↵
- Sah RK

- ↵
- ↵
- Watts D

- ↵
- Baxter GJ,
- Dorogovtsev SN,
- Goltsev AV,
- Mendes JFF

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- Rand DG,
- Tarnita CE,
- Ohtsuki H,
- Nowak MA

- ↵
- ↵
- Austerweil JL,
- Gershman SJ,
- Tenenbaum JB,
- Griffiths TL

- ↵
- Keizer K

## Citation Manager Formats

## Sign up for Article Alerts

## Article Classifications

- Social Sciences
- Psychological and Cognitive Sciences