Manipulability of comparative tests

doi:10.1073/pnas.0812602106

Manipulability of comparative tests

Wojciech Olszewski^a,¹ and
Alvaro Sandroni^b,^c

^aDepartment of Economics, Northwestern University, 2001 Sheridan Road, Evanston, IL 60208;
^bDepartment of Economics, University of Pennsylvania, 3718 Locust Walk, Phildadelphia, PA 19104; and
^cDepartment of Managerial Economics and Decision Sciences, Kellogg School of Management, Northwestern University, 2001 Sheridan Road, Evanston, IL 60208

Edited by Avinash K. Dixit, Princeton University, Princeton, NJ, and approved January 30, 2009 (received for review December 17, 2008)

Abstract

Multiple self-proclaimed experts claim that they know the probabilities of future events. A tester does not know the odds of future events and she also does not know whether, among the multiple experts, there are some who do know the relevant probabilities. So the tester requires each expert to announce, before any data are observed, the probabilities of all future events. A test either rejects or does not reject each expert based on the observed data and the profile of the probabilities announced by the experts. We assume that the test controls for the type I error of rejecting the true probabilities. However, consider the case in which all experts are uninformed (i.e., they do not know anything about true probabilities). We show that they can still independently produce false forecasts that are likely to both pass the test, no matter how the data evolve in the future. Hence, the data may not suffice to effectively discredit uninformed, but strategic, experts.

Keywords:

Footnotes

¹To whom correspondence should be addressed. E-mail: wo{at}northwestern.edu
Author contributions: W.O. and A.S. designed research, performed research, analyzed data, and wrote the paper.
The authors declare no conflict of interest.
This article is a PNAS Direct Submission.
This article contains supporting information online at www.pnas.org/cgi/content/full/0812602106/DCSupplemental.
↵† As we argue in section 4.2, the substantive point of our result holds for general contracts where the experts' motivations may go beyond avoiding rejection.
↵‡ In ref. 10, the authors show that, if it is known that some of the experts are informed, then (under some conditions) the data may be able to identify which experts are informed.
↵§ Rejection sets are assumed to be such that if a theory is rejected at a history h^t = (ω₁,…,ω_t−1), then it is also rejected at all extensions of history h^t, i.e., at all histories h^m = (ω′₁,…,ω′_t−1,ω′_t,…,ω′_m−1) such that m > t and (ω₁,…,ω_t−1) = (ω′₁,…,ω′_t−1).
↵¶ For expositional simplicity, we assume that the experts do not discount the future. That is, the disutility d does not depend on the period in which their theories are rejected. However, all our arguments remain valid in the discounted case.
↵∥ Any theory f uniquely defines a probability distribution on the set {0,1}^∞ of infinite histories. So, one can interpret a theory as a probability distribution on {0,1}^∞, parametrized by the conditional probabilities Pr(ω_t | h^t).
↵** This definition requires the set F × F to be equipped with a σ algebra and the provision that for i = 1 and i = 2, and any h^t ∈ H_∞, the set revelation R_i(h^t) is measurable with respect to that σ algebra.

↵The set of all theories F is the set of functions from a countable set to [0,1]. If we equip [0,1] with the σ algebra of Borel sets, then F inherits the product Borel structure. Similarly, F × F inherits the product Borel structure of two copies of F. All tests C in this article are assumed to be such that the revelation sets R_i(h^t) are measurable with respect to this σ algebra.
↵†† In several cases, the tester can use only future-independent tests. This is true, for example, in the case in which the experts claim that they will know the probability that 1 occurs at period t + 1 no earlier than at period t. We refer the reader to ref. 12 for a detailed discussion of future independence.

↵The assumption of future independence can be dispensed with if (unlike the model in this article) the tester has bounded datasets or the experts discount future payoffs. In general, future independence can be relaxed, but not completely dispensed with. We refer the reader to refs. 13 and 14 for future-dependent tests that are likely to pass the true theory and cannot be ignorantly passed.
↵‡‡ If the set of expert 2's pure strategies is restricted, then we may no longer have the property that for every mixed strategy of nature, there is a strategy for expert 2 that gives him a payoff of 1−ɛ, or higher. However, an additional step in our proof shows that this property is preserved for properly chosen sets of predictions R_t ⊂ [0,1].
↵§§ This statement must be modified, of course, to the effect that, as we assumed in section 2 (see footnote §), every rejection set automatically includes histories h^t with t > m, whose first m outcomes coincide with the outcomes of a history that belongs to $\text{[math]}$ _i(f¹, f². The fact that the rejection sets consist of m histories (and their extensions) ensures that test $\text{[math]}$ satisfies the required measurability provision (see footnote **).
↵¶¶ To see an example of such a history, take in each period s = 1,…,m an outcome that is more likely according to $\text{[math]}$ ⁱ than according to fⁱ.
Freely available online through the PNAS open access option.