Skip to main content
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Accessibility Statement
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian
  • Log in
  • My Cart

Main menu

  • Home
  • Articles
    • Current
    • Special Feature Articles - Most Recent
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • List of Issues
  • Front Matter
  • News
    • For the Press
    • This Week In PNAS
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Editorial and Journal Policies
    • Submission Procedures
    • Fees and Licenses
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Accessibility Statement
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian

User menu

  • Log in
  • My Cart

Search

  • Advanced search
Home
Home

Advanced Search

  • Home
  • Articles
    • Current
    • Special Feature Articles - Most Recent
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • List of Issues
  • Front Matter
  • News
    • For the Press
    • This Week In PNAS
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Editorial and Journal Policies
    • Submission Procedures
    • Fees and Licenses

New Research In

Physical Sciences

Featured Portals

  • Physics
  • Chemistry
  • Sustainability Science

Articles by Topic

  • Applied Mathematics
  • Applied Physical Sciences
  • Astronomy
  • Computer Sciences
  • Earth, Atmospheric, and Planetary Sciences
  • Engineering
  • Environmental Sciences
  • Mathematics
  • Statistics

Social Sciences

Featured Portals

  • Anthropology
  • Sustainability Science

Articles by Topic

  • Economic Sciences
  • Environmental Sciences
  • Political Sciences
  • Psychological and Cognitive Sciences
  • Social Sciences

Biological Sciences

Featured Portals

  • Sustainability Science

Articles by Topic

  • Agricultural Sciences
  • Anthropology
  • Applied Biological Sciences
  • Biochemistry
  • Biophysics and Computational Biology
  • Cell Biology
  • Developmental Biology
  • Ecology
  • Environmental Sciences
  • Evolution
  • Genetics
  • Immunology and Inflammation
  • Medical Sciences
  • Microbiology
  • Neuroscience
  • Pharmacology
  • Physiology
  • Plant Biology
  • Population Biology
  • Psychological and Cognitive Sciences
  • Sustainability Science
  • Systems Biology
Research Article

Asymptotic theory of rerandomization in treatment–control experiments

Xinran Li, Peng Ding, and Donald B. Rubin
PNAS September 11, 2018 115 (37) 9157-9162; first published August 27, 2018; https://doi.org/10.1073/pnas.1808191115
Xinran Li
aDepartment of Statistics, Harvard University, Cambridge, MA 02138;
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Peng Ding
bDepartment of Statistics, University of California, Berkeley, CA 94720
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Donald B. Rubin
aDepartment of Statistics, Harvard University, Cambridge, MA 02138;
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: dbrubin@mac.com
  1. Contributed by Donald B. Rubin, June 29, 2018 (sent for review May 17, 2018; reviewed by Robert J. Tibshirani and C. F. Jeff Wu)

  • Article
  • Figures & SI
  • Info & Metrics
  • PDF
Loading

Significance

Rerandomization refers to experimental designs that enforce covariate balance. This paper studies the asymptotic properties of the difference-in-means estimator under rerandomization, based on the randomness of the treatment assignment without imposing any parametric modeling assumptions on the covariates or outcome. The non-Gaussian asymptotic distribution allows for constructing large-sample confidence intervals for the average treatment effect and demonstrates the advantages of rerandomization over complete randomization.

Abstract

Although complete randomization ensures covariate balance on average, the chance of observing significant differences between treatment and control covariate distributions increases with many covariates. Rerandomization discards randomizations that do not satisfy a predetermined covariate balance criterion, generally resulting in better covariate balance and more precise estimates of causal effects. Previous theory has derived finite sample theory for rerandomization under the assumptions of equal treatment group sizes, Gaussian covariate and outcome distributions, or additive causal effects, but not for the general sampling distribution of the difference-in-means estimator for the average causal effect. We develop asymptotic theory for rerandomization without these assumptions, which reveals a non-Gaussian asymptotic distribution for this estimator, specifically a linear combination of a Gaussian random variable and truncated Gaussian random variables. This distribution follows because rerandomization affects only the projection of potential outcomes onto the covariate space but does not affect the corresponding orthogonal residuals. We demonstrate that, compared with complete randomization, rerandomization reduces the asymptotic quantile ranges of the difference-in-means estimator. Moreover, our work constructs accurate large-sample confidence intervals for the average causal effect.

  • causal inference
  • covariate balance
  • geometry of rerandomization
  • Mahalanobis distance
  • quantile range

Ever since Fisher’s (1⇓–3) seminal work, randomized experiments have become the “gold standard” for drawing causal inferences. Complete randomization balances the covariate distributions between treatment groups in expectation, thereby ensuring the existence of unbiased estimators of average causal effects. Covariate imbalance, however, often occurs in specific randomized experiments, as recognized by Fisher (2) and later researchers (e.g., refs. 4⇓⇓⇓⇓–9). The standard approach advocated by Fisher (3), stratification or blocking, ensures balance with a few discrete covariates (e.g., refs. 10⇓–12).

When a randomized allocation is unbalanced, it is reasonable to discard that allocation and redraw another one until a certain predetermined covariate balance criterion is satisfied. This is rerandomization, an experimental design hinted at by Fisher (cf. ref. 13, p. 88) and Cox (14, 15) and formally proposed by Rubin (16) and Morgan and Rubin (17). Morgan and Rubin (17) showed that the difference-in-means estimator is generally unbiased for the average causal effect under rerandomization with equal-sized treatment groups and obtained the sampling variance of this estimator under additional assumptions of Gaussian covariate and outcome distributions and additive causal effects. When rerandomization is applied but these assumptions do not hold, statistical inference becomes more challenging, because the theory that is justified by the central limit theorem under complete randomization (18, 19) no longer generally holds. Some applied researchers believe that “the only analysis that we can be completely confident in is a permutation test or rerandomization test” (ref. 7, p. 219). However, permutation tests based on randomization require sharp null hypotheses that imply all missing potential outcomes are known.

Analogous to the repeated sampling properties for complete randomization (11, 20), we evaluate the repeated sampling properties of the difference-in-means estimator when rerandomization is used, where all potential outcomes and covariates are regarded as fixed quantities and all randomness arises solely from the random treatment assignments. The geometry of rerandomization reveals non-Gaussian asymptotic distributions, which serve as the foundation for constructing large-sample confidence intervals for average causal effects. Furthermore, we compare the lengths of quantile ranges of the asymptotic distributions of the difference-in-means estimator under rerandomization and complete randomization, extending Morgan and Rubin’s (17, 21) comparison of their sampling variances.

Framework, Notation, and Basic Results

Covariate Imbalance and Rerandomization.

Inferring the causal effect of some binary treatment on an outcome Y is of central interest in many studies. We consider an experiment with n units, with n1 assigned to treatment and n0 assigned to control, n=n1+n0. Before conducting the experiment, we collect K covariates with values Xi=(X1i,X2i,…,XKi)′ for the ith unit, which can possibly include transformations of basic covariates and their interactions. Let Zi be the treatment indicator for unit i (Zi=1 if the active treatment level; Zi=0 if the control level) and Z=(Z1,Z2,…,Zn)′ be the treatment assignment vector with n1≡∑i=1nZi and n0≡∑i=1n(1−Zi). In a completely randomized experiment (CRE), n1 and n0 are fixed, and the distribution of Z is such that each possible value, z=(z1,…,zn)′, of Z has probability n1!n0!/n!. The difference-in-means vector of the covariates between treatment and control groups is τ^X=n1−1∑i=1nZiXi−n0−1∑i=1n(1−Zi)Xi. Although on average τ^X has mean zero over all n!/(n1!n0!) randomizations, for any realized value of Z, imbalancedness in covariate distributions between treatment groups often occurs. For 10 uncorrelated covariates, with probability 40%, at least one of the absolute values of the t statistics comparing covariate means will be larger than 1.96, the 0.975 quantile of the standard Gaussian distribution (17).

When covariate imbalance arises in a drawn allocation, it is reasonable to discard that allocation and draw another until some a priori covariate balance criterion is satisfied. This is rerandomization, an intuitive experimental design tool apparently personally advocated by R. A. Fisher (discussion in ref. 16) and formally discussed by Morgan and Rubin (17).

In general, rerandomization entails the following steps: (i) Collect covariate data; (ii) specify a balance criterion to determine whether a randomization is acceptable or not; (iii) randomize the units to treatment and control groups; (iv) if the balance criterion is satisfied, proceed to step v, and otherwise, return to step iii; (v) conduct the experiment using the final randomization obtained in step iv; and (vi) analyze the data, taking into account the rerandomization used in steps ii–iv.

Although the balance criterion in step ii can be general, Morgan and Rubin (17) suggested using the Mahalanobis distance between covariate means in treatment and control groups, and they (21) suggested considering tiers of covariates according to their presumed importance in predicting the outcomes in this experiment. We discuss these two types of rerandomization in detail and apposite statistical inference after these rerandomizations as implied by step vi. We then extend the theory to general rerandomizations in SI Appendix, section A1.

Potential Outcomes and Definitions of Finite Population Quantities.

We use the potential outcomes framework to define causal effects and let Yi(1) and Yi(0) denote the potential outcomes of unit i under active treatment and control, respectively. On the difference scale, the individual causal effect for unit i is τi=Yi(1)−Yi(0), and the average causal effect for the finite population of n units is τ=∑i=1nτi/n. Let Ȳ(z)=∑i=1nYi(z)/n be the finite population average of potential outcomes under treatment arm z and X¯=∑i=1nXi/n be the finite population average of covariates. Let SY(z)2=∑i=1n{Yi(z)−Ȳ(z)}2/(n−1) be the finite population variance (with divisor n−1) of the potential outcomes under treatment arm z, Sτ2=∑i=1n(τi−τ)2/(n−1) be the finite population variance of the individual causal effects, SY(z),X=SX,Y(z)′=∑i=1n{Yi(z)−Ȳ(z)}(Xi−X¯)′/(n−1) be the finite population covariance between potential outcomes and covariates, and SX2=∑i=1n(Xi−X¯)(Xi−X¯)′/(n−1) be the finite population covariance matrix of covariates. These fixed quantities depend on n implicitly, but do not depend on the randomization or rerandomization scheme.

Repeated Sampling Inference in a CRE.

The observed outcome for unit i is Yi=ZiYi(1)+(1−Zi)Yi(0), a function of treatment assignment and potential outcomes. In a CRE, Neyman (20) showed that, for estimating τ, the difference-in-means estimator τ^=n1−1∑i=1nZiYi−n0−1∑i=1n(1−Zi)Yi is unbiased (the expectation of τ^ over all possible randomizations is τ) and obtained its sampling variance over all randomizations for constructing a large-sample confidence interval for τ. However, Neyman’s (20) interval is not accurate if rerandomization is used.

Let r1=n1/n and r0=n0/n be the proportions of units receiving treatment and control. According to the finite population central limit theorem (19), under some regularity conditions, the large-n sampling distribution, over all randomizations, of n(τ^−τ,τ^X′) is Gaussian with mean zero and covariance matrixV=VττVτxVxτVxx =r1−1SY(1)2+r0−1SY(0)2−Sτ2r1−1SY(1),X+r0−1SY(0), Xr1−1SX,Y(1)+r0−1SX,Y(0)(r1r0)−1SX2.We are conducting randomization-based inference, where all of the covariates and potential outcomes are fixed numbers, and randomness comes solely from the treatment assignment. We embed n units into an infinite sequence of finite populations with increasing sizes, and a sufficient condition for the asymptotic Gaussianity of n(τ^−τ,τ^X′) is as follows (19).

Condition 1:

As n→∞, for z=0,1, (i) rz, the proportion of units under treatment arm z, has positive limits; (ii) the finite population variances and covariances SY(z)2,Sτ2,SX2 and SX,Y(z) have finite limiting values, and the limit of SX2 is nonsingular; and (iii) max1≤i≤n|Yi(z)−Ȳ(z)|2/n→0 and max1≤i≤n‖Xi−X¯‖22/n→0.

We introduce the notation ∼. for two sequences of random vectors converging weakly to the same distribution. Therefore, under the CRE and Condition 1, n(τ^−τ,τ^X′)∼̇(A,B′), where (A,B′) is a random vector from N(0,V) (19).

Rerandomization Using the Mahalanobis Distance

Mahalanobis Distance.

The Mahalanobis distance between the covariate means in treatment and control groups isM=τ^X′{Var(τ^X)}−1τ^X=nτ^X′ Vxx−1nτ^X,where Vxx=(r1r0)−1SX2 is a fixed and known K×K matrix in our setting. A rerandomization scheme proposed by Morgan and Rubin (17) accepts only those randomizations with the Mahalanobis distance less than or equal to a, a prespecified threshold. Let M denote the event that a treatment assignment Z is accepted; that is, M≤a. Below we use rerandomization using the Mahalanobis distance (ReM) to denote rerandomization using this criterion, which, as a design, depends on both the covariates and the threshold a.

When we allow transformations and interactions of X, ReM can incorporate a wide class of rerandomization schemes. For small sample sizes, there may not exist any randomization satisfying some balance criterion. However, according to the finite population central limit theorem (19), the acceptance probability of a randomization is asymptotically pa=P(χK2≤a). Therefore, for relatively large sample sizes, there usually exist many randomizations satisfying the balance criterion. In practice, we want to choose the asymptotic acceptance probability to be small, e.g., pa=0.001. We comment on this issue in Discussion.

Multiple Correlation Between τ^ and τ^X.

The sampling distribution of τ^ under ReM depends on the squared multiple correlation between τ^ and τ^X under the CRE, which is also the proportion of the variance of τ^ explained by τ^X in linear projection: R2=Cov(τ^,τ^X)Var(τ^X)−1Cov(τ^X,τ^)/Var(τ^)=VτxVxx−1Vxτ/Vττ. Define the variance of the linear projection of Y(z) on X as SY(z)∣X2=SY(z), XSX2−1SX,Y(z) for z=0,1. We similarly define Sτ∣X2, the variance of the linear projection of τ on X.

Proposition 1.

R2 can be expressed in terms of the variances of the potential outcomes and of their projections on X:R2=r1−1SY(1)∣X2+r0−1SY(0)∣X2−Sτ∣X2r1−1SY(1)2+r0−1SY(0)2−Sτ2.

When the causal effect is additive, Sτ2=0, Sτ,X=0, and SY(1),X=SY(0),X, and then R2=SY(0)∣X2/SY(0)2 is the squared multiple correlation between X and Y(0).

Asymptotic Sampling Distribution of τ^ Under ReM.

Simply stated, n(τ^−τ) has two parts: the part unrelated to the covariates, which we call ε0, and is thus unaffected by rerandomization, and the other part related to the covariates, which we call LK,a, and is thus affected by rerandomization. Therefore, the asymptotic distribution of τ^ is a linear combination of two independent random variables: ε0∼N(0,1) is a standard Gaussian random variable, and LK,a∼D1∣D′D≤a, where D=(D1,…,DK)′∼N(0,IK).

Theorem 1.

Under ReM and Condition 1,n(τ^−τ)∣M∼̇ Vττ1−R2⋅ε0+R2⋅LK,a,[1]where ε0 is independent of LK,a.

The coefficients of the linear combination are functions of R2, which measures the association between the potential outcomes and the covariates. When R2=0, the right-hand side of [1] becomes a Gaussian random variable, the same as the asymptotic distribution of n(τ^−τ) in Repeated Sampling Inference in a CRE. Importantly, the definition of R2 is based on linear projections instead of linear models of the potential outcomes. Our asymptotic theory is based on the distribution of the randomization without imposing any modeling assumptions on the potential outcomes.

Representation of the Asymptotic Distribution Under ReM.

The asymptotic distribution in [1] involves a random variable LK,a that does not appear in standard statistical problems. The spherical symmetry of the standard Gaussian vector allows us to represent LK,a using some known distributions, which allows for easy simulation of LK,a.

Let χK,a2∼χK2∣χK2≤a be a truncated χ2 random variable, UK be the first coordinate of the uniform random vector over the (K−1)-dimensional unit sphere, S be a random sign taking ±1 with probability 1/2, and βK∼Beta1/2,(K−1)/2 be a Beta random variable degenerating to point mass at 1 when K=1.

Proposition 2.

LK,a can be represented asLK,a∼D1∣D′D≤a∼χK,aUK∼χK,aSβK,[2]where (χK,a,UK) are mutually independent, and (χK,a,S,βK) are jointly independent. LK,a is symmetric and unimodal around zero, with Var(LK,a)=vK,a=P(χK+22≤a)/P(χK2≤a)<1.

Because both ε0 and LK,a are symmetric and both are unimodal at zero, their linear combination is also symmetric and unimodal at zero according to Wintner’s (22) theorem. The same is true for the asymptotic distribution of n(τ^−τ) in [1]. Moreover, the unimodal property plays an important role in the conservativeness of confidence intervals, discussed shortly for ReM. The representation in [2] allows for easy simulation of LK,a, as well as the asymptotic distribution of n(τ^−τ) in [1], which is relevant for statistical inference discussed later.

In SI Appendix, section A2, we give more detailed explanations regarding the geometry and the shape of the asymptotic distribution in [1].

Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.

Theorem 1 characterizes the asymptotic behavior of τ^ over ReM, which immediately implies the following conclusions.

First, the asymptotic distribution in [1] is symmetric around 0, implying that τ^ is asymptotically unbiased for τ. Let Ea(⋅) denote the expectation of the asymptotic sampling distribution of a sequence of random vectors.

Corollary 1.

Under ReM and Condition 1, Ean(τ^−τ)|M=0.

Morgan and Rubin (17) gave a counterexample showing that, in an experiment with unequal treatment group sizes, τ^ can be biased for τ over ReM. Our result confirms a conjecture in ref. 21 that the bias is often small with large samples. Corollary 1 extends their theorem 2.1 (17) and ensures the asymptotic unbiasedness of τ^ for experiments with any ratio of n1/n0.

Covariates, whether observed or unobserved, are variables unaffected by the treatments. Therefore, the average causal effect on any covariate is 0, and Corollary 1 implies that any covariate asymptotically has the same means under treatment and control.

Furthermore, from Proposition 2 and Theorem 1, we can calculate the asymptotic sampling variances of τ^X and τ^ and the percentage reductions in asymptotic sampling variances (PRIASV) under ReM compared with the CRE. Recalling that vK,a=P(χK+22≤a)/P(χK2≤a), we summarize the results in Corollary 2.

Corollary 2.

Under ReM and Condition 1, the asymptotic sampling covariance of nτ^X is vK,aVxx, and the PRIASV of any component of nτ^X is 1−vK,a. The asymptotic sampling variance of n(τ^−τ) is Vττ1−(1−vK,a)R2, and the PRIASV of n(τ^−τ) is (1−vK,a)R2.

Rigorously, the asymptotic sampling covariance and variance of τ^X and τ^ should be the limits of vK,aVxx and Vττ{1−(1−vK,a)R2} in the sequence of finite populations. However, for descriptive convenience, we omit these limit signs when discussing the expectation and covariance of asymptotic sampling distributions. When a is close to 0, that is, when the asymptotic acceptance probability is small, the asymptotic sampling variance Vττ1−(1−vK,a)R2 reduces to Vττ(1−R2), which is identical to the asymptotic sampling variance of the regression-adjusted estimator under the CRE (18). Therefore, rerandomization accomplishes covariate adjustment in the design stage, whereas regression accomplishes covariate adjustment in the analysis stage.

When the causal effect is additive, R2 equals the finite population squared multiple correlation between X and Y(0). Therefore, Corollary 2 is an asymptotic extension of theorem 3.2 in Morgan and Rubin (17).

Under ReM, in addition to the sampling variance reduction result concerning τ^ in Corollary 2, we consider the reduction in the length of the (1−α) quantile range of τ^ compared with that under the CRE. We choose the length of the (1−α) quantile range, because of its connection to constructing confidence intervals as discussed shortly.

Let zξ be the ξth quantile of a standard Gaussian distribution. Let νξ(R2,pa,K) be the ξth quantile of 1−R2⋅ε0+R2⋅LK,a, with νξ(0,pa,K)=zξ. Because pa and K are usually known by design, we write νξ(R2,pa,K) as νξ(R2) for notational simplicity. Under ReM, the (1−α) quantile range of the asymptotic distribution of n(τ^−τ) isQRα(Vττ,R2)=να/2(R2)Vττ,  ν1−α/2(R2)Vττ,[3]and the corresponding quantile range under the CRE isQRα(Vττ,0)=zα/2Vττ,  z1−α/2Vττ.[4]

Theorem 2.

Under Condition 1, the length of the (1−α) quantile range of the asymptotic sampling distribution of n(τ^−τ) under ReM is less than or equal to that under the CRE, with the difference nondecreasing in R2 and nonincreasing in pa and K.

Sampling Variance Estimation and Confidence Intervals.

Asymptotic sampling variance and quantile ranges for τ^ depend on Vττ and R2, which are determined by the covariances among potential outcomes and covariates. To obtain a sampling variance estimator and to construct an asymptotic confidence interval for τ, we need to estimate these variances and covariances. Let sY(z)2, sY(z)∣X2, and sY(z),X be the sample variance of Y, sample variance of linear projection of Y on X, and sample covariance between Y and X in treatment arm z. We show in SI Appendix, section A4 that under ReM they are consistent for their population analogues. Therefore, we can then estimate Sτ∣X2 bysτ∣X2=(sY(1), X−sY(0), X)(SX2)−1(sX,Y(1)−sX,Y(0))and Vττ by (23)V^ττ=r1−1sY(1)2+r0−1sY(0)2−sτ∣X2.We can then estimate R2 byR^2=V^ττ−1r1−1sY(1)∣X2+r0−1sY(0)∣X2−sτ∣X2.[5]We set R^2 to be 0 if the estimator in [5] is negative.

According to Corollary 2, we can estimate the asymptotic sampling variance of τ^ by V^ττ{1−(1−vK,a)R^2}/n, and according to [3], we can construct a large sample (1−α) confidence interval for τ using τ^−QRα(V^ττ,R^2)/n. Not surprisingly, similar to Neyman’s (20) analysis of the CRE, unless the residual from the linear projection of the individual causal effect on the covariates is constant, the above sampling variance estimator and the associated confidence interval are both asymptotically conservative, in the sense that the probability limit of the variance estimator is larger than or equal to the actual sampling variance, and the limit of coverage probability of the confidence interval is larger than or equal to (1−α).

Moreover, the sampling variance estimator is smaller than Neyman’s (20) sampling variance estimator for the CRE, and the confidence interval is shorter than Neyman’s (20) confidence interval for the CRE. Therefore, if we conduct ReM in the design stage but analyze data as in the CRE, the consequential sampling variance estimator and confidence intervals will be overly conservative.

The above results are all intuitive, and we present the algebraic details for the proofs of these results in SI Appendix, section A4. Interestingly, as shown in SI Appendix, section A4, we do not need conditions beyond Condition 1 to ensure the asymptotic properties of the sampling variance estimator and the confidence intervals.

Rerandomization with Tiers of Covariates

Mahalanobis Distance with Tiers of Covariates.

When covariates are thought to have different levels of importance for the outcomes, Morgan and Rubin (21) proposed rerandomization using the Mahalanobis distance with differing criteria for different tiers of covariates. We partition the covariates into T tiers indexed by t=1,…,T with decreasing importance, with kt covariates in tier t. Let Xi=(Xi[1],…,Xi[T]), where Xi[t] denotes the covariates in tier t. Define Xi[t¯]=(Xi[1],…,Xi[t]), the covariates in the first t tiers. Following Morgan and Rubin (21), let SX[t−1¯]2 be the finite population covariance matrix of the covariates in the first t−1 tiers and SX[t],X[t−1¯] be the finite population covariance between X[t] and X[t−1¯]. We first apply a block-wise Gram–Schmidt orthogonalization to the covariates to create the orthogonalized covariates. Let Ei[1]=Xi[1], and for 2≤t≤T, letEi[t]=Xi[t]−SX[t],X[t−1¯]SX[t−1¯]2−1 Xi[t−1¯],where Ei[t] is the residual of the projection of the covariates Xi[t] in tier t onto the space spanned by the covariates in previous tiers. Let Ei=(Ei[1],…,Ei[T]). Let τ^E[t] be the difference-in-means vector of Ei[t] between treatment and control groups and SE[t]2 be the finite population covariance matrix of Ei[t]. The Mahalanobis distance in tier t is Mt=(n1n0)/n⋅τ^E[t]′(SE[t]2)−1τ^E[t], and rerandomization using the Mahalanobis distance with tiers of covariates (ReMT) accepts those treatment assignments with Mt≤at, where the ats are predetermined constants (1≤t≤T). If T=1, then ReMT is simply ReM. Let T denote the event that a treatment assignment Z is accepted under ReMT. The theory below extends Morgan and Rubin (21), using the concepts from our Rerandomization Using the Mahalanobis Distance section.

Multiple Correlation Between τ^ and τ^E[t].

Let ρt2 be the squared multiple correlation between τ^ and the difference-in-means vector of the orthogonalized covariates in tier t τ^E[t]: ρt2=Covτ^,τ^E[t]Varτ^E[t]−1Covτ^E[t],τ^/Var(τ^). Define the finite population variance of the projection of Y(z) on E[t] as SY(z)∣E[t]2=SY(z),E[t]SE[t]2−1SE[t],Y(z) for z=0,1, where SY(z),E[t] is the finite population covariance between potential outcomes and orthogonalized covariates in tier t. We can similarly define Sτ∣E[t]2. According to Proposition 1,ρt2=r1−1SY(1)∣E[t]2+r0−1SY(0)∣E[t]2−Sτ∣E[t]2r1−1SY(1)2+r0−1SY(0)2−Sτ2, (1≤t≤T).When the causal effect is additive, ρt2=SY(0)∣E[t]2/SY(0)2 reduces to the squared multiple correlation between Y(0) and E[t]. For descriptive simplicity, we introduce ρT+12=1−∑t=1Tρt2=1−R2 for later discussion.

Asymptotic Sampling Distribution of τ^ Under ReMT.

Intuitively, n(τ^−τ) can be decomposed into (T+1) parts: the part unrelated to covariates and the T projections onto the spaces spanned by the orthogonalized covariates in T tiers. Due to their construction, these (T+1) parts are orthogonal to each other, and the constraint for balance in tier t affects only the tth projection.

As earlier, let ε0∼N(0,1), and extending earlier notation using the subscript t, let Lkt,at∼Dt1∣Dt′Dt≤at, where Dt=(Dt1,…,Dtkt)∼N(0,Ikt) for 1≤t≤T.

Theorem 3.

Under ReMT and Condition 1,n(τ^−τ)∣T ∼̇ VττρT+1⋅ε0+∑t=1Tρt⋅Lkt,at,[6]where (ε0,Lk1,a1,…,LkT,aT) are jointly independent.

In [6], ε0 is the part of n(τ^−τ) that is unrelated to the covariates, and Lkt,at is the part related to the orthogonalized covariates Ei[t] in tier t. According to Proposition 2, the distribution in Theorem 3 is easy to simulate.

Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.

Theorem 3 characterizes the asymptotic behavior of n(τ^−τ) under ReMT, which extends Morgan and Rubin (21) as follows.

First, the asymptotic distribution in [6] is symmetric around 0, implying that τ^ is asymptotically unbiased for τ. Therefore, all observed or unobserved covariates have asymptotically zero difference in means.

Corollary 3.

Under ReMT and Condition 1, Ea{n(τ^−τ)∣T}=0.

The asymptotic sampling covariance of τ^X under ReMT has a complicated but conceptually obvious form, and we give it in SI Appendix, section A3. Below we present only the PRIASV of τ^; the PRIASVs for covariates are special cases of the same corollary because covariates are formally “outcomes” unaffected by the treatment. Recall that vkt,at=P(χkt+22≤at)/P(χkt2≤at).

Corollary 4.

Under ReMT and Condition 1, the asymptotic sampling variance of n(τ^−τ) is Vττ{1−∑t=1T(1−vkt,at)ρt2}, and the PRIASV of n(τ^−τ) is ∑t=1T(1−vkt,at)ρt2.

When the causal effect is additive, ρt2 becomes the squared multiple correlation between E[t] and Y(0). Therefore, Corollary 4 is an asymptotic extension of Morgan and Rubin’s theorem 4.2 (21). When the thresholds ats are close to zero, the asymptotic sampling variance Vττ1−∑t=1T(1−vkt,at)ρt2 reduces to Vττ(1−∑t=1Tρt2)=Vττ(1−R2), which is identical to that of the regression-adjusted estimator under the CRE (18).

We now compare the quantile range under ReMT to that under the CRE. Let νξ(ρ12,ρ22,…,ρT2) be the ξth quantile of ρT+1ε0+∑t=1TρtLkt,at. Although νξ(ρ12,ρ22,…,ρT2) depends also on pat and kt (1≤t≤K), we suppress the dependence to avoid notational clutter. The (1−α) quantile range of the asymptotic distribution of n(τ^−τ) under ReMT isQRα(Vττ,ρ12,…,ρT2)=να/2(ρ12,…,ρT2)Vττ,  ν1−α/2(ρ12,…,ρT2)Vττ.[7]The stronger the squared correlation is between the outcome and the orthogonalized covariates in tier t, the more reduction in quantile range when using ReMT rather than the CRE. The following Theorem 4 is intuitive.

Theorem 4.

Under Condition 1, the (1−α) quantile range of the asymptotic distribution of n(τ^−τ) under ReMT is less than, or equal to, the range under the CRE, and the reduction in length is nondecreasing in ρt2 and nonincreasing in pat and kt, for all 1≤t≤T.

Sampling Variance Estimation and Confidence Interval.

We can estimate Vττ and ρt2 (1≤t≤T) in the same way as in ReM, and we estimate ρT+12 by 1−R^2. In practice, we set ρ^t2 (1≤t≤T) to 0 when it is negative due to sampling variability and standardize their sum to R^2. According to Corollary 4 and [7], we can estimate the sampling variance of τ^ and construct confidence intervals for τ by replacing the unknown quantities with their point estimates. The sampling variance estimator is smaller than Neyman’s (20) sampling variance estimator for the CRE, and the confidence interval is shorter than Neyman’s (20) confidence interval for the CRE; both are asymptotically conservative in general, and only when the residual from the linear projection of the individual causal effect on the covariates is constant, are they asymptotically exact. Therefore, analyzing data from ReMT as if they arose from a CRE, the resulting sampling variance estimator and confidence intervals are overly conservative. These intuitive statements appear to require lengthy proofs, which are relegated to SI Appendix, section A4.

An Education Example with Tiers of Covariates

We illustrate our theory using the data from the Student Achievement and Retention Project (24), a randomized evaluation of academic services and incentives at one of the satellite campuses of a large Canadian university. A treatment group of 150 students was offered an array of support services and substantial cash awards for meeting a target first-year grade-point average (GPA), and a control group of many more (1,006) students received only standard university support services.

To illustrate the benefit of rerandomization, we use the 15 covariates listed in Table 1 and exclude students with missing values, resulting in n1=118 and n0=856. To make the simulation relevant to the real data, we fix unknown parameters based on some simple model fitting: We fit a linear regression of the observed first-year GPA on the treatment indicator, all covariates and their interactions, and use the fitted model to generate all potential outcomes eschewing additivity. Note that the generating models for the potential outcomes are nonlinear in the basic covariates. To make the data-generating process realistic, we simulate eight pseudosets of potential outcomes, using the fitted model with different choices for the variance of the residuals. The error terms for Y(1) and Y(0) are independent, and therefore conditional on the covariates, and the potential outcomes are simulated as uncorrelated, but they have a positive correlation marginally. The final potential outcomes are all truncated to lie on [0,4], mimicking the value of the GPA. We choose different variances of residuals such that the values of R2 for the eight simulated datasets are located approximately evenly within interval [0,0.5]. One choice for the variance of residuals is the one estimated from the fitted linear model, and the corresponding R2 is about 0.23.

View this table:
  • View inline
  • View popup
Table 1.

Three tiers of covariates

Table 1 partitions the 15 covariates into three tiers. We choose at such that P(χkt2≤at)=(0.001)1/3=0.1 for t=1,2,3. We simulate data under ReMT and obtain the confidence intervals based on our asymptotic theory for ReMT and Neyman’s (20) results for the CRE. Fig. 1, Left shows the empirical coverage probabilities of our and Neyman’s (20) confidence intervals, showing that Neyman’s (20) CRE confidence intervals are very conservative.

Fig. 1.
  • Download figure
  • Open in new tab
  • Download powerpoint
Fig. 1.

Eight datasets simulated based on the Student Achievement and Retention Project. Left shows the empirical coverage probabilities of our and Neyman’s (20) 95% confidence intervals under ReMT, and Right shows the percentage reductions of average lengths of confidence intervals and quantile ranges comparing ReMT with a CRE.

To evaluate the performance of ReMT compared with a CRE, we compare the average length of Neyman’s (20) confidence interval under a CRE with the confidence interval under ReMT. From Fig. 1, Right, the percentage reduction in average lengths of the 95% confidence intervals under ReMT compared with Neyman’s (20) under a CRE is nondecreasing in R2. We also compare the empirical 95% quantile range of τ^ under ReMT and a CRE: The percentage reductions in the lengths of quantile ranges are close to the percentage reductions for average lengths of confidence intervals. When R2 is close to that of the real dataset (i.e., 0.23), the percentage increase in the effective sample size, that is, the sample size needed with a CRE for τ^ to achieve the same 95% quantile range under ReMT, is about 24%. When R2 is about twice as large as with the real data (i.e., 0.5), the percentage increase in the effective sample size increases to 80%.

Discussion

Our theory suggests that choosing a small pa will lead to more precise difference in means in general. However, we do not suggest choosing pa to be too small, such as accepting only those assignments with the smallest Mahalanobis distance. The extreme rerandomization choosing an allocation that balances observed covariates as well as possible has an undesirable consequence. Because it is deterministic, randomization distributions are degenerate, rendering randomization inference impossible. Even if we randomize over all allocations satisfying the best allocation, randomization inference has little power due to very few possible allocations. How to choose pa remains an open problem.

Materials and Methods

We did not conduct the experiment, and we are analyzing secondary data without any personal identifying information. As such, this study is exempt from human subjects review. The original experiments underwent human subjects review in Canada (24).

Acknowledgments

P.D. acknowledges support from the National Science Foundation (DMS 1713152). D.B.R. acknowledges support from the National Institute of Allergy and Infectious Diseases/NIH (R01AI102710), National Science Foundation (IIS-1409177), Office of Naval Research (N00014-17-1-2131), and a Google Faculty Fellowship.

Footnotes

  • ↵1To whom correspondence should be addressed. Email: dbrubin{at}mac.com.
  • Author contributions: X.L., P.D., and D.B.R. designed research, performed research, contributed new reagents/analytic tools, analyzed data, and wrote the paper.

  • Reviewers: R.J.T., Stanford University; and C.F.J.W., Georgia Institute of Technology.

  • The authors declare no conflict of interest.

  • This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1808191115/-/DCSupplemental.

Published under the PNAS license.

View Abstract

References

  1. ↵
    1. Fisher RA
    (1925) Statistical Methods for Research Workers (Oliver and Boyd, Edinburgh), 1st Ed.
  2. ↵
    1. Fisher RA
    (1926) The arrangement of field experiments. J Minist Agric G B 33:503–513.
    OpenUrl
  3. ↵
    1. Fisher RA
    (1935) The Design of Experiments (Oliver and Boyd, Edinburgh), 1st Ed.
  4. ↵
    1. Student
    (1938) Comparison between balanced and random arrangements of field plots. Biometrika 29:363–378.
    OpenUrlCrossRef
  5. ↵
    1. Greevy R,
    2. Lu B,
    3. Silber JH,
    4. Rosenbaum P
    (2004) Optimal multivariate matching before randomization. Biostatistics 5:263–275.
    OpenUrlCrossRefPubMed
  6. ↵
    1. Hansen BB,
    2. Bowers J
    (2008) Covariate balance in simple, stratified and clustered comparative studies. Stat Sci 23:219–236.
    OpenUrlCrossRef
  7. ↵
    1. Bruhn M,
    2. McKenzie D
    (2009) In pursuit of balance: Randomization in practice in development field experiments. Am Econ J Appl Econ 1:200–232.
    OpenUrlCrossRef
  8. ↵
    1. Krieger AM,
    2. Azriel D,
    3. Kapelner A
    (2016) Nearly random designs with greatly improved balance. arXiv:1612.02315.
  9. ↵
    1. Banerjee A,
    2. Esther D
    1. Athey S,
    2. Imbens GW
    (2017) The econometrics of randomized experiments. Handbook of Economic Field Experiments, eds Banerjee A, Esther D (North-Holland, Amsterdam), Vol 1, pp 73–140.
    OpenUrl
  10. ↵
    1. Cochran WG,
    2. Cox GM
    (1992) Experimental Designs (Wiley, New York), 2nd Ed.
  11. ↵
    1. Imbens GW,
    2. Rubin DB
    (2015) Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction (Cambridge Univ Press, Cambridge, UK).
  12. ↵
    1. Higgins MJ,
    2. Sävje F,
    3. Sekhon JS
    (2016) Improving massive experiments with threshold blocking. Proc Natl Acad Sci USA 113:7369–7376.
    OpenUrlAbstract/FREE Full Text
  13. ↵
    1. Savage LJ
    (1962) The Foundations of Statistical Inference (Methuen and Co Ltd, London).
  14. ↵
    1. Krishnaiah PR,
    2. Kallianpur G,
    3. Ghosh JK
    1. Cox DR
    (1982) Randomization and concomitant variables in the design of experiments. Statistics and Probability: Essays in Honor of C. R. Rao, eds Krishnaiah PR, Kallianpur G, Ghosh JK (North-Holland, Amsterdam), pp 197–202.
  15. ↵
    1. Cox DR
    (2009) Randomization in the design of experiments. Int Stat Rev 77:415–429.
    OpenUrl
  16. ↵
    1. Rubin DB
    (2008) Comment to W. R. Shadish, M. H. Clark and P. M Steiner. J Am Stat Assoc 103:1350–1353.
    OpenUrlCrossRef
  17. ↵
    1. Morgan KL,
    2. Rubin DB
    (2012) Rerandomization to improve covariate balance in experiments. Ann Stat 40:1263–1282.
    OpenUrlCrossRef
  18. ↵
    1. Lin W
    (2013) Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. Ann Appl Stat 7:295–318.
    OpenUrlCrossRef
  19. ↵
    1. Li X,
    2. Ding P
    (2017) General forms of finite population central limit theorems with applications to causal inference. J Am Stat Assoc 112:1759–1769.
    OpenUrl
  20. ↵
    1. Neyman J
    (1923) On the application of probability theory to agricultural experiments. Essay on principles (with discussion). Section 9 (translated). Reprinted. Stat Sci 5:465–472.
    OpenUrl
  21. ↵
    1. Morgan KL,
    2. Rubin DB
    (2015) Rerandomization to balance tiers of covariates. J Am Stat Assoc 110:1412–1421.
    OpenUrl
  22. ↵
    1. Wintner A
    (1936) On a class of Fourier transforms. Am J Math 58:45–90.
    OpenUrl
  23. ↵
    1. Ding P,
    2. Feller A,
    3. Miratrix L
    (2018) Decomposing treatment effect variation. J Am Stat Assoc, in press.
  24. ↵
    1. Angrist J,
    2. Lang D,
    3. Oreopoulos P
    (2009) Incentives and services for college achievement: Evidence from a randomized trial. Am Econ J Appl Econ 1:136–163.
    OpenUrl
PreviousNext
Back to top
Article Alerts
Email Article

Thank you for your interest in spreading the word on PNAS.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Asymptotic theory of rerandomization in treatment–control experiments
(Your Name) has sent you a message from PNAS
(Your Name) thought you would like to see the PNAS web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Asymptotic theory of rerandomization in treatment–control experiments
Xinran Li, Peng Ding, Donald B. Rubin
Proceedings of the National Academy of Sciences Sep 2018, 115 (37) 9157-9162; DOI: 10.1073/pnas.1808191115

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Request Permissions
Share
Asymptotic theory of rerandomization in treatment–control experiments
Xinran Li, Peng Ding, Donald B. Rubin
Proceedings of the National Academy of Sciences Sep 2018, 115 (37) 9157-9162; DOI: 10.1073/pnas.1808191115
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Mendeley logo Mendeley
Proceedings of the National Academy of Sciences: 115 (37)
Table of Contents

Submit

Sign up for Article Alerts

Article Classifications

  • Physical Sciences
  • Statistics

Jump to section

  • Article
    • Abstract
    • Framework, Notation, and Basic Results
    • Rerandomization Using the Mahalanobis Distance
    • Rerandomization with Tiers of Covariates
    • An Education Example with Tiers of Covariates
    • Discussion
    • Materials and Methods
    • Acknowledgments
    • Footnotes
    • References
  • Figures & SI
  • Info & Metrics
  • PDF

You May Also be Interested in

Abstract depiction of a guitar and musical note
Science & Culture: At the nexus of music and medicine, some see disease treatments
Although the evidence is still limited, a growing body of research suggests music may have beneficial effects for diseases such as Parkinson’s.
Image credit: Shutterstock/agsandrew.
Large piece of gold
News Feature: Tracing gold's cosmic origins
Astronomers thought they’d finally figured out where gold and other heavy elements in the universe came from. In light of recent results, they’re not so sure.
Image credit: Science Source/Tom McHugh.
Dancers in red dresses
Journal Club: Friends appear to share patterns of brain activity
Researchers are still trying to understand what causes this strong correlation between neural and social networks.
Image credit: Shutterstock/Yeongsik Im.
White and blue bird
Hazards of ozone pollution to birds
Amanda Rodewald, Ivan Rudik, and Catherine Kling talk about the hazards of ozone pollution to birds.
Listen
Past PodcastsSubscribe
Goats standing in a pin
Transplantation of sperm-producing stem cells
CRISPR-Cas9 gene editing can improve the effectiveness of spermatogonial stem cell transplantation in mice and livestock, a study finds.
Image credit: Jon M. Oatley.

Similar Articles

Site Logo
Powered by HighWire
  • Submit Manuscript
  • Twitter
  • Facebook
  • RSS Feeds
  • Email Alerts

Articles

  • Current Issue
  • Special Feature Articles – Most Recent
  • List of Issues

PNAS Portals

  • Anthropology
  • Chemistry
  • Classics
  • Front Matter
  • Physics
  • Sustainability Science
  • Teaching Resources

Information

  • Authors
  • Editorial Board
  • Reviewers
  • Librarians
  • Press
  • Site Map
  • PNAS Updates

Feedback    Privacy/Legal

Copyright © 2021 National Academy of Sciences. Online ISSN 1091-6490