Asymptotic theory of rerandomization in treatment–control experiments

Covariate Imbalance and Rerandomization.

Inferring the causal effect of some binary treatment on an outcome Y is of central interest in many studies. We consider an experiment with n units, with n1 assigned to treatment and n0 assigned to control, n=n1+n0. Before conducting the experiment, we collect K covariates with values Xi=(X1i,X2i,…,XKi)′ for the ith unit, which can possibly include transformations of basic covariates and their interactions. Let Zi be the treatment indicator for unit i (Zi=1 if the active treatment level; Zi=0 if the control level) and Z=(Z1,Z2,…,Zn)′ be the treatment assignment vector with n1≡∑i=1nZi and n0≡∑i=1n(1−Zi). In a completely randomized experiment (CRE), n1 and n0 are fixed, and the distribution of Z is such that each possible value, z=(z1,…,zn)′, of Z has probability n1!n0!/n!. The difference-in-means vector of the covariates between treatment and control groups is τ^X=n1−1∑i=1nZiXi−n0−1∑i=1n(1−Zi)Xi. Although on average τ^X has mean zero over all n!/(n1!n0!) randomizations, for any realized value of Z, imbalancedness in covariate distributions between treatment groups often occurs. For 10 uncorrelated covariates, with probability 40%, at least one of the absolute values of the t statistics comparing covariate means will be larger than 1.96, the 0.975 quantile of the standard Gaussian distribution (17).

When covariate imbalance arises in a drawn allocation, it is reasonable to discard that allocation and draw another until some a priori covariate balance criterion is satisfied. This is rerandomization, an intuitive experimental design tool apparently personally advocated by R. A. Fisher (discussion in ref. 16) and formally discussed by Morgan and Rubin (17).

In general, rerandomization entails the following steps: (i) Collect covariate data; (ii) specify a balance criterion to determine whether a randomization is acceptable or not; (iii) randomize the units to treatment and control groups; (iv) if the balance criterion is satisfied, proceed to step v, and otherwise, return to step iii; (v) conduct the experiment using the final randomization obtained in step iv; and (vi) analyze the data, taking into account the rerandomization used in steps ii–iv.

Although the balance criterion in step ii can be general, Morgan and Rubin (17) suggested using the Mahalanobis distance between covariate means in treatment and control groups, and they (21) suggested considering tiers of covariates according to their presumed importance in predicting the outcomes in this experiment. We discuss these two types of rerandomization in detail and apposite statistical inference after these rerandomizations as implied by step vi. We then extend the theory to general rerandomizations in SI Appendix, section A1.

Potential Outcomes and Definitions of Finite Population Quantities.

We use the potential outcomes framework to define causal effects and let Yi(1) and Yi(0) denote the potential outcomes of unit i under active treatment and control, respectively. On the difference scale, the individual causal effect for unit i is τi=Yi(1)−Yi(0), and the average causal effect for the finite population of n units is τ=∑i=1nτi/n. Let Ȳ(z)=∑i=1nYi(z)/n be the finite population average of potential outcomes under treatment arm z and X¯=∑i=1nXi/n be the finite population average of covariates. Let SY(z)2=∑i=1n{Yi(z)−Ȳ(z)}2/(n−1) be the finite population variance (with divisor n−1) of the potential outcomes under treatment arm z, Sτ2=∑i=1n(τi−τ)2/(n−1) be the finite population variance of the individual causal effects, SY(z),X=SX,Y(z)′=∑i=1n{Yi(z)−Ȳ(z)}(Xi−X¯)′/(n−1) be the finite population covariance between potential outcomes and covariates, and SX2=∑i=1n(Xi−X¯)(Xi−X¯)′/(n−1) be the finite population covariance matrix of covariates. These fixed quantities depend on n implicitly, but do not depend on the randomization or rerandomization scheme.

Repeated Sampling Inference in a CRE.

The observed outcome for unit i is Yi=ZiYi(1)+(1−Zi)Yi(0), a function of treatment assignment and potential outcomes. In a CRE, Neyman (20) showed that, for estimating τ, the difference-in-means estimator τ^=n1−1∑i=1nZiYi−n0−1∑i=1n(1−Zi)Yi is unbiased (the expectation of τ^ over all possible randomizations is τ) and obtained its sampling variance over all randomizations for constructing a large-sample confidence interval for τ. However, Neyman’s (20) interval is not accurate if rerandomization is used.

Let r1=n1/n and r0=n0/n be the proportions of units receiving treatment and control. According to the finite population central limit theorem (19), under some regularity conditions, the large-n sampling distribution, over all randomizations, of n(τ^−τ,τ^X′) is Gaussian with mean zero and covariance matrixV=VττVτxVxτVxx =r1−1SY(1)2+r0−1SY(0)2−Sτ2r1−1SY(1),X+r0−1SY(0), Xr1−1SX,Y(1)+r0−1SX,Y(0)(r1r0)−1SX2.We are conducting randomization-based inference, where all of the covariates and potential outcomes are fixed numbers, and randomness comes solely from the treatment assignment. We embed n units into an infinite sequence of finite populations with increasing sizes, and a sufficient condition for the asymptotic Gaussianity of n(τ^−τ,τ^X′) is as follows (19).

Mahalanobis Distance.

The Mahalanobis distance between the covariate means in treatment and control groups isM=τ^X′{Var(τ^X)}−1τ^X=nτ^X′ Vxx−1nτ^X,where Vxx=(r1r0)−1SX2 is a fixed and known K×K matrix in our setting. A rerandomization scheme proposed by Morgan and Rubin (17) accepts only those randomizations with the Mahalanobis distance less than or equal to a, a prespecified threshold. Let M denote the event that a treatment assignment Z is accepted; that is, M≤a. Below we use rerandomization using the Mahalanobis distance (ReM) to denote rerandomization using this criterion, which, as a design, depends on both the covariates and the threshold a.

When we allow transformations and interactions of X, ReM can incorporate a wide class of rerandomization schemes. For small sample sizes, there may not exist any randomization satisfying some balance criterion. However, according to the finite population central limit theorem (19), the acceptance probability of a randomization is asymptotically pa=P(χK2≤a). Therefore, for relatively large sample sizes, there usually exist many randomizations satisfying the balance criterion. In practice, we want to choose the asymptotic acceptance probability to be small, e.g., pa=0.001. We comment on this issue in Discussion.

Multiple Correlation Between τ^ and τ^X.

The sampling distribution of τ^ under ReM depends on the squared multiple correlation between τ^ and τ^X under the CRE, which is also the proportion of the variance of τ^ explained by τ^X in linear projection: R2=Cov(τ^,τ^X)Var(τ^X)−1Cov(τ^X,τ^)/Var(τ^)=VτxVxx−1Vxτ/Vττ. Define the variance of the linear projection of Y(z) on X as SY(z)∣X2=SY(z), XSX2−1SX,Y(z) for z=0,1. We similarly define Sτ∣X2, the variance of the linear projection of τ on X.

Proposition 1.

R2 can be expressed in terms of the variances of the potential outcomes and of their projections on X:R2=r1−1SY(1)∣X2+r0−1SY(0)∣X2−Sτ∣X2r1−1SY(1)2+r0−1SY(0)2−Sτ2.

When the causal effect is additive, Sτ2=0, Sτ,X=0, and SY(1),X=SY(0),X, and then R2=SY(0)∣X2/SY(0)2 is the squared multiple correlation between X and Y(0).

Asymptotic Sampling Distribution of τ^ Under ReM.

Simply stated, n(τ^−τ) has two parts: the part unrelated to the covariates, which we call ε0, and is thus unaffected by rerandomization, and the other part related to the covariates, which we call LK,a, and is thus affected by rerandomization. Therefore, the asymptotic distribution of τ^ is a linear combination of two independent random variables: ε0∼N(0,1) is a standard Gaussian random variable, and LK,a∼D1∣D′D≤a, where D=(D1,…,DK)′∼N(0,IK).

Theorem 1.

Under ReM and Condition 1,n(τ^−τ)∣M∼̇ Vττ1−R2⋅ε0+R2⋅LK,a,[1]where ε0 is independent of LK,a.

The coefficients of the linear combination are functions of R2, which measures the association between the potential outcomes and the covariates. When R2=0, the right-hand side of [1] becomes a Gaussian random variable, the same as the asymptotic distribution of n(τ^−τ) in Repeated Sampling Inference in a CRE. Importantly, the definition of R2 is based on linear projections instead of linear models of the potential outcomes. Our asymptotic theory is based on the distribution of the randomization without imposing any modeling assumptions on the potential outcomes.

Representation of the Asymptotic Distribution Under ReM.

The asymptotic distribution in [1] involves a random variable LK,a that does not appear in standard statistical problems. The spherical symmetry of the standard Gaussian vector allows us to represent LK,a using some known distributions, which allows for easy simulation of LK,a.

Let χK,a2∼χK2∣χK2≤a be a truncated χ2 random variable, UK be the first coordinate of the uniform random vector over the (K−1)-dimensional unit sphere, S be a random sign taking ±1 with probability 1/2, and βK∼Beta1/2,(K−1)/2 be a Beta random variable degenerating to point mass at 1 when K=1.

Proposition 2.

LK,a can be represented asLK,a∼D1∣D′D≤a∼χK,aUK∼χK,aSβK,[2]where (χK,a,UK) are mutually independent, and (χK,a,S,βK) are jointly independent. LK,a is symmetric and unimodal around zero, with Var(LK,a)=vK,a=P(χK+22≤a)/P(χK2≤a)<1.

Because both ε0 and LK,a are symmetric and both are unimodal at zero, their linear combination is also symmetric and unimodal at zero according to Wintner’s (22) theorem. The same is true for the asymptotic distribution of n(τ^−τ) in [1]. Moreover, the unimodal property plays an important role in the conservativeness of confidence intervals, discussed shortly for ReM. The representation in [2] allows for easy simulation of LK,a, as well as the asymptotic distribution of n(τ^−τ) in [1], which is relevant for statistical inference discussed later.

In SI Appendix, section A2, we give more detailed explanations regarding the geometry and the shape of the asymptotic distribution in [1].

Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.

Theorem 1 characterizes the asymptotic behavior of τ^ over ReM, which immediately implies the following conclusions.

First, the asymptotic distribution in [1] is symmetric around 0, implying that τ^ is asymptotically unbiased for τ. Let Ea(⋅) denote the expectation of the asymptotic sampling distribution of a sequence of random vectors.

Corollary 1.

Under ReM and Condition 1, Ean(τ^−τ)|M=0.

Morgan and Rubin (17) gave a counterexample showing that, in an experiment with unequal treatment group sizes, τ^ can be biased for τ over ReM. Our result confirms a conjecture in ref. 21 that the bias is often small with large samples. Corollary 1 extends their theorem 2.1 (17) and ensures the asymptotic unbiasedness of τ^ for experiments with any ratio of n1/n0.

Covariates, whether observed or unobserved, are variables unaffected by the treatments. Therefore, the average causal effect on any covariate is 0, and Corollary 1 implies that any covariate asymptotically has the same means under treatment and control.

Furthermore, from Proposition 2 and Theorem 1, we can calculate the asymptotic sampling variances of τ^X and τ^ and the percentage reductions in asymptotic sampling variances (PRIASV) under ReM compared with the CRE. Recalling that vK,a=P(χK+22≤a)/P(χK2≤a), we summarize the results in Corollary 2.

Corollary 2.

Under ReM and Condition 1, the asymptotic sampling covariance of nτ^X is vK,aVxx, and the PRIASV of any component of nτ^X is 1−vK,a. The asymptotic sampling variance of n(τ^−τ) is Vττ1−(1−vK,a)R2, and the PRIASV of n(τ^−τ) is (1−vK,a)R2.

Rigorously, the asymptotic sampling covariance and variance of τ^X and τ^ should be the limits of vK,aVxx and Vττ{1−(1−vK,a)R2} in the sequence of finite populations. However, for descriptive convenience, we omit these limit signs when discussing the expectation and covariance of asymptotic sampling distributions. When a is close to 0, that is, when the asymptotic acceptance probability is small, the asymptotic sampling variance Vττ1−(1−vK,a)R2 reduces to Vττ(1−R2), which is identical to the asymptotic sampling variance of the regression-adjusted estimator under the CRE (18). Therefore, rerandomization accomplishes covariate adjustment in the design stage, whereas regression accomplishes covariate adjustment in the analysis stage.

When the causal effect is additive, R2 equals the finite population squared multiple correlation between X and Y(0). Therefore, Corollary 2 is an asymptotic extension of theorem 3.2 in Morgan and Rubin (17).

Under ReM, in addition to the sampling variance reduction result concerning τ^ in Corollary 2, we consider the reduction in the length of the (1−α) quantile range of τ^ compared with that under the CRE. We choose the length of the (1−α) quantile range, because of its connection to constructing confidence intervals as discussed shortly.

Let zξ be the ξth quantile of a standard Gaussian distribution. Let νξ(R2,pa,K) be the ξth quantile of 1−R2⋅ε0+R2⋅LK,a, with νξ(0,pa,K)=zξ. Because pa and K are usually known by design, we write νξ(R2,pa,K) as νξ(R2) for notational simplicity. Under ReM, the (1−α) quantile range of the asymptotic distribution of n(τ^−τ) isQRα(Vττ,R2)=να/2(R2)Vττ,  ν1−α/2(R2)Vττ,[3]and the corresponding quantile range under the CRE isQRα(Vττ,0)=zα/2Vττ,  z1−α/2Vττ.[4]

Theorem 2.

Under Condition 1, the length of the (1−α) quantile range of the asymptotic sampling distribution of n(τ^−τ) under ReM is less than or equal to that under the CRE, with the difference nondecreasing in R2 and nonincreasing in pa and K.

Sampling Variance Estimation and Confidence Intervals.

Asymptotic sampling variance and quantile ranges for τ^ depend on Vττ and R2, which are determined by the covariances among potential outcomes and covariates. To obtain a sampling variance estimator and to construct an asymptotic confidence interval for τ, we need to estimate these variances and covariances. Let sY(z)2, sY(z)∣X2, and sY(z),X be the sample variance of Y, sample variance of linear projection of Y on X, and sample covariance between Y and X in treatment arm z. We show in SI Appendix, section A4 that under ReM they are consistent for their population analogues. Therefore, we can then estimate Sτ∣X2 bysτ∣X2=(sY(1), X−sY(0), X)(SX2)−1(sX,Y(1)−sX,Y(0))and Vττ by (23)V^ττ=r1−1sY(1)2+r0−1sY(0)2−sτ∣X2.We can then estimate R2 byR^2=V^ττ−1r1−1sY(1)∣X2+r0−1sY(0)∣X2−sτ∣X2.[5]We set R^2 to be 0 if the estimator in [5] is negative.

According to Corollary 2, we can estimate the asymptotic sampling variance of τ^ by V^ττ{1−(1−vK,a)R^2}/n, and according to [3], we can construct a large sample (1−α) confidence interval for τ using τ^−QRα(V^ττ,R^2)/n. Not surprisingly, similar to Neyman’s (20) analysis of the CRE, unless the residual from the linear projection of the individual causal effect on the covariates is constant, the above sampling variance estimator and the associated confidence interval are both asymptotically conservative, in the sense that the probability limit of the variance estimator is larger than or equal to the actual sampling variance, and the limit of coverage probability of the confidence interval is larger than or equal to (1−α).

Moreover, the sampling variance estimator is smaller than Neyman’s (20) sampling variance estimator for the CRE, and the confidence interval is shorter than Neyman’s (20) confidence interval for the CRE. Therefore, if we conduct ReM in the design stage but analyze data as in the CRE, the consequential sampling variance estimator and confidence intervals will be overly conservative.

The above results are all intuitive, and we present the algebraic details for the proofs of these results in SI Appendix, section A4. Interestingly, as shown in SI Appendix, section A4, we do not need conditions beyond Condition 1 to ensure the asymptotic properties of the sampling variance estimator and the confidence intervals.

Mahalanobis Distance with Tiers of Covariates.

When covariates are thought to have different levels of importance for the outcomes, Morgan and Rubin (21) proposed rerandomization using the Mahalanobis distance with differing criteria for different tiers of covariates. We partition the covariates into T tiers indexed by t=1,…,T with decreasing importance, with kt covariates in tier t. Let Xi=(Xi[1],…,Xi[T]), where Xi[t] denotes the covariates in tier t. Define Xi[t¯]=(Xi[1],…,Xi[t]), the covariates in the first t tiers. Following Morgan and Rubin (21), let SX[t−1¯]2 be the finite population covariance matrix of the covariates in the first t−1 tiers and SX[t],X[t−1¯] be the finite population covariance between X[t] and X[t−1¯]. We first apply a block-wise Gram–Schmidt orthogonalization to the covariates to create the orthogonalized covariates. Let Ei[1]=Xi[1], and for 2≤t≤T, letEi[t]=Xi[t]−SX[t],X[t−1¯]SX[t−1¯]2−1 Xi[t−1¯],where Ei[t] is the residual of the projection of the covariates Xi[t] in tier t onto the space spanned by the covariates in previous tiers. Let Ei=(Ei[1],…,Ei[T]). Let τ^E[t] be the difference-in-means vector of Ei[t] between treatment and control groups and SE[t]2 be the finite population covariance matrix of Ei[t]. The Mahalanobis distance in tier t is Mt=(n1n0)/n⋅τ^E[t]′(SE[t]2)−1τ^E[t], and rerandomization using the Mahalanobis distance with tiers of covariates (ReMT) accepts those treatment assignments with Mt≤at, where the ats are predetermined constants (1≤t≤T). If T=1, then ReMT is simply ReM. Let T denote the event that a treatment assignment Z is accepted under ReMT. The theory below extends Morgan and Rubin (21), using the concepts from our Rerandomization Using the Mahalanobis Distance section.

Multiple Correlation Between τ^ and τ^E[t].

Let ρt2 be the squared multiple correlation between τ^ and the difference-in-means vector of the orthogonalized covariates in tier t τ^E[t]: ρt2=Covτ^,τ^E[t]Varτ^E[t]−1Covτ^E[t],τ^/Var(τ^). Define the finite population variance of the projection of Y(z) on E[t] as SY(z)∣E[t]2=SY(z),E[t]SE[t]2−1SE[t],Y(z) for z=0,1, where SY(z),E[t] is the finite population covariance between potential outcomes and orthogonalized covariates in tier t. We can similarly define Sτ∣E[t]2. According to Proposition 1,ρt2=r1−1SY(1)∣E[t]2+r0−1SY(0)∣E[t]2−Sτ∣E[t]2r1−1SY(1)2+r0−1SY(0)2−Sτ2, (1≤t≤T).When the causal effect is additive, ρt2=SY(0)∣E[t]2/SY(0)2 reduces to the squared multiple correlation between Y(0) and E[t]. For descriptive simplicity, we introduce ρT+12=1−∑t=1Tρt2=1−R2 for later discussion.

Asymptotic Sampling Distribution of τ^ Under ReMT.

Intuitively, n(τ^−τ) can be decomposed into (T+1) parts: the part unrelated to covariates and the T projections onto the spaces spanned by the orthogonalized covariates in T tiers. Due to their construction, these (T+1) parts are orthogonal to each other, and the constraint for balance in tier t affects only the tth projection.

As earlier, let ε0∼N(0,1), and extending earlier notation using the subscript t, let Lkt,at∼Dt1∣Dt′Dt≤at, where Dt=(Dt1,…,Dtkt)∼N(0,Ikt) for 1≤t≤T.

Theorem 3.

Under ReMT and Condition 1,n(τ^−τ)∣T ∼̇ VττρT+1⋅ε0+∑t=1Tρt⋅Lkt,at,[6]where (ε0,Lk1,a1,…,LkT,aT) are jointly independent.

In [6], ε0 is the part of n(τ^−τ) that is unrelated to the covariates, and Lkt,at is the part related to the orthogonalized covariates Ei[t] in tier t. According to Proposition 2, the distribution in Theorem 3 is easy to simulate.

Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.

Theorem 3 characterizes the asymptotic behavior of n(τ^−τ) under ReMT, which extends Morgan and Rubin (21) as follows.

First, the asymptotic distribution in [6] is symmetric around 0, implying that τ^ is asymptotically unbiased for τ. Therefore, all observed or unobserved covariates have asymptotically zero difference in means.

Corollary 3.

Under ReMT and Condition 1, Ea{n(τ^−τ)∣T}=0.

The asymptotic sampling covariance of τ^X under ReMT has a complicated but conceptually obvious form, and we give it in SI Appendix, section A3. Below we present only the PRIASV of τ^; the PRIASVs for covariates are special cases of the same corollary because covariates are formally “outcomes” unaffected by the treatment. Recall that vkt,at=P(χkt+22≤at)/P(χkt2≤at).

Corollary 4.

Under ReMT and Condition 1, the asymptotic sampling variance of n(τ^−τ) is Vττ{1−∑t=1T(1−vkt,at)ρt2}, and the PRIASV of n(τ^−τ) is ∑t=1T(1−vkt,at)ρt2.

When the causal effect is additive, ρt2 becomes the squared multiple correlation between E[t] and Y(0). Therefore, Corollary 4 is an asymptotic extension of Morgan and Rubin’s theorem 4.2 (21). When the thresholds ats are close to zero, the asymptotic sampling variance Vττ1−∑t=1T(1−vkt,at)ρt2 reduces to Vττ(1−∑t=1Tρt2)=Vττ(1−R2), which is identical to that of the regression-adjusted estimator under the CRE (18).

We now compare the quantile range under ReMT to that under the CRE. Let νξ(ρ12,ρ22,…,ρT2) be the ξth quantile of ρT+1ε0+∑t=1TρtLkt,at. Although νξ(ρ12,ρ22,…,ρT2) depends also on pat and kt (1≤t≤K), we suppress the dependence to avoid notational clutter. The (1−α) quantile range of the asymptotic distribution of n(τ^−τ) under ReMT isQRα(Vττ,ρ12,…,ρT2)=να/2(ρ12,…,ρT2)Vττ,  ν1−α/2(ρ12,…,ρT2)Vττ.[7]The stronger the squared correlation is between the outcome and the orthogonalized covariates in tier t, the more reduction in quantile range when using ReMT rather than the CRE. The following Theorem 4 is intuitive.

Theorem 4.

Under Condition 1, the (1−α) quantile range of the asymptotic distribution of n(τ^−τ) under ReMT is less than, or equal to, the range under the CRE, and the reduction in length is nondecreasing in ρt2 and nonincreasing in pat and kt, for all 1≤t≤T.

Sampling Variance Estimation and Confidence Interval.

We can estimate Vττ and ρt2 (1≤t≤T) in the same way as in ReM, and we estimate ρT+12 by 1−R^2. In practice, we set ρ^t2 (1≤t≤T) to 0 when it is negative due to sampling variability and standardize their sum to R^2. According to Corollary 4 and [7], we can estimate the sampling variance of τ^ and construct confidence intervals for τ by replacing the unknown quantities with their point estimates. The sampling variance estimator is smaller than Neyman’s (20) sampling variance estimator for the CRE, and the confidence interval is shorter than Neyman’s (20) confidence interval for the CRE; both are asymptotically conservative in general, and only when the residual from the linear projection of the individual causal effect on the covariates is constant, are they asymptotically exact. Therefore, analyzing data from ReMT as if they arose from a CRE, the resulting sampling variance estimator and confidence intervals are overly conservative. These intuitive statements appear to require lengthy proofs, which are relegated to SI Appendix, section A4.