New Research In
Physical Sciences
Social Sciences
Featured Portals
Articles by Topic
Biological Sciences
Featured Portals
Articles by Topic
- Agricultural Sciences
- Anthropology
- Applied Biological Sciences
- Biochemistry
- Biophysics and Computational Biology
- Cell Biology
- Developmental Biology
- Ecology
- Environmental Sciences
- Evolution
- Genetics
- Immunology and Inflammation
- Medical Sciences
- Microbiology
- Neuroscience
- Pharmacology
- Physiology
- Plant Biology
- Population Biology
- Psychological and Cognitive Sciences
- Sustainability Science
- Systems Biology
Asymptotic theory of rerandomization in treatment–control experiments
Contributed by Donald B. Rubin, June 29, 2018 (sent for review May 17, 2018; reviewed by Robert J. Tibshirani and C. F. Jeff Wu)

Significance
Rerandomization refers to experimental designs that enforce covariate balance. This paper studies the asymptotic properties of the difference-in-means estimator under rerandomization, based on the randomness of the treatment assignment without imposing any parametric modeling assumptions on the covariates or outcome. The non-Gaussian asymptotic distribution allows for constructing large-sample confidence intervals for the average treatment effect and demonstrates the advantages of rerandomization over complete randomization.
Abstract
Although complete randomization ensures covariate balance on average, the chance of observing significant differences between treatment and control covariate distributions increases with many covariates. Rerandomization discards randomizations that do not satisfy a predetermined covariate balance criterion, generally resulting in better covariate balance and more precise estimates of causal effects. Previous theory has derived finite sample theory for rerandomization under the assumptions of equal treatment group sizes, Gaussian covariate and outcome distributions, or additive causal effects, but not for the general sampling distribution of the difference-in-means estimator for the average causal effect. We develop asymptotic theory for rerandomization without these assumptions, which reveals a non-Gaussian asymptotic distribution for this estimator, specifically a linear combination of a Gaussian random variable and truncated Gaussian random variables. This distribution follows because rerandomization affects only the projection of potential outcomes onto the covariate space but does not affect the corresponding orthogonal residuals. We demonstrate that, compared with complete randomization, rerandomization reduces the asymptotic quantile ranges of the difference-in-means estimator. Moreover, our work constructs accurate large-sample confidence intervals for the average causal effect.
Ever since Fisher’s (1⇓–3) seminal work, randomized experiments have become the “gold standard” for drawing causal inferences. Complete randomization balances the covariate distributions between treatment groups in expectation, thereby ensuring the existence of unbiased estimators of average causal effects. Covariate imbalance, however, often occurs in specific randomized experiments, as recognized by Fisher (2) and later researchers (e.g., refs. 4⇓⇓⇓⇓–9). The standard approach advocated by Fisher (3), stratification or blocking, ensures balance with a few discrete covariates (e.g., refs. 10⇓–12).
When a randomized allocation is unbalanced, it is reasonable to discard that allocation and redraw another one until a certain predetermined covariate balance criterion is satisfied. This is rerandomization, an experimental design hinted at by Fisher (cf. ref. 13, p. 88) and Cox (14, 15) and formally proposed by Rubin (16) and Morgan and Rubin (17). Morgan and Rubin (17) showed that the difference-in-means estimator is generally unbiased for the average causal effect under rerandomization with equal-sized treatment groups and obtained the sampling variance of this estimator under additional assumptions of Gaussian covariate and outcome distributions and additive causal effects. When rerandomization is applied but these assumptions do not hold, statistical inference becomes more challenging, because the theory that is justified by the central limit theorem under complete randomization (18, 19) no longer generally holds. Some applied researchers believe that “the only analysis that we can be completely confident in is a permutation test or rerandomization test” (ref. 7, p. 219). However, permutation tests based on randomization require sharp null hypotheses that imply all missing potential outcomes are known.
Analogous to the repeated sampling properties for complete randomization (11, 20), we evaluate the repeated sampling properties of the difference-in-means estimator when rerandomization is used, where all potential outcomes and covariates are regarded as fixed quantities and all randomness arises solely from the random treatment assignments. The geometry of rerandomization reveals non-Gaussian asymptotic distributions, which serve as the foundation for constructing large-sample confidence intervals for average causal effects. Furthermore, we compare the lengths of quantile ranges of the asymptotic distributions of the difference-in-means estimator under rerandomization and complete randomization, extending Morgan and Rubin’s (17, 21) comparison of their sampling variances.
Framework, Notation, and Basic Results
Covariate Imbalance and Rerandomization.
Inferring the causal effect of some binary treatment on an outcome Y is of central interest in many studies. We consider an experiment with n units, with
When covariate imbalance arises in a drawn allocation, it is reasonable to discard that allocation and draw another until some a priori covariate balance criterion is satisfied. This is rerandomization, an intuitive experimental design tool apparently personally advocated by R. A. Fisher (discussion in ref. 16) and formally discussed by Morgan and Rubin (17).
In general, rerandomization entails the following steps: (i) Collect covariate data; (ii) specify a balance criterion to determine whether a randomization is acceptable or not; (iii) randomize the units to treatment and control groups; (iv) if the balance criterion is satisfied, proceed to step v, and otherwise, return to step iii; (v) conduct the experiment using the final randomization obtained in step iv; and (vi) analyze the data, taking into account the rerandomization used in steps ii–iv.
Although the balance criterion in step ii can be general, Morgan and Rubin (17) suggested using the Mahalanobis distance between covariate means in treatment and control groups, and they (21) suggested considering tiers of covariates according to their presumed importance in predicting the outcomes in this experiment. We discuss these two types of rerandomization in detail and apposite statistical inference after these rerandomizations as implied by step vi. We then extend the theory to general rerandomizations in SI Appendix, section A1.
Potential Outcomes and Definitions of Finite Population Quantities.
We use the potential outcomes framework to define causal effects and let
Repeated Sampling Inference in a CRE.
The observed outcome for unit i is
Let
Condition 1:
As
We introduce the notation
Rerandomization Using the Mahalanobis Distance
Mahalanobis Distance.
The Mahalanobis distance between the covariate means in treatment and control groups is
When we allow transformations and interactions of X, ReM can incorporate a wide class of rerandomization schemes. For small sample sizes, there may not exist any randomization satisfying some balance criterion. However, according to the finite population central limit theorem (19), the acceptance probability of a randomization is asymptotically
Multiple Correlation Between τ ^ and τ ^ X .
The sampling distribution of
Proposition 1.
When the causal effect is additive,
Asymptotic Sampling Distribution of τ ^ Under ReM.
Simply stated,
Theorem 1.
Under ReM and Condition 1,
The coefficients of the linear combination are functions of
Representation of the Asymptotic Distribution Under ReM.
The asymptotic distribution in [1] involves a random variable
Let
Proposition 2.
Because both
In SI Appendix, section A2, we give more detailed explanations regarding the geometry and the shape of the asymptotic distribution in [1].
Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.
Theorem 1 characterizes the asymptotic behavior of
First, the asymptotic distribution in [1] is symmetric around 0, implying that
Corollary 1.
Under ReM and Condition 1,
Morgan and Rubin (17) gave a counterexample showing that, in an experiment with unequal treatment group sizes,
Covariates, whether observed or unobserved, are variables unaffected by the treatments. Therefore, the average causal effect on any covariate is 0, and Corollary 1 implies that any covariate asymptotically has the same means under treatment and control.
Furthermore, from Proposition 2 and Theorem 1, we can calculate the asymptotic sampling variances of
Corollary 2.
Under ReM and Condition 1, the asymptotic sampling covariance of
Rigorously, the asymptotic sampling covariance and variance of
When the causal effect is additive,
Under ReM, in addition to the sampling variance reduction result concerning
Let
Theorem 2.
Under Condition 1, the length of the
Sampling Variance Estimation and Confidence Intervals.
Asymptotic sampling variance and quantile ranges for
According to Corollary 2, we can estimate the asymptotic sampling variance of
Moreover, the sampling variance estimator is smaller than Neyman’s (20) sampling variance estimator for the CRE, and the confidence interval is shorter than Neyman’s (20) confidence interval for the CRE. Therefore, if we conduct ReM in the design stage but analyze data as in the CRE, the consequential sampling variance estimator and confidence intervals will be overly conservative.
The above results are all intuitive, and we present the algebraic details for the proofs of these results in SI Appendix, section A4. Interestingly, as shown in SI Appendix, section A4, we do not need conditions beyond Condition 1 to ensure the asymptotic properties of the sampling variance estimator and the confidence intervals.
Rerandomization with Tiers of Covariates
Mahalanobis Distance with Tiers of Covariates.
When covariates are thought to have different levels of importance for the outcomes, Morgan and Rubin (21) proposed rerandomization using the Mahalanobis distance with differing criteria for different tiers of covariates. We partition the covariates into T tiers indexed by
Multiple Correlation Between τ ^ and τ ^ E [ t ] .
Let
Asymptotic Sampling Distribution of τ ^ Under ReMT.
Intuitively,
As earlier, let
Theorem 3.
Under ReMT and Condition 1,
In [6],
Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.
Theorem 3 characterizes the asymptotic behavior of
First, the asymptotic distribution in [6] is symmetric around 0, implying that
Corollary 3.
Under ReMT and Condition 1,
The asymptotic sampling covariance of
Corollary 4.
Under ReMT and Condition 1, the asymptotic sampling variance of
When the causal effect is additive,
We now compare the quantile range under ReMT to that under the CRE. Let
Theorem 4.
Under Condition 1, the
Sampling Variance Estimation and Confidence Interval.
We can estimate
An Education Example with Tiers of Covariates
We illustrate our theory using the data from the Student Achievement and Retention Project (24), a randomized evaluation of academic services and incentives at one of the satellite campuses of a large Canadian university. A treatment group of 150 students was offered an array of support services and substantial cash awards for meeting a target first-year grade-point average (GPA), and a control group of many more (1,006) students received only standard university support services.
To illustrate the benefit of rerandomization, we use the 15 covariates listed in Table 1 and exclude students with missing values, resulting in
Three tiers of covariates
Table 1 partitions the 15 covariates into three tiers. We choose
Eight datasets simulated based on the Student Achievement and Retention Project. Left shows the empirical coverage probabilities of our and Neyman’s (20)
To evaluate the performance of ReMT compared with a CRE, we compare the average length of Neyman’s (20) confidence interval under a CRE with the confidence interval under ReMT. From Fig. 1, Right, the percentage reduction in average lengths of the 95% confidence intervals under ReMT compared with Neyman’s (20) under a CRE is nondecreasing in
Discussion
Our theory suggests that choosing a small
Materials and Methods
We did not conduct the experiment, and we are analyzing secondary data without any personal identifying information. As such, this study is exempt from human subjects review. The original experiments underwent human subjects review in Canada (24).
Acknowledgments
P.D. acknowledges support from the National Science Foundation (DMS 1713152). D.B.R. acknowledges support from the National Institute of Allergy and Infectious Diseases/NIH (R01AI102710), National Science Foundation (IIS-1409177), Office of Naval Research (N00014-17-1-2131), and a Google Faculty Fellowship.
Footnotes
- ↵1To whom correspondence should be addressed. Email: dbrubin{at}mac.com.
Author contributions: X.L., P.D., and D.B.R. designed research, performed research, contributed new reagents/analytic tools, analyzed data, and wrote the paper.
Reviewers: R.J.T., Stanford University; and C.F.J.W., Georgia Institute of Technology.
The authors declare no conflict of interest.
This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1808191115/-/DCSupplemental.
Published under the PNAS license.
References
- ↵
- Fisher RA
- ↵
- Fisher RA
- ↵
- Fisher RA
- ↵
- ↵
- ↵
- ↵
- ↵
- Krieger AM,
- Azriel D,
- Kapelner A
- ↵
- Banerjee A,
- Esther D
- Athey S,
- Imbens GW
- ↵
- Cochran WG,
- Cox GM
- ↵
- Imbens GW,
- Rubin DB
- ↵
- Higgins MJ,
- Sävje F,
- Sekhon JS
- ↵
- Savage LJ
- ↵
- Krishnaiah PR,
- Kallianpur G,
- Ghosh JK
- Cox DR
- ↵
- Cox DR
- ↵
- ↵
- ↵
- ↵
- Li X,
- Ding P
- ↵
- Neyman J
- ↵
- Morgan KL,
- Rubin DB
- ↵
- Wintner A
- ↵
- Ding P,
- Feller A,
- Miratrix L
- ↵
- Angrist J,
- Lang D,
- Oreopoulos P
Citation Manager Formats
Sign up for Article Alerts
Article Classifications
- Physical Sciences
- Statistics