## New Research In

### Physical Sciences

### Social Sciences

#### Featured Portals

#### Articles by Topic

### Biological Sciences

#### Featured Portals

#### Articles by Topic

- Agricultural Sciences
- Anthropology
- Applied Biological Sciences
- Biochemistry
- Biophysics and Computational Biology
- Cell Biology
- Developmental Biology
- Ecology
- Environmental Sciences
- Evolution
- Genetics
- Immunology and Inflammation
- Medical Sciences
- Microbiology
- Neuroscience
- Pharmacology
- Physiology
- Plant Biology
- Population Biology
- Psychological and Cognitive Sciences
- Sustainability Science
- Systems Biology

# Asymptotic theory of rerandomization in treatment–control experiments

Contributed by Donald B. Rubin, June 29, 2018 (sent for review May 17, 2018; reviewed by Robert J. Tibshirani and C. F. Jeff Wu)

## Significance

Rerandomization refers to experimental designs that enforce covariate balance. This paper studies the asymptotic properties of the difference-in-means estimator under rerandomization, based on the randomness of the treatment assignment without imposing any parametric modeling assumptions on the covariates or outcome. The non-Gaussian asymptotic distribution allows for constructing large-sample confidence intervals for the average treatment effect and demonstrates the advantages of rerandomization over complete randomization.

## Abstract

Although complete randomization ensures covariate balance on average, the chance of observing significant differences between treatment and control covariate distributions increases with many covariates. Rerandomization discards randomizations that do not satisfy a predetermined covariate balance criterion, generally resulting in better covariate balance and more precise estimates of causal effects. Previous theory has derived finite sample theory for rerandomization under the assumptions of equal treatment group sizes, Gaussian covariate and outcome distributions, or additive causal effects, but not for the general sampling distribution of the difference-in-means estimator for the average causal effect. We develop asymptotic theory for rerandomization without these assumptions, which reveals a non-Gaussian asymptotic distribution for this estimator, specifically a linear combination of a Gaussian random variable and truncated Gaussian random variables. This distribution follows because rerandomization affects only the projection of potential outcomes onto the covariate space but does not affect the corresponding orthogonal residuals. We demonstrate that, compared with complete randomization, rerandomization reduces the asymptotic quantile ranges of the difference-in-means estimator. Moreover, our work constructs accurate large-sample confidence intervals for the average causal effect.

Ever since Fisher’s (1⇓–3) seminal work, randomized experiments have become the “gold standard” for drawing causal inferences. Complete randomization balances the covariate distributions between treatment groups in expectation, thereby ensuring the existence of unbiased estimators of average causal effects. Covariate imbalance, however, often occurs in specific randomized experiments, as recognized by Fisher (2) and later researchers (e.g., refs. 4⇓⇓⇓⇓–9). The standard approach advocated by Fisher (3), stratification or blocking, ensures balance with a few discrete covariates (e.g., refs. 10⇓–12).

When a randomized allocation is unbalanced, it is reasonable to discard that allocation and redraw another one until a certain predetermined covariate balance criterion is satisfied. This is rerandomization, an experimental design hinted at by Fisher (cf. ref. 13, p. 88) and Cox (14, 15) and formally proposed by Rubin (16) and Morgan and Rubin (17). Morgan and Rubin (17) showed that the difference-in-means estimator is generally unbiased for the average causal effect under rerandomization with equal-sized treatment groups and obtained the sampling variance of this estimator under additional assumptions of Gaussian covariate and outcome distributions and additive causal effects. When rerandomization is applied but these assumptions do not hold, statistical inference becomes more challenging, because the theory that is justified by the central limit theorem under complete randomization (18, 19) no longer generally holds. Some applied researchers believe that “the only analysis that we can be completely confident in is a permutation test or rerandomization test” (ref. 7, p. 219). However, permutation tests based on randomization require sharp null hypotheses that imply all missing potential outcomes are known.

Analogous to the repeated sampling properties for complete randomization (11, 20), we evaluate the repeated sampling properties of the difference-in-means estimator when rerandomization is used, where all potential outcomes and covariates are regarded as fixed quantities and all randomness arises solely from the random treatment assignments. The geometry of rerandomization reveals non-Gaussian asymptotic distributions, which serve as the foundation for constructing large-sample confidence intervals for average causal effects. Furthermore, we compare the lengths of quantile ranges of the asymptotic distributions of the difference-in-means estimator under rerandomization and complete randomization, extending Morgan and Rubin’s (17, 21) comparison of their sampling variances.

## Framework, Notation, and Basic Results

### Covariate Imbalance and Rerandomization.

Inferring the causal effect of some binary treatment on an outcome Y is of central interest in many studies. We consider an experiment with n units, with

When covariate imbalance arises in a drawn allocation, it is reasonable to discard that allocation and draw another until some a priori covariate balance criterion is satisfied. This is rerandomization, an intuitive experimental design tool apparently personally advocated by R. A. Fisher (discussion in ref. 16) and formally discussed by Morgan and Rubin (17).

In general, rerandomization entails the following steps: (*i*) Collect covariate data; (*ii*) specify a balance criterion to determine whether a randomization is acceptable or not; (*iii*) randomize the units to treatment and control groups; (*iv*) if the balance criterion is satisfied, proceed to step *v*, and otherwise, return to step *iii*; (*v*) conduct the experiment using the final randomization obtained in step *iv*; and (*vi*) analyze the data, taking into account the rerandomization used in steps *ii–iv*.

Although the balance criterion in step *ii* can be general, Morgan and Rubin (17) suggested using the Mahalanobis distance between covariate means in treatment and control groups, and they (21) suggested considering tiers of covariates according to their presumed importance in predicting the outcomes in this experiment. We discuss these two types of rerandomization in detail and apposite statistical inference after these rerandomizations as implied by step *vi*. We then extend the theory to general rerandomizations in *SI Appendix*, section A1.

### Potential Outcomes and Definitions of Finite Population Quantities.

We use the potential outcomes framework to define causal effects and let

### Repeated Sampling Inference in a CRE.

The observed outcome for unit i is

Let

#### Condition 1:

As *i*) *ii*) the finite population variances and covariances *iii*)

We introduce the notation *Condition 1*,

## Rerandomization Using the Mahalanobis Distance

### Mahalanobis Distance.

The Mahalanobis distance between the covariate means in treatment and control groups is

When we allow transformations and interactions of X, ReM can incorporate a wide class of rerandomization schemes. For small sample sizes, there may not exist any randomization satisfying some balance criterion. However, according to the finite population central limit theorem (19), the acceptance probability of a randomization is asymptotically *Discussion*.

### Multiple Correlation Between τ ^ and τ ^ X .

The sampling distribution of

### Proposition 1.

*can be expressed in terms of the variances of the potential outcomes and of their projections on* X:

When the causal effect is additive,

### Asymptotic Sampling Distribution of τ ^ Under ReM.

Simply stated,

### Theorem 1.

*Under ReM and Condition 1*,*where* *is independent of*

The coefficients of the linear combination are functions of **[****1****]** becomes a Gaussian random variable, the same as the asymptotic distribution of *Repeated Sampling Inference in a CRE*. Importantly, the definition of

### Representation of the Asymptotic Distribution Under ReM.

The asymptotic distribution in **[****1****]** involves a random variable

Let

### Proposition 2.

*can be represented as**where* *are mutually independent*, *and* *are jointly independent*. *is symmetric and unimodal around zero*, *with*

Because both **[****1****]**. Moreover, the unimodal property plays an important role in the conservativeness of confidence intervals, discussed shortly for ReM. The representation in **[****2****]** allows for easy simulation of **[****1****]**, which is relevant for statistical inference discussed later.

In *SI Appendix*, section A2, we give more detailed explanations regarding the geometry and the shape of the asymptotic distribution in **[1]**.

### Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.

*Theorem 1* characterizes the asymptotic behavior of

First, the asymptotic distribution in **[****1****]** is symmetric around 0, implying that

### Corollary 1.

*Under ReM and Condition 1*,

Morgan and Rubin (17) gave a counterexample showing that, in an experiment with unequal treatment group sizes, *Corollary 1* extends their theorem 2.1 (17) and ensures the asymptotic unbiasedness of

Covariates, whether observed or unobserved, are variables unaffected by the treatments. Therefore, the average causal effect on any covariate is 0, and *Corollary 1* implies that any covariate asymptotically has the same means under treatment and control.

Furthermore, from *Proposition 2* and *Theorem 1*, we can calculate the asymptotic sampling variances of *Corollary 2*.

### Corollary 2.

*Under ReM and Condition 1*, *the asymptotic sampling covariance of* *is* *and the PRIASV of any component of* *is* *The asymptotic sampling variance of* *is* *and the PRIASV of* *is*

Rigorously, the asymptotic sampling covariance and variance of

When the causal effect is additive, *Corollary 2* is an asymptotic extension of theorem 3.2 in Morgan and Rubin (17).

Under ReM, in addition to the sampling variance reduction result concerning *Corollary 2*, we consider the reduction in the length of the

Let

### Theorem 2.

*Under Condition 1*, *the length of the* *quantile range of the asymptotic sampling distribution of* *under ReM is less than or equal to that under the CRE*, *with the difference nondecreasing in* *and nonincreasing in* *and* K.

### Sampling Variance Estimation and Confidence Intervals.

Asymptotic sampling variance and quantile ranges for *SI Appendix*, section A4 that under ReM they are consistent for their population analogues. Therefore, we can then estimate **[****5****]** is negative.

According to *Corollary 2*, we can estimate the asymptotic sampling variance of **[****3****]**, we can construct a large sample

Moreover, the sampling variance estimator is smaller than Neyman’s (20) sampling variance estimator for the CRE, and the confidence interval is shorter than Neyman’s (20) confidence interval for the CRE. Therefore, if we conduct ReM in the design stage but analyze data as in the CRE, the consequential sampling variance estimator and confidence intervals will be overly conservative.

The above results are all intuitive, and we present the algebraic details for the proofs of these results in *SI Appendix*, section A4. Interestingly, as shown in *SI Appendix*, section A4, we do not need conditions beyond *Condition 1* to ensure the asymptotic properties of the sampling variance estimator and the confidence intervals.

## Rerandomization with Tiers of Covariates

### Mahalanobis Distance with Tiers of Covariates.

When covariates are thought to have different levels of importance for the outcomes, Morgan and Rubin (21) proposed rerandomization using the Mahalanobis distance with differing criteria for different tiers of covariates. We partition the covariates into T tiers indexed by *Rerandomization Using the Mahalanobis Distance* section.

### Multiple Correlation Between τ ^ and τ ^ E [ t ] .

Let *Proposition 1*,

### Asymptotic Sampling Distribution of τ ^ Under ReMT.

Intuitively,

As earlier, let

### Theorem 3.

*Under ReMT and Condition 1*,*where* *are jointly independent*.

In **[****6****]**, *Proposition 2*, the distribution in *Theorem 3* is easy to simulate.

### Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.

*Theorem 3* characterizes the asymptotic behavior of

First, the asymptotic distribution in **[****6****]** is symmetric around 0, implying that

### Corollary 3.

*Under ReMT and Condition 1*,

The asymptotic sampling covariance of *SI Appendix*, section A3. Below we present only the PRIASV of

### Corollary 4.

*Under ReMT and Condition 1*, *the asymptotic sampling variance of* *is* *and the PRIASV of* *is*

When the causal effect is additive, *Corollary 4* is an asymptotic extension of Morgan and Rubin’s theorem 4.2 (21). When the thresholds

We now compare the quantile range under ReMT to that under the CRE. Let *Theorem 4* is intuitive.

### Theorem 4.

*Under Condition 1*, *the* *quantile range of the asymptotic distribution of* *under ReMT is less than*, *or equal to*, *the range under the CRE*, *and the reduction in length is nondecreasing in* *and nonincreasing in* *and* *for all*

### Sampling Variance Estimation and Confidence Interval.

We can estimate *Corollary 4* and **[****7****]**, we can estimate the sampling variance of *SI Appendix*, section A4.

## An Education Example with Tiers of Covariates

We illustrate our theory using the data from the Student Achievement and Retention Project (24), a randomized evaluation of academic services and incentives at one of the satellite campuses of a large Canadian university. A treatment group of 150 students was offered an array of support services and substantial cash awards for meeting a target first-year grade-point average (GPA), and a control group of many more (1,006) students received only standard university support services.

To illustrate the benefit of rerandomization, we use the 15 covariates listed in Table 1 and exclude students with missing values, resulting in

Table 1 partitions the 15 covariates into three tiers. We choose *Left* shows the empirical coverage probabilities of our and Neyman’s (20) confidence intervals, showing that Neyman’s (20) CRE confidence intervals are very conservative.

To evaluate the performance of ReMT compared with a CRE, we compare the average length of Neyman’s (20) confidence interval under a CRE with the confidence interval under ReMT. From Fig. 1, *Right*, the percentage reduction in average lengths of the 95% confidence intervals under ReMT compared with Neyman’s (20) under a CRE is nondecreasing in

## Discussion

Our theory suggests that choosing a small

## Materials and Methods

We did not conduct the experiment, and we are analyzing secondary data without any personal identifying information. As such, this study is exempt from human subjects review. The original experiments underwent human subjects review in Canada (24).

## Acknowledgments

P.D. acknowledges support from the National Science Foundation (DMS 1713152). D.B.R. acknowledges support from the National Institute of Allergy and Infectious Diseases/NIH (R01AI102710), National Science Foundation (IIS-1409177), Office of Naval Research (N00014-17-1-2131), and a Google Faculty Fellowship.

## Footnotes

- ↵
^{1}To whom correspondence should be addressed. Email: dbrubin{at}mac.com.

Author contributions: X.L., P.D., and D.B.R. designed research, performed research, contributed new reagents/analytic tools, analyzed data, and wrote the paper.

Reviewers: R.J.T., Stanford University; and C.F.J.W., Georgia Institute of Technology.

The authors declare no conflict of interest.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1808191115/-/DCSupplemental.

Published under the PNAS license.

## References

- ↵
- Fisher RA

- ↵
- Fisher RA

- ↵
- Fisher RA

- ↵
- ↵
- ↵
- ↵
- ↵
- Krieger AM,
- Azriel D,
- Kapelner A

- ↵
- Banerjee A,
- Esther D

- Athey S,
- Imbens GW

- ↵
- Cochran WG,
- Cox GM

- ↵
- Imbens GW,
- Rubin DB

- ↵
- Higgins MJ,
- Sävje F,
- Sekhon JS

- ↵
- Savage LJ

- ↵
- Krishnaiah PR,
- Kallianpur G,
- Ghosh JK

- Cox DR

- ↵
- Cox DR

- ↵
- ↵
- ↵
- ↵
- Li X,
- Ding P

- ↵
- Neyman J

- ↵
- Morgan KL,
- Rubin DB

- ↵
- Wintner A

- ↵
- Ding P,
- Feller A,
- Miratrix L

- ↵
- Angrist J,
- Lang D,
- Oreopoulos P

## Citation Manager Formats

## Sign up for Article Alerts

## Article Classifications

- Physical Sciences
- Statistics