# Part mutual information for quantifying direct associations in networks

^{a}Key Laboratory of Systems Biology, Innovation Center for Cell Signaling Network, Institute of Biochemistry and Cell Biology, Shanghai Institutes for Biological Sciences, Chinese Academy of Sciences, University of the Chinese Academy of Sciences, Shanghai 200031, China;^{b}School of Life Science and Technology, ShanghaiTech University, Shanghai 200031, China;^{c}Collaborative Research Center for Innovative Mathematical Modelling, Institute of Industrial Science, University of Tokyo, Tokyo 113-8654, Japan

See allHide authors and affiliations

Edited by Wing Hung Wong, Stanford University, Stanford, CA, and approved March 22, 2016 (received for review November 17, 2015)

## Significance

Measuring direct associations between variables is of great importance in various areas of science, especially in the era of big data. Although mutual information and conditional mutual information are widely used in quantifying both linear and nonlinear associations, they suffer from the serious problems of overestimation and underestimation. To overcome these problems, in contrast to conditional independence, we propose a novel concept of “partial independence” with a new measure, “part mutual information,” based on information theory that can accurately quantify the nonlinearly direct associations between the measured variables.

## Abstract

Quantitatively identifying direct dependencies between variables is an important task in data analysis, in particular for reconstructing various types of networks and causal relations in science and engineering. One of the most widely used criteria is partial correlation, but it can only measure linearly direct association and miss nonlinear associations. However, based on conditional independence, conditional mutual information (CMI) is able to quantify nonlinearly direct relationships among variables from the observed data, superior to linear measures, but suffers from a serious problem of underestimation, in particular for those variables with tight associations in a network, which severely limits its applications. In this work, we propose a new concept, “partial independence,” with a new measure, “part mutual information” (PMI), which not only can overcome the problem of CMI but also retains the quantification properties of both mutual information (MI) and CMI. Specifically, we first defined PMI to measure nonlinearly direct dependencies between variables and then derived its relations with MI and CMI. Finally, we used a number of simulated data as benchmark examples to numerically demonstrate PMI features and further real gene expression data from *Escherichia coli* and yeast to reconstruct gene regulatory networks, which all validated the advantages of PMI for accurately quantifying nonlinearly direct associations in networks.

Big data provide unprecedented information and opportunities to uncover ambiguous correlations among measured variables, but how to further infer direct associations, which means two variables are dependent given all of the remaining variables (1), quantitatively from those correlations or data remains a challenging task, in particular in science and engineering. For instance, distinguishing dependencies or direct associations between molecules is of great importance in reconstructing gene regulatory networks in biology (2⇓–4), which can elucidate the molecular mechanisms of complex biological processes at a network level. Traditionally, correlation [e.g., the Pearson correlation coefficient (PCC)] is widely used to evaluate linear relations between the measured variables (2, 5), but it cannot distinguish indirect and direct associations due to only relying on the information of co-occurring events. Partial correlation (PC) avoids this problem by considering additional information of conditional events and can detect the direct associations. Thus, PC becomes one of the most widely used criteria to infer direct associations in various areas. As applications of PC to network reconstruction (6), recently Barzel and Barabási (7) proposed a dynamical correlation-based method to discriminate direct and indirect associations by silencing indirect effects in networks, and Feizi et al. (8) developed a network deconvolution method to distinguish direct dependencies by removing the combined effect of all indirect associations. However, PC-based methods including these two works (8) can only measure linearly direct associations and will miss nonlinear relations, which play essential roles in many nonlinear systems, such as biological systems. Analogous to PCC and PC, distance correlation (9, 10) and partial distance correlation (Pdcor) (11) were proposed to measure dependence of random vectors, and these statistics are sensitive to many types of departures from dependence (9). However, Pdcor suffers from the false-positive problem, that is, Pdcor(X;Y|Z) may be nonzero even if X and Y are conditional independent given Z (11).

Based on mutual independence, mutual information (MI) can be considered to be a nonlinear version of correlation that can detect nonlinear correlations but not direct associations or dependencies owing to the information of only joint probability, having the same overestimation problem as PCC (12, 13). As one variant of MI, the maximum information coefficient (MIC) method (3) was proposed to detect both linear and nonlinear correlations between variables. MIC is based on MI but has a few different features on measuring nonlinear associations. However, recently, it has been shown that MI is actually more equitable than MIC (14). By further considering conditional independence from MI, conditional mutual information (CMI) as a powerful tool can quantify nonlinearly direct dependencies among variables from the observed data, superior to PC and also MI, and thus has been widely used to infer networks and direct dependencies in many fields (12, 15⇓–17), such as gene regulatory networks in biology (16). However, CMI-based methods theoretically suffer from a serious problem of underestimation, in particular when quantifying the dependencies among those variables with tight associations among them, which severely limits its applications. For example, when we measure the dependency between variables X and Y given variable Z, CMI cannot correctly measure the direct association or dependency if X (or Y) is strongly associated with Z. For an extreme case, if X and Z are almost always equal [i.e., the conditional probability p(*x*|*z*) = 1 provided that the joint probability p(*x*,*y*,*z*)

To overcome these problems of MI and CMI, we propose a new measurement, “part mutual information” (PMI), by defining a new type of “partial independence” and also deriving an extended Kullback–Leibler (KL) divergence (*SI Appendix*, section S1) to accurately quantify the nonlinearly direct associations from the measured variables. We show that PMI not only can theoretically avoid the problems of CMI and MI but also keeps the quantification properties. Similar to CMI, it can also be used to reconstruct various types of networks even with loops or circles, for which Bayesian-based methods generally failed. Specifically, in this paper we first define PMI to measure nonlinearly direct dependencies between variables, and then derive its theoretical relations with MI and CMI. Finally, we use a number of benchmark examples to numerically demonstrate PMI features and, further, real gene expression data to reconstruct gene regulatory networks in *Escherichia coli* and yeast, which all validated advantages of PMI for quantifying nonlinearly direct associations in networks.

## Results

### MI and CMI.

Assuming that X and Y are two random variables, MI in information theory represents the information that we need when using Y to code X, and vice versa. MI is defined on the basis of KL divergence *D* (13). For discrete variables X and Y, MI is calculated as**1** is evaluated against the “mutual independence” of X and Y, which is defined as*A*, if two independent variables X and Y are both associated with a common random variable Z, MI between X and Y is larger than zero, thus wrongly indicating the direct association between X and Y. The reason is that MI relies on the information of mutual independence in Eq. **1** and cannot correctly measure the direct association, which leads to overestimation (i.e., the false-positive problem) (16). However, CMI can correctly measure the direct association for the case of Fig. 1*A* by further considering the information of conditional probability or conditional independence. In contrast to partial correlation for linearly direct association, CMI for variables X and Y given Z can further detect nonlinearly direct association and is defined as*p*(*z*) is the marginal probability, and *p*(*x|z*) and *p*(*y|z*) are conditional marginal probability distributions. Clearly, in contrast to MI in Eq. **1** or the mutual independence of X and Y in Eq. **2**, CMI in Eq. **3** is evaluated against the “conditional independence” of X and Y given Z, which is defined as*n* > 2), and thus Fig. 1 represents a general case for evaluating the direct association of any pair of variables X and Y in a network with n nodes. CMI is also nonnegative, and thus a positive or high CMI implies the dependency of X and Y. In addition to Fig. 1*A*, as shown in Fig. 1*B* where X and Y have direct dependency CMI can also correctly measure the nonlinear dependency between X and Y, in particular for those cases where both X and Y have weak associations with Z (e.g., both X and Y are almost independent of Z). However, CMI theoretically suffers from the false-negative or underestimation problem, that is, even when CMI equals zero, X and Y are not necessarily independent (see *SI Appendix*, *Property S6*). As shown in Fig. 1*C*, *Left*, when X or Y is strongly associated with Z, CMI between X and Y always approaches zero even though X and Y have direct dependency, which means that CMI fails to measure the correct association. In fact, theoretically CMI(X;Y|Z) always suffers from this underestimation problem provided that there are certain associations between X (or Y) and Z (see *SI Appendix*, *Property S6*) in a general network. Such cases are general in various natural or engineered systems where variables are associated or interconnected to form a network.

### PMI.

To overcome those problems of MI and CMI, we propose a novel measure, PMI, to quantify nonlinearly direct dependencies between variables, by introducing a new concept, partial independence. Specifically, in contrast to the conditional independence Eq. **5**, partial independence of X and Y given Z is defined as*p**(*x|z*) and *p**(*y|z*) are defined (18) as**6**.

##### Definition 1:

X and Y are two scalar variables and Z is an (n-2)-dimensional vector (*n* > 2). Then, the PMI between variables X and Y given Z is defined as*p*(*x,y,z*) is the joint probability distribution of X, Y, and Z, and *D*(*p*(*x,y,z*)*||p**(*x|z*)*p**(*y|z*)*p*(*z*)) represents the extended KL divergence (see *SI Appendix*, *Definition S1*) from *p*(*x,y,z*) to *p**(*x|z*)*p**(*y|z*)*p*(*z*)*.*

From Eq. **8**, it is obvious that PMI is always larger than or equal to zero owing to the extended KL divergence (see *SI Appendix*, *Theorem S1*). Eq. **8** can also be rewritten as**4** and Eq. **9**, clearly PMI is similarly defined as CMI by replacing *p*(*x|z*) *p*(*y|z*) in CMI with **6**. In addition, Eq. **9** can be further decomposed as follows:**10** correspond to a′ and a′′ in Fig. 1*C*, respectively. The proof of Eq. **10** is given in *SI Appendix*. Clearly, CMI(X;Y|Z) is the first term of PMI in Eq. **10**, and the second and third terms implicit include the associations between X and Y, respectively, due to Eq. **7**.

Next, we show that PMI can overcome the problems of MI and CMI owing to the term of partial independence in Eq. **8**. From Eqs. **8**–**10**, we can derive the specific properties of PMI and its relations with CMI and MI, shown in *SI Appendix*, Table S15 and *Properties S1–S9*. The properties of the conditional independence Eq. **5** and partial independence Eq. **6** are given in *SI Appendix*, *Properties S7* and *S8* and Fig. S1.

First, PMI is symmetric, that is, PMI(X;Y|Z) = PMI(Y;X|Z). Second, PMI is always larger than or equal to CMI, that is, PMI(X;Y|Z) ≥ CMI(X;Y|Z), which implies the potential of PMI to overcome the underestimation problem of CMI. Third, PMI(X;Y|Z) = 0 only if X and Y given Z are independent, and otherwise PMI(X;Y|Z) > 0, which implies that PMI can measure nonlinearly direct dependency between X and Y given Z. All major features of PMI and their related proofs can be found in *SI Appendix*, *Properties S1–S9*. When X and Y are independent of Z, that is, *SI Appendix*, *Property S5*) when the variables of interest are isolated from others of the system.

Next, we theoretically show that PMI overcomes those problems of CMI and MI. If X and Y are strongly associated, we denote the relation as X ≈ Y, which implies the strongly mutual dependency between X and Y. If X or Y is strongly associated with Z, for example, X ≈ Z or Y ≈ Z, then CMI(X;Y|Z) = 0 regardless of the dependency between X and Y, but PMI(X;Y|Z) relying on the direct dependency of X and Y (see *SI Appendix*) is generally not necessarily zero. Hence, in Fig. 1*C*, PMI can detect the direct association, for which CMI failed. Actually, PMI can measure the direct associations correctly for all cases in Fig. 1 (see *SI Appendix*, *Properties S1–S9* for the proofs).

The analyses on the differences between the conditional independence and partial independence are given in *SI Appendix*, *Properties S7* and *S8* and Fig. S1. Actually, we can show that both the conditional independence Eq. **5** and the partial independence Eq. **6** hold, that is, *p*(*x|z*)*p*(*y|z*) *= p*(*x,y|z*) and *p**(*x|z*)*p**(*y|z*) *= p*(*x,y|z*), if X and Y are conditionally independent given Z (see *SI Appendix*, *Property S7*). Thus, both CMI and PMI based on Eq. **5** and Eq. **6** can detect the conditional independence of X and Y given Z (i.e., there is no direct association). However, if X and Y are conditionally dependent given Z (i.e., there is a direct association), the conditional independence Eq. **5** always holds approximately provided that X (or Y) strongly depends on Z, but the partial independence Eq. **6** does not necessarily hold (see *SI Appendix*, *Property S8*) and it relies on the direct association of X and Y. Therefore, although both CMI and PMI can give the correct results on the independence of X and Y given Z, PMI can also give the correct results on the dependence of X and Y given Z due to the partial independence Eq. **6**, and CMI generally cannot give the correct results on the dependence due to the conditional independence Eq. **5**.

Similar to MI and CMI, there are many ways to numerically estimate joint and marginal probabilities (20⇓–22) in PMI directly from the observed data. The most straightforward and widespread method to estimate *p*(*x,y,z*) is partitioning the supports of X, Y, and Z into bins of finite size, approximating the continuous distribution by the finite discrete distribution, and converting the integration operators into the finite sum operators. The number of the bins may have some impact on the computing result, but it would not affect the statistical *P* value so much. Empirically, if Z has n-2 dimensions and there are N data points, the whole n-dimension space will be partitioned into bins with a magnitude of *n* is the number of total measured variables. However, this approximate method works in the case that N>>n-2. Otherwise, the error of the estimation will be quite large because of the “curse of dimensionality.” If we assume the Gaussian distribution for variables, MI between X and Y can be simplified as below (16, 22):**11** provides a simple way to calculate MI. By analogy to MI and CMI (16, 22), PMI can be accurately expressed from the covariance matrix of X, Y, and Z.

### Theorem 1.

*Assume that* X *and* Y *are two one-dimensional variables*, *and* Z *is an* n-2 *dimensional* (n-2 > 0) *vector*, *and they are defined in an appropriate outcome space*. *Letting the vector* (X,Y,Z) *follow multivariate Gaussian distribution*, *then PMI between* X *and* Y *given* Z *is simplified as follows*:*where* *C*, *C*_{0}, *and* Σ *are all measured matrices given in* *SI Appendix*, *section S2* *and* n *is the dimension of vector* (X,Y,Z).

The proof of *Theorem 1* is given in *SI Appendix*, section S2. *Theorem 1* yields the formula to calculate PMI in a simple but approximate way due to the Gaussian distribution assumption. Besides, we can derive the following relation between PMI and PC:*SI Appendix*, section S2, *Lemmas S1* and *S2* and *Theorem S2*.

### Numerical Studies by Simulated Data.

Theoretically, we have shown that PMI is superior to CMI and other linear measurements for quantifying the nonlinearly direct dependencies among variables (*SI Appendix*, section S1), in particular for those cases that CMI or MI fails. To numerically investigate these, we first used simulated data to study both linear and nonlinear relationships as in the previous work of Barzel and Barabási (7). We compared the performances of PMI with CMI and PC about these direct associations, including linear, parabolic, cubic, sinusoidal, exponential, checkerboard, circular, cross-shaped, sigmoid, and random relationships. Three variables, X, Y, and Z, were simulated, satisfying *f* (X,Z) given Z can be linear, parabolic, cubic, sinusoidal, exponential, checkerboard, circular, cross-shaped, sigmoid, and random functional relations and nonfunctional relations (see *SI Appendix*, Table S1 and Fig. S2). Various cases, for example X (or Y) is independent of Z or strongly associated with Z, were considered during simulations, and the results for all are displayed in Table 1 and *SI Appendix*, Tables S4 and S5, where PC(X;Y|Z) is the partial correlation between X and Y given Z and PS(X;Y|Z) is the partial spearman correlation between X and Y given Z. All of the *P* values are provided in *SI Appendix*, Tables S2, S3, S6, and S7. We also compared PMI with CMI and PC when X is moderately correlated with Z and results indicated that PMI performs the best, as shown in *SI Appendix*, Tables S13 and S14.

All PMI, CMI, and PC between X and Y were calculated as the means over 1,000 repeats, and the values less than 0.01 were set to zero. The probability distributions were estimated by “bins” -based method (see *SI Appendix*, section S3) (21). We considered both uniform and normal distributions, shown in Table 1 for uniform distribution and *SI Appendix*, Tables S4 and S5 for normal distribution, respectively. When Z is independent of X, Table 1 shows clearly that PMI and CMI can detect the true dependency effectively in all relations with high significance (see *SI Appendix*, Table S2), whereas PC and PS are correct almost only for the linear relation due to its linear nature, and they are also sensitive to cubic, exponential, and sigmoid because these relations are mostly resemble to linear relations in the given interval. However, in Table 1 where X is strongly associated with Z, that is, *p*(*x*|*z*) = 1 for *p*(*x,y,z*)*p*(*y*|*z*) = 1 for *p*(*x,y,z*)*SI Appendix*, Tables S4 and S5 and the matched *P* values in *SI Appendix*, Tables S6 and S7, we have similar conclusions. In addition to the nonlinear relations in *SI Appendix*, Table S1, we also conducted the simulations on other types of relations in *SI Appendix*, Table S8, and, as shown in *SI Appendix*, Tables S9–S12, all of the results demonstrate the robustness of the PMI.

### Comparison for Statistical Power.

As an association measure, statistical power is an important and widely used tool to interpret the results calculated by the measure as well as statistical significance (23). Here, statistical power, or simply “power,” defined similarly as in ref. 14, is the probability that we reject the hypothesis of variables X and Y given Z are independent when X and Y given Z are truly associated. The PMI value shows a significant difference from the value that X and Y given Z are independent. If the false-negative rate, namely a type II error in statistics, is equal to

The relationships considered here are the same as in *SI Appendix*, Table S1. The power of PMI and CMI was presented as a heat map. Fig. 2 *A* and *D* show the power of PMI and CMI, where the two measures present almost the same results in the situation when X and Z are independent (i.e., no association of X and Z). This is consistent with the theoretical conclusion that PMI performs as well as CMI when X or Y is not associated with Z. However, in Fig. 2 *B* and *E*, when the association between X and Z becomes moderate or stronger, the power of PMI is higher than that of CMI. Also, in Fig. 2 *C* and *F* when X is almost always equal to Z (or strongly associated with Z), that is, *p*(*x|z*) = 1 for *p*(*x,y,z*)*SI Appendix*, section S4 and Fig. S3.

### DREAM3 Challenge Datasets for *E. coli* and Yeast.

We applied PMI to the widely used DREAM3 challenge datasets for reconstructing gene regulatory networks (24), where the gold standard networks were from yeast and *E. coli*. The gene expression data were generated with the nonlinear ordinary differential equation systems (19, 25). In this work, we used DREAM3 challenging data with sizes 10, 50, and 100 to construct the gene regulatory network by PMI-based PC (path consistency) algorithm, which is a greedy iteration algorithm for inferring networks and terminates until the network converges (16). Simply by replacing CMI with PMI in the original CMI-based algorithm (i.e., PCA-CMI) (16), we can obtain PMI-based PC-algorithm, called PCA-PMI, which is based on Eq. **12** in *Theorem 1*. The detailed algorithm of PCA-PMI is given in *SI Appendix**.* The performance of PMI was evaluated by the true-positive rate (TPR) and false-positive rate (FPR). They were defined as TPR = TP/(TP + FN) and FPR = FP/(FP + TN), where TP, FP, TN, and FN represent the numbers of true positives, false positives, true negatives, and false negatives, respectively. TPR and FPR are used for plotting the receiver operating characteristic (ROC) curves and the area under ROC curve (AUC) is calculated. In addition, we compared PMI and CMI by inferring the gene regulatory networks.

We first analyzed the yeast knock-out gene expression data with 10 genes and 10 samples by PCA-PMI algorithm. If PMI of two genes is less than the threshold, we consider that there is no direct association between these two genes; otherwise, the two genes may have the direct regulation. The joint and marginal probabilities as well as covariance matrices were estimated by Eq. **12** in *Theorem 1* with the assumption of the Gaussian distribution due to a large number of variables. Fig. 3*A* shows the true gene regulatory network, with 10 nodes and 10 edges, and Fig. 3*C* is the network inferred by PCA-PMI with 10 nodes and 9 edges, where only one edge between G4 and G9 was false-negative. For comparison, we used CMI-based PC-algorithm (PCA-CMI) to construct the same network as shown in Fig. 3*B*, and the threshold was selected when the accuracy was maximized. Fig. 3*B* shows that the network predicted by CMI has two false-negative edges, the edge between G1 and G3 and the edge between G4 and G9. Clearly, PMI is more accurate than CMI for this case. To further evaluate the performance of PMI, the AUC was reported. Fig. 4*A* shows the ROC curve of yeast network with 10 genes, where PMI outperforms CMI apparently in terms of AUC. We also compared PMI with partial correlation, Lasso, and Pdcor, where PMI performed the best.

Next, we tested the data of yeast gene expression with 100 genes for datasets of 100 samples. We chose different thresholds ranging from 0 to 0.5 for PCA-PMI (see *SI Appendix*, Fig. S4*A*). The true network of 100 genes with 166 edges is the real and verified network. We selected the threshold 0.25 with the maximum accuracy 0.97. The ROC curve is shown in Fig. 4*C*. As a whole, PMI is superior to CMI in terms of ROC curves.

Finally, we performed PMI and CMI computation on two datasets of *E. coli* network with 50 and 100 genes. Fig. 4 *B* and *D* are the ROC curves for these data, where AUC values of PMI are 0.843 and 0.858, respectively, higher than those of CMI. *SI Appendix*, Fig. S4 *A* and *B* show that the accuracy is consistent with changing threshold between 0 and 0.5. As shown in *SI Appendix*, Fig. S4, with the increase of the network size, the accuracy is improved. We also compared PMI with the partial correlation, Lasso, and Pdcor, and PMI performed the best in all the cases in Fig. 4. In all datasets from the DREAM3 challenge, PMI was verified to be efficient and effective to reconstruct real gene regulatory networks from the measured expression data. Interestingly, when replacing the covariance matrix of the measured variables by the distance covariance matrix (8), we have kPCA-PMI algorithm (see *SI Appendix*), which has better performance on the above cases than PCA-PMI, shown in *SI Appendix*, Fig. S5.

## Discussion

MI based on mutual independence and CMI based on conditional independence suffer from the problems of overestimation and underestimation, respectively. However, recently, the causal strength *C*_{X→Y}(X;Y|Z) (see *SI Appendix*, *Property S9*) for quantifying causal strength was proposed by Janzing et al. (18). The causal strength (CS) can be used to quantify causal influences among variables in a network. However, CS is asymmetric compared with CMI, and it partially solves the problem of CMI (see *SI Appendix*, *Property S9*). In this work, to solve these problems we propose the novel concept of PMI by defining a new “partial independence” in Eq. **6** that can measure nonlinearly direct dependencies between variables with higher statistical power. Owing to this partial independence, PMI can quantify the direct associations effectively by theoretically avoiding the false-negative/-positive problems, in contrast to CMI and also other measures. Moreover, PMI can be also used for reconstructing a network with or without loops. If there is no influence on the two variables of interest from other variables in a network, PMI for measuring the association of these two variables is reduced to MI. Combining with PC-algorithm (16), PMI can efficiently reconstruct various types of large-scale networks from the measured data, such as gene regulatory networks in living organisms. In addition to the theoretical results, both simulated and real data demonstrate the effectiveness of PMI as a quantitative measure for nonlinearly direct associations.

To theoretically define PMI, we introduced partial independence in Eq. **6**, in contrast to the conditional independence in Eq. **5** for CMI and mutual independence in Eq. **2** for MI. As shown in *SI Appendix*, *Property S8* and Fig. S1, as well as the computational results, the conditional independence Eq. **5** always holds approximately for the case that X (or/and Y) is strongly associated with Z, even if there is strong dependence of X and Y given Z, but the partial independence Eq. **6** does not necessarily hold. Therefore, when the associations among variables in a network or system are weak, both PMI and CMI can correctly quantify the direct dependencies, but when some associations among variables are moderate or strong (it is actually a general case in a network or system), CMI generally failed due to its false-negative features and actually the error increases with the association strength, whereas PMI can detect their true dependencies due to the effect of the partial independence.

Intuitively, if X (or Y) is strongly associated with Z, CMI(X;Y|Z) = H(X|Z) − H(X|Y,Z) vanishes due to H(X|Z)*p*(*x|z*) *p*(*y|z*) with *p**(*x|z*) *p**(*y|z*), where *p**(*x|z*) [or *p**(*y|z*)] implicitly includes the association information between X and Y, different from *p*(*x|z*) [or *p*(*y|z*)].

We also derived an extended KL divergence (see *SI Appendix*, *Definition S1* and *Theorem S1*). The difference between PMI and CMI is the two conditional probability distributions [*p*(*x|z*)*p*(*y|z*) and *p**(*x|z*)*p**(*y|z*)] (see *SI Appendix*, Fig. S1), and their difference is clearly related to the relation between X and Y (see *SI Appendix*, Fig. S1). For continuous random variables, PMI can be defined in the same way as Eqs. **7**–**9**, that is,*p*(*x*,*y*,*z*) is now the joint probability density function of X, Y, and Z, and all others are the marginal (or conditional) probability density functions of random variables X, Y, and Z.

This work mainly introduced a new information criterion, PMI, to measure direct associations between nodes in a network based on the observed data, but it cannot give the directions of the measured associations, which should be detected by other methods, similar to CMI and MI. Also, PMI is not designed to test the hypothesis on the causal relations for observational studies (26, 27), for which CMI may be appropriate. In addition, how to numerically estimate PMI accurately from a small number of samples without the “bins”-based approximation or Gaussian distribution approximation is a topic for future research. To serve the communities of biology and medicine for inferring gene regulatory networks from high throughput data, all of the source codes and the web tool of our algorithm PCA-PMI can be accessed at www.sysbio.ac.cn/cb/chenlab/software/PCA-PMI/.

## Acknowledgments

This work was supported by Strategic Priority Research Program of the Chinese Academy of Sciences Grant XDB13040700 and National Natural Science Foundation of China Grants 61134013, 91439103, and 91529303.

## Footnotes

↵

^{1}J.Z. and Y.Z. contributed equally to this work.- ↵
^{2}To whom correspondence should be addressed. Email: lnchen{at}sibs.ac.cn.

Author contributions: J.Z. and L.C. designed research; J.Z. and Y.Z. performed research; J.Z., Y.Z., X.Z., and L.C. analyzed data; and J.Z. and L.C. wrote the paper.

The authors declare no conflict of interest.

This article is a PNAS Direct Submission.

Data deposition: All analysis code reported in this paper is available at www.sysbio.ac.cn/cb/chenlab/software/PCA-PMI/.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1522586113/-/DCSupplemental.

## References

- ↵.
- Whittaker J

- ↵.
- Stuart JM,
- Segal E,
- Koller D,
- Kim SK

- ↵.
- Reshef DN, et al.

- ↵.
- Wang YXR,
- Waterman MS,
- Huang H

- ↵.
- Eisen MB,
- Spellman PT,
- Brown PO,
- Botstein D

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵.
- Kinney JB,
- Atwal GS

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵.
- Marbach D, et al.

- ↵
- ↵.
- Rosenbaum PR

- ↵.
- Hernan MA,
- Robins JM

## Citation Manager Formats

## Article Classifications

- Biological Sciences
- Systems Biology