Analysis of gene expression in pathophysiological states: Balancing false discovery and false negative rates

Norris and Kahn. 10.1073/pnas.0510115103.

Supporting Information

Files in this Data Supplement:

Supporting Table 3
Supporting Text

Table 3. Multiple-hypothesis-testing outcomes

 

Supporting Text

FNR Procedure Motivation. Adopting established nomenclature (1), one can define the numeric outcomes from multiple-hypothesis testing (Table 3). The hypotheses (i.e., genes) can be sorted by the magnitude of their test statistics into descending order of significance: T₁>T₂..>T_m. The false negative rate (FNR) is simply

FNR = E(G/m_a) = 1 – E(S)/m_a.

Because m_a is invariant for a given analysis, it was extracted from the expectation operator. We can rewrite the expected value of S for any collection of genes 1…k as

E(S_k) = å E(s_i), i = 1…k,

where s_i is the probability that gene i is truly changing and declared positive, notated Pr(H_a & call pos). By using conditional probability,

E(s_i) = Pr(call pos | H_a) × Pr(H_a).

Recall that Pr(call pos | H_a) is statistical power, notated here as pow, which can be calculated through resampling. In single-hypothesis testing, Pr(H_a) is the statistical confidence = 1 – p value. This approach obviously fails in multiple-hypothesis testing. Below, we detail the derivation of an estimator, termed c, for Pr(H_a) in the multiple testing context:

c_i = 1 – (m/m₀) × ( i × FDR_i – (i –1) × FDR_{i – 1} ).

Thus, the value S_k for collection of genes 1…k is

E(S_k) = å (pow_i × c_i), i = 1…k.

The FNR at gene k is then simply FNR_k = 1 – E(S_k)/m_a. However, because, necessarily, FNR₀ = 1 and FNR_m = 0, and because m_a cannot be truly known, it is prudent to drop m_a from the equation and, instead, substitute the terminal value of S:

FNR_k = 1 – E(S_k)/E(S_m).

FNR Resampling Algorithm. Hypotheses (i.e., genes) are sorted by the magnitude of their test-statistics T_i = |t_i| into descending order of significance: T₁>T₂..>T_m.

1. The null distribution is sampled by B independent permutations of the data labels, producing null statisticsT⁰_i,b for each gene i.

2. The alternate distribution is sampled by B independent bootstraps of the data, preserving category identity, producing alternate statistics T^a_i,b for each gene i.

3. The FDR is calculated as outlined in Box 5 of Ge et al. (1).

4. For each gene i, calculate c_i = 1 – FDR_i × i + FDR_{i – 1} × (i – 1).

5. For each gene i, sort T⁰_i,1>T⁰_i,2..>T⁰_i,m. Find the null statistic at the significance level a as T⁰_{i,(a × B)}.

6. For each gene i, determine its power as the fraction of alternate distribution test statistics >T⁰_{i,(a × B)}. Formally, pow_i = #{k: T^a_i,k>T⁰_{i,(a × B)}}/B.

7. Calculate S_k = å (pow_i × c_i), i = 1…k, across the entire sorted gene list.

8. Finally, determine the FNR at each gene k as FNR_k = 1 – S_k/S_m.

Derivation of c. For any single gene, the statistical confidence corrected for multiple-hypothesis testing can be derived from the FDR, as outlined here. We assume that the FDR has been calculated for a gene list that has been sorted by from strongest to weakest test-statistic magnitudes, i.e., T₁>T₂..>T_m. The FDR is thus defined as

FDR_i = E(V_i/R_i).

At position i, R_i = i, that is, i genes are being declared positives. Thus,

FDR_i = E(V_i)/i.

We can define V_i as the sum of the relevant v_, the genewise probabilities for V, notated as follows

V_i = å v_k , k = 1…i.

For each gene k, v_k can be calculated as the joint probability that the gene is declared significant and that it is truly negative, which can be notated as Pr(declared significant & truly negative). By using conditional probability, this can be rearranged as

v_k = Pr(declared significant | truly negative) × Pr(truly negative).

The probability for any gene being declared significant, given that it is truly negative, namely, Pr(declared significant | truly negative), is the significance level, a_k. Because this term is derived in the context of the FDR, it is a significance level corrected for multiple-hypothesis testing, so we will notate it a_k,FDR. We will take the unconditional probability of being truly negative simply as m₀/m. Thus,

v_k = a_k,FDR × (m₀/m)

Returning now to the definition of the FDR, we find that

FDR_i = å a_k,FDR × (m₀/m)/i, k = 1…i.

Rearranging and splitting of terms then yields the following:

i × FDR_i = å a_k,FDR × (m₀/m) + a_i,FDR × (m₀/m), k = 1…i – 1,

and

i × FDR_i = (i – 1) × FDR_{i – 1} + a_i,FDR × (m₀/m),

and finally

a_i,FDR = (m/m₀) × (i × FDR_i – (i – 1) × FDR_{i – 1}).

We are interested in the statistical confidence for each single gene, corrected for multiple-hypothesis testing. This can then be easily derived, knowing that statistical confidence and significance sum to one. Thus,

c_i = 1 – (m/m₀) × (i × FDR_i – (i – 1) × FDR_{i – 1}).

Because the estimated FDR is imperfect, c_i, when estimated from the FDR, can have considerable point-to-point variability. Thus, it is not suitable, in the present form, for asking questions about individual genes. But when used for collections of genes, the amassed behavior of c is sufficiently accurate for use in aggregate calculations such as the FNR algorithm presented above.

1. Ge, Y. C., Dudoit, S. & Speed, T. P. (2003) Test 12, 1–77.

• Early Edition
• Archives
• Online Submission

• Feature Articles

  (Expanded research articles of exceptional breadth)

• Commentaries

  (Expert commentary on leading research)

• Letters

  (Online-only letters to the editor)

• Inaugural Articles

  (Articles by recently elected NAS members)

• PNAS Profiles

  (Biographical profiles of Academy members)

• Sustainability Science

  (Interactions of human and environmental systems)

• Collected Papers

  (Editorials, Perspectives, Reviews, Colloquia, etc.)

• Special Features

  (Solicited articles on exceptional research topics)

• Sackler Colloquia

  (Scientific reports held under NAS auspices)