Research Article

Visualizing probabilistic models and data with Intensive Principal Component Analysis

See allHide authors and affiliations

PNAS July 9, 2019 116 (28) 13762-13767; first published June 24, 2019; https://doi.org/10.1073/pnas.1817218116

  1. Edited by William Bialek, Princeton University, Princeton, NJ, and approved May 31, 2019 (received for review October 5, 2018)

Article Figures & SI

Figures

  • Fig. 1.
    Fig. 1.

    (A–C) Hypersphere embedding, illustrating an embedding of the 2D Ising model. Points were generated through a Monte Carlo sampling and visualized by projecting the probability distributions onto the first three principal components (28). The points are colored by magnetic field strength. As the system size increases from 2×2 to 4×4, the orthogonality problem is demonstrated by an increase in “wrapping” around the hypersphere. This effect can also be produced by instead considering four replicas of the original system, motivating the replica trick which takes the embedding dimension or number of replicas to zero.

  • Fig. 2.
    Fig. 2.

    Replicated Ising model illustrating the derivation of our intensive embedding. All points are colored by magnetic field strength. (A) Large dimensions are characterized by large system sizes; here we mimic a 128×128 Ising model which is of dimension 21282. The orthogonality problem becomes manifest as all points are effectively orthogonal, producing a useless visualization with all points clustered in the cusp. (B) Using replica theory, we tune the dimensionality of the system and consider the limit as the number of replicas goes to zero. In this way, we derive our intensive embedding. Note that the z axis reflects a negative-squared distance, a property which allows violations of the triangle inequality and is discussed in the text.

  • Fig. 3.
    Fig. 3.

    Stages of training a CNN. Each point in the 3D projections represents one of 10,000 test images supplied to the CNN (29). At the first epoch, the neural network is untrained and so is unable to reliably classify images, with about a 90% error rate—an effect reflected in the cloud of points. As training progresses and error rate decreases, the cloud begins to cluster as shown by InPCA at the 20th epoch. Finally, when completely trained, the clustered regions are manifest at the 2000th epoch with 10 clusters representing the 10 digits.

  • Fig. 4.
    Fig. 4.

    InPCA visualization of biased coins (30). (A) The first two InPCA components correspond to the coin bias and variance, yet the first one is real and the second one is imaginary (the aspect ratio between axes is one). The contour lines represent constant distances from a fair coin and are hyperbolas: Points can be a finite distance from a fair coin yet an infinite distance from each other. (B) The ordered eigenvalues correspond to the manifold lengths, illustrating the hierarchical nature of the components extracted from InPCA.

  • Fig. 5.
    Fig. 5.

    Model manifold of the six-parameter ΛCDM cosmological model predictions of temperature and polarization power spectra in the CMB using InPCA, t-SNE, and the diffusion map. Axes reflect the true aspect ratio from extracted components in all cases. Here the model manifold is colored by the primordial fluctuation amplitude, the most prominent feature in CMB data. (A) InPCA extracts, as the first and second component, this amplitude term as well as the Hubble constant. These parameters control the two most dominant features in the Planck data and so reflect a physically meaningful hierarchy of importance. In contrast, (B) t-SNE extracts only the amplitude term and (C) the diffusion map extracts the amplitude term and a different parameter, the scalar spectral index η, which reflects the scale variance of the density fluctuations in the early universe. In all plots, the orange point represents our universe, as represented by Planck 2015 data.

Data supplements

  • Supporting Information

    • Download Appendix (PDF)
    • Download Movie_S01 (GIF) - Stages of training a convolutional neural network (CNN). Each point in this animation represents one of the 10,000 handwritten digits in the MNIST test data set. Each of the ten colors represents one of the ten digits (dark blue = 0, ..., yellow = 9). The time evolution, or epoch, reflects the training of the neural network. At the initial frame, the untrained network is unable to classify images – it has a 90% error rate, just as one would expect from random guessing. As the training progresses and the error rate decreases, the digits in the cloud begin to cluster. (The network was trained on a separate set of 55,000 images, so the clustering represents success in recognizing new digits.)
Back to top
Article Alerts
Email Article
Citation Tools
Request Permissions
Visualizing probabilistic models and data with Intensive Principal Component Analysis
Katherine N. Quinn, Colin B. Clement, Francesco De Bernardis, Michael D. Niemack, James P. Sethna
Proceedings of the National Academy of Sciences Jul 2019, 116 (28) 13762-13767; DOI: 10.1073/pnas.1817218116
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo

Article Classifications

Proceedings of the National Academy of Sciences: 116 (28)
Table of Contents

Submit

Jump to section

You May Also be Interested in

Similar Articles