Visualizing probabilistic models and data with Intensive Principal Component Analysis

Katherine N. Quinn, Colin B. Clement, Francesco De Bernardis, Michael D. Niemack, and James P. Sethna
PNAS July 9, 2019 116 (28) 13762-13767; first published June 24, 2019 https://doi.org/10.1073/pnas.1817218116

  1. Edited by William Bialek, Princeton University, Princeton, NJ, and approved May 31, 2019 (received for review October 5, 2018)

This article requires a subscription to view the full text. If you have a subscription you may use the login form below to view the article. Access to this article can also be purchased.

Significance

We introduce Intensive Principal Component Analysis (InPCA), a widely applicable manifold-learning method to visualize general probabilistic models and data. Using replicas to tune dimensionality in high-dimensional data, we use the zero-replica limit to discover a distance metric, which preserves distinguishability in high dimensions, and an embedding with superior visualization performance. We apply InPCA to the model of cosmology which predicts the angular power spectrum of the cosmic microwave background, allowing visualization of the space of model predictions (i.e., different universes).

Abstract

Unsupervised learning makes manifest the underlying structure of data without curated training and specific problem definitions. However, the inference of relationships between data points is frustrated by the “curse of dimensionality” in high dimensions. Inspired by replica theory from statistical mechanics, we consider replicas of the system to tune the dimensionality and take the limit as the number of replicas goes to zero. The result is intensive embedding, which not only is isometric (preserving local distances) but also allows global structure to be more transparently visualized. We develop the Intensive Principal Component Analysis (InPCA) and demonstrate clear improvements in visualizations of the Ising model of magnetic spins, a neural network, and the dark energy cold dark matter (ΛCDM) model as applied to the cosmic microwave background.

Footnotes

  • ↵1To whom correspondence may be addressed. Email: knq2{at}cornell.edu.

Published under the PNAS license.

References

  1. ↵
    1. M. F. De Oliveira,
    2. H. Levkowitz
    , From visual data exploration to visual data mining: A survey. IEEE Trans. Visualization Comput. Graphics 9, 378–394 (2003).
    OpenUrl
  2. ↵
    1. S. Liu,
    2. D. Maljovec,
    3. B. Wang,
    4. P. T. Bremer,
    5. V. Pascucci
    , Visualizing high-dimensional data: Advances in the past decade. IEEE Trans. Visualization Comput. Graphics 23, 1249–1268 (2017).
    OpenUrl
  3. ↵
    1. J. A. Lee,
    2. M. Verleysen
    , Nonlinear Dimensionality Reduction (Springer, New York, NY, 2007).
  4. ↵
    1. A. Zimek,
    2. E. Schubert,
    3. H. P. Kriegel
    , A survey on unsupervised outlier detection in high-dimensional numerical data. Stat. Anal. Data Mining ASA Data Sci. J. 5, 363–387 (2012).
    OpenUrl
  5. ↵
    1. K. P. Murphy
    , Machine Learning: A Probabilistic Perspective (The MIT Press, 2012).
  6. ↵
    1. H. P. Kriegel,
    2. P. Kröger,
    3. A. Zimek
    , Clustering high-dimensional data: A survey on subspace clustering, pattern-based clustering, and correlation clustering. ACM Trans. Knowl. Discov. Data 3, 1–58 (2009).
    OpenUrl
  7. ↵
    1. H. Hotelling
    , Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 24, 417–441 (1933).
    OpenUrlCrossRef
  8. ↵
    1. W. S. Torgerson
    , Multidimensional scaling: I. Theory and method. Psychometrika 17, 401–419 (1952).
    OpenUrlCrossRef
  9. ↵
    1. L. van derMaaten,
    2. G. Hinton
    , Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).
    OpenUrlCrossRefPubMed
  10. ↵
    1. R. R. Coifman et al.
    , Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. Proc. Natl. Acad. Sci. U.S.A. 102, 7426–7431 (2005).
    OpenUrlAbstract/FREE Full Text
  11. ↵
    1. L. McInnes,
    2. J. Healy,
    3. J. Melville
    , Umap: Uniform manifold approximation and projection for dimension reduction. arXiv:1802.03426 (6 December 2018).
  12. ↵
    1. M. Mézard,
    2. G. Parisi,
    3. M. Virasoro
    , Spin Glass Theory and Beyond (World Scientific, 1986).
  13. ↵
    1. B. B. Machta,
    2. R. Chachra,
    3. M. K. Transtrum,
    4. J. P. Sethna
    , Parameter space compression underlies emergent theories and predictive models. Science 342, 604–607 (2013).
    OpenUrlAbstract/FREE Full Text
  14. ↵
    1. M. K. Transtrum,
    2. P. Qiu
    , Model reduction by manifold boundaries. Phys. Rev. Lett. 113, 098701 (2014).
    OpenUrlCrossRef
  15. ↵
    1. M. K. Transtrum et al.
    , Perspective: Sloppiness and emergent theories in physics, biology, and beyond. J. Chem. Phys. 143, 010901 (2015).
    OpenUrlCrossRef
  16. ↵
    1. E. Hellinger
    , Neue Begründung der Theorie quadratischer Formen von unendlichvielen Veränderlichen. J. Reine Angew. Math. 136, 210–271(1909).
    OpenUrl
  17. ↵
    1. M. Gromov
    , In a search for a structure, part 1: On entropy. Entropy 17, 1273–1277 (2013).
    OpenUrl
  18. ↵
    1. S. Amari,
    2. H. Nagaoka
    , Translations of Mathematical Monographs: Methods of Information Geometry (Oxford University Press, 2000), vol. 191.
  19. ↵
    1. K. Beyer,
    2. J. Goldstein,
    3. R. Ramakrishnan,
    4. U. Shaft
    , “When is “nearest neighbor” meaningful?” in Database Theory— ICDT’99, C. Beeri, P. Buneman, Eds. (Springer Berlin Heidelberg, Berlin, Heidelberg, Germany, 1999), pp. 217–235.
  20. ↵
    1. G. Parisi
    , Infinite number of order parameters for spin-glasses. Phys. Rev. Lett. 43, 1754–1756 (1979).
    OpenUrlCrossRef
  21. ↵
    1. A. Bhattacharyya
    , On a measure of divergence between two multinomial populations. Sankhyā Indian J. Stat. (1933-1960) 7, 401–406 (1946).
    OpenUrl
  22. ↵
    1. M. Abadi et al.
    , TensorFlow: Large-scale machine learning on heterogeneous systems. https://www.tensorflow.org/. Accessed 1 December 2017.
  23. ↵
    1. Y. LeCun,
    2. C. Cortes,
    3. C. J. Burges
    , “MNIST database”. http://yann.lecun.com/exdb/mnist/. Accessed 1 December 2017.
  24. ↵
    1. C. M. Bishop
    , Pattern Recognition and Machine Learning (Springer, New York, NY, 2006).
  25. ↵
    1. Planck Collaboration
    , Planck 2015 results - i. Overview of products and scientific results. A&A 594, A1 (2016).
    OpenUrl
  26. ↵
    1. A. Lewis,
    2. A. Challinor,
    3. A. Lasenby
    , Efficient computation of cosmic microwave background anisotropies in closed Friedmann-Robertson-Walker models. Astrophys. J. 538, 473–476 (2000).
    OpenUrlCrossRef
  27. ↵
    1. W. Hu
    , CMB tutorials. http://background.uchicago.edu/. Accessed 1 August 2018.
  28. ↵
    1. K. Quinn
    , Ising Model Manifold. GitHub. https://github.com/katnquinn/Ising_ModelManifold. Deposited 23 July 2018.
  29. ↵
    1. K. Quinn
    , Intensive Embedding. GitHub. https://github.com/katnquinn/IntensiveEmbedding. Deposited 11 March 2019.
  30. ↵
    1. K. Quinn
    , 1 Spin. GitHub. https://github.com/katnquinn/1Spin. Deposited 13 March 2019.
View Full Text

Purchase access

You may purchase access to this article. This will require you to create an account if you don't already have one.
Back to top
Article Alerts
Email Article
Citation Tools
Request Permissions
Visualizing probabilistic models and data with Intensive Principal Component Analysis
Katherine N. Quinn, Colin B. Clement, Francesco De Bernardis, Michael D. Niemack, James P. Sethna
Proceedings of the National Academy of Sciences Jul 2019, 116 (28) 13762-13767; DOI: 10.1073/pnas.1817218116
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
Proceedings of the National Academy of Sciences: 116 (38)
Current Issue

Submit

Article Classifications

Jump to section

You May Also be Interested in

Habituation and training of the ravens in the experimental compartments, Haidlhof Research Station, Austria. Image courtesy of Jessie E. C. Adriaense.
Shared emotional states in ravens
Researchers report negative emotional contagion, or the coordination of emotional states between multiple individuals, in common ravens, suggesting convergent emotional evolution in birds and mammals.
Image courtesy of Jessie E. C. Adriaense.
Fishing fleet. Image courtesy of Pixabay/3282700.
Increase in global fishing fleets
A study examines trends in global fishing fleets and finds that by 2015, 68% of the global fishing fleet became motorized, and that the overall number of fleet vessels increased to 3.7 million, despite a consistent decrease in the catch per unit of effort.
Image courtesy of Pixabay/3282700.
Magnified views of fingerprints found on fragments of 11th-century AD jars from the U S Southwest enabled archaeologists to determine the sex of the potter. Image courtesy of John Kantner.
Fingerprints reveal gender roles in ancient society
A method to determine gender from fingerprints suggests pottery making was not a primarily female activity in ancient Puebloan society, challenging previous assumptions about gendered divisions of labor in ancient societies.
Image courtesy of John Kantner.

Similar Articles