Convolutional networks for fast, energy-efficient neuromorphic computing

Steven K. Esser, Paul A. Merolla, John V. Arthur, Andrew S. Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J. Berg, Jeffrey L. McKinstry, Timothy Melano, Davis R. Barch, Carmelo di Nolfo, Pallab Datta, Arnon Amir, Brian Taba, Myron D. Flickner, and Dharmendra S. Modha
PNAS October 11, 2016. 113 (41) 11441-11446; published ahead of print September 20, 2016. https://doi.org/10.1073/pnas.1604850113

  1. Edited by Karlheinz Meier, University of Heidelberg, Heidelberg, Germany, and accepted by Editorial Board Member J. A. Movshon August 9, 2016 (received for review March 24, 2016)

Article Figures & SI

Figures

  • Fig. 1.
    Fig. 1.

    (A) Two layers of a convolutional network. Colors (green, purple, blue, orange) designate neurons (individual boxes) belonging to the same group (partitioning the feature dimension) at the same location (partitioning the spatial dimensions). (B) A TrueNorth chip (shown far right socketed in IBM’s NS1e board) comprises 4,096 cores, each with 256 inputs, 256 neurons, and a 256 × 256 synaptic array. Convolutional network neurons for one group at one topographic location are implemented using neurons on the same TrueNorth core (TrueNorth neuron colors correspond to convolutional network neuron colors in A), with their corresponding filter support region implemented using the core’s inputs, and filter weights implemented using the core’s synaptic array. (C) Neuron dynamics showing that the internal state variable V(t) of a TrueNorth neuron changes in response to positive and negative weighted inputs. Following input integration in each tick, a spike is emitted if V(t) is greater than or equal to the threshold θ=1. V(t) is reset to 0 before input integration in the next tick. (D) Convolutional network filter weights (numbers in black diamonds) implemented using TrueNorth, which supports weights with individually configured on/off state and strength assigned by lookup table. In our scheme, each feature is represented with pairs of neuron copies. Each pair connects to two inputs on the same target core, with the inputs assigned types 1 and 2, which via the look up table assign strengths of +1 or −1 to synapses on the corresponding input lines. By turning on the appropriate synapses, each synapse pair can be used to represent −1, 0, or +1.

  • Fig. 2.
    Fig. 2.

    Dataset samples. (A) CIFAR10 examples of airplane and automobile. (B) SVHN examples of the digits 4 and 7. (C) GTSRB examples of the German traffic signs for priority road and ahead only. (D) Flickr-Logos32 examples of corporate logos for FedEx and Texaco. (E) VAD example showing voice activity (red box) and no voice activity at 0 dB SNR. (F) TIMIT examples of the phonemes pcl, p, l, ah, z (red box), en, l, and ix.

  • Fig. 3.
    Fig. 3.

    Each row shows an example image from CIFAR10 (column 1) and the corresponding output of 12 typical transduction filters (columns 2–13).

  • Fig. 4.
    Fig. 4.

    Accuracy of different sized networks running on one or more TrueNorth chips to perform inference on eight datasets. For comparison, accuracy of state-of-the-art unconstrained approaches are shown as bold horizontal lines (hardware resources used for these networks are not indicated).

Tables

  • Table 1.

    Structure of convolution networks used in this work

    1/2 chip1 chip2 chip4 chip
    S-12S-16S-32S-64
    P4-128 (4)P4-252 (2)S-128 (4)S-256 (8)
    DN-256 (2)N-128 (1)N-256 (2)
    S-256 (16)P-256 (8)P-128 (4)P-256 (8)
    N-256 (2)S-512 (32)S-256 (16)S-512 (32)
    P-512 (16)N-512 (4)N-256 (2)N-512 (4)
    S-1020 (4)N-512 (4)P-256 (8)P-512 (16)
    (6,528/class)N-512 (4)S-512 (32)S-1024 (64)
    P-512 (16)N-512 (4)N-1024 (8)
    S-1024 (64)P-512 (16)P-1024 (32)
    N-1024 (8)S-2048 (64)S-2048 (128)
    P-1024 (32)N-2048 (16)N-2048 (16)
    N-1024 (8)N-2048 (16)N-2048 (16)
    N-1024 (8)N-2048 (16)N-2048 (16)
    N-2040 (8)N-4096 (16)N-4096 (16)
    (816/class)(6,553/class)(6,553/class)

    • Each layer is described as type-features (groups), where type can be S for spatial filter layers with filter size 3×3 and stride 1, N for network-in-network layers with filter size 1×1 and stride 1, P for convolutional pooling layer with filter size 2×2 and stride 2, P4 for convolutional pooling layer with filter size 4×4 and stride 2, and D for dropout layers. The number of output features assigned to each of the 10 CIFAR10 classes is indicated below the final layer as (features/class). The eight-chip network is the same as a four-chip network with twice as many features per layer.

  • Table 2.

    Summary of datasets

    DatasetClassesInputDescription
    CIFAR10 (26)1032 row × 32 column × 3 RGBNatural and manufactured objects in their environment
    CIFAR100 (26)10032 row × 32 column × 3 RGBNatural and manufactured objects in their environment
    SVHN (27)1032 row × 32 column × 3 RGBSingle digits of house addresses from Google’s Street View
    GTSRB (28)4332 row × 32 column × 3 RGBGerman traffic signs in multiple environments
    Flickr-Logos32 (29)3232 row × 32 column × 3 RGBLocalized corporate logos in their environment
    VAD (30, 31)216 sample × 26 MFCCVoice activity present or absent, with noise (TIMIT + NOISEX)
    TIMIT class (30).3932 sample × 16 MFCC × 3 deltaPhonemes from English speakers, with phoneme boundaries
    TIMIT frame (30)3916 sample × 39 MFCCPhonemes from English speakers, without phoneme boundaries

    • GTSRB and Flickr-Logos32 are cropped and/or downsampled from larger images. VAD and TIMIT datasets have Mel-frequency cepstral coefficients (MFCC) computed from 16-kHz audio data.

  • Table 3.

    Summary of results

    DatasetState of the artTrueNorth best accuracyTrueNorth 1 chip
    ApproachAccuracyAccuracy#coresAccuracy#coresFPSmWFPS/W
    CIFAR10CNN (11)91.73%89.32%3149283.41%40421249204.46108.6
    CIFAR100CNN (34)65.43%65.48%3149255.64%40421526207.87343.7
    SVHNCNN (34)98.08%97.46%3149296.66%40422526256.59849.9
    GTSRBCNN (35)99.46%97.21%3149296.50%40421615200.68051.8
    LOGO32CNN93.70%90.39%1360685.70%32361775171.710335.5
    VADMLP (36)95.00%97.00%175895.42%423153926.159010.7
    TIMIT Class.HGMM (37)83.30%82.18%880279.16%19432610142.618300.1
    TIMIT FramesBLSTM (38)72.10%73.46%2003871.17%24762107165.912698.0

    • The network for LOGO32 was an internal implementation. BLSTM, bidirectional long short-term memory; CNN, convolutional neural network; FPS, frames/second; FPS/W, fames/second per Watt; HGMM, hierarchical Gaussian mixture model; MLP, multilayer perceptron. Accuracy of TrueNorth networks is shown in bold.

Supporting information

Back to top
Article Alerts
Email Article
Citation Tools
Request Permissions
CNNs for energy-efficient neuromorphic computing
Steven K. Esser, Paul A. Merolla, John V. Arthur, Andrew S. Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J. Berg, Jeffrey L. McKinstry, Timothy Melano, Davis R. Barch, Carmelo di Nolfo, Pallab Datta, Arnon Amir, Brian Taba, Myron D. Flickner, Dharmendra S. Modha
Proceedings of the National Academy of Sciences Oct 2016, 113 (41) 11441-11446; DOI: 10.1073/pnas.1604850113
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo

You May Also be Interested in

In the midst of a microscopy boom, stem cell researchers are coming up with inventive techniques for capturing live images of these enigmatic cells in their native habitats. Image courtesy of Panteleimon Rompolas (University of Pennsylvania, Philadelphia).
Inner Workings: Microscopy lights up stem cells in action
In the midst of a microscopy boom, stem cell researchers are coming up with inventive techniques for capturing live images of these enigmatic cells in their native habitats.
Image courtesy of Panteleimon Rompolas (University of Pennsylvania, Philadelphia).
Sixty-six million years ago, an asteroid wiped out a huge swath of life on planet Earth. But could this and similar impacts have helped kick-start life itself? Image courtesy of Detlev van Ravenswaay/ScienceSource.
News Feature: Life after the asteroid apocalypse
Sixty-six million years ago, an asteroid wiped out a huge swath of life on planet Earth. But could this and similar impacts have helped kick-start life itself?
Image courtesy of Detlev van Ravenswaay/ScienceSource.
PNAS QnAs with Joanne Chory, winner of the Breakthrough Prize in life sciences.
PNAS QnAs
PNAS QnAs with Joanne Chory, winner of the Breakthrough Prize in life sciences.
Proceedings of the National Academy of Sciences: 115 (25)
Current Issue

Submit

Jump to section