Neural network interpretation using descrambler groups

Significance Artificial neural networks are famously opaque—it is often unclear how they work. In this communication, we propose a group-theoretical way of finding out. It reveals considerable internal sophistication, even in simple neural networks: our nets apparently invented an elegant digital filter, a regularized integral transform, and even Chebyshev polynomials. This is a step toward saving reductionism. For centuries, the philosophical approach to science has been to find fundamental laws that govern reality, to test those laws, and to use their predictive power. Black-box neural networks amount to blasphemy within that school, but they are irresistible because they “just work.” Explaining how they work is a notoriously difficult problem, to which this paper offers a partial solution.


S1. Gradient of the Tikhonov descrambling functional
The gradient of the functional in Equation (8) of the main text: (S.1) may be obtained using matrix differentiation rules. The weighted array of column vector signals arriving from the preceding layers will be abbreviated as: With the result that the descrambling functional at layer k becomes: where we chose D to be a second Fourier derivative (1) matrix. Using the chain rule: The derivative of P with respect to an element of Q is another instance of the chain rule:  This eliminates all derivatives and all explicit sums from the right-hand side: The explicit sum can now be collapsed: The derivative of  with respect to P is obtained using the Frobenius norm differentiation rule: The final result is: Numerical evaluation of both the functional and the gradient may be accelerated by pre-computing the terms enclosed in square brackets. Convergence criteria for training and descrambling must be sufficiently tight to suppress the numerical noise from the random numbers that are often used as the starting point in the neural network training process. Because smoothness is used as a criterion here, descrambling would fail if the intermediate signals cannot be made simultaneously smooth.
At the extremum, the descrambling functional in Equation (S.1) is stable with respect to small perturbations in Q by definition, because at extremum . Frobenius norm is submultiplicative, and therefore: where  is a finite positive real number. Thus, T  is also stable with respect to the input data: small variations in X are guaranteed to yield proportionally small variations in the interpretability score.

S2. Gradients of maximum diagonality descrambling functionals
The gradients of the two functionals (maximum diagonal sum and maximum diagonal norm squared) in Equation (14) are obtained in a similar way to the above: denotes element-wise multiplication.

S3. DEERNet topology and training
A training database containing 10 5 DEER traces was generated as we previously described (2); networks of the following general topology were trained using scaled conjugate gradient backpropagation on an NVidia Titan V card using Matlab R2020a Machine Learning Toolbox (3); technical details may be found in the same paper (2). The input layer and the inner layers of DEERNet use tangent sigmoidal activation functions, and the output layer uses the log-sigmoidal activation function to ensure that the output cannot become negative. The dimension of the inner layer weight matrices is chosen based on the weight matrix rank analysis (2).

S4. The extra layer of a deeper DEERNet
Descrambling the DEERNet featuring three fully connected layers revealed that the first layer applies a similar combination of digital filters to those shown in Figure 2 of the main text, and the last layer is a time-distance transform similar to the one in Figures 3 and 4. However, the middle fully connected layer turned out to have unexpected mathematical depth.
The descrambled weight matrix appears to be inverting the phases of the harmonics that it receives ( Figure S3). However, this inversion is frequency-dependent: within the pass band of the digital filter in the previous fully connected layer, the matrix in Figure S4 is antidiagonal for some frequencies, but diagonal for others. Initially, this is puzzlingup to some tenacious noise that had survived the training, this corresponds to trivial phase rotations and inversions. The essential need for phase inversions becomes clear once the group-theoretical nature of the DEER signal processing problem is considered. The integral transform in Eq (9) of the main text is, in some Euclidean metric, orthogonal. For input vectors of dimension n , the corresponding group is ( ) On, and we are asking the network to approximate a particular element of this group. However, ( ) On is not a path-connected manifold because it has a discrete factor: n is the special orthogonal group (which is path-connected), and I is the inversion group.
Loosely speaking, there are two disconnected "copies" of ( )

SO n in ( )
On, and the transformation that the network needs to approximate would belong to one of these copies.
Consider the situation when the solution belongs to one instance of ( ) SO n , but the initial guess randomly drops the network into an approximation of the other instance. Because ( ) On is not path-connected, there is no way that a continuous function optimiser like gradient descent would be able jump from one copy to the otherunless the middle layer evolves an inversion component, which appears here to come embedded into a representation of ( ) 1 U group of phase rotations. The network is apparently making use of the fact is path-connected, and that: Thus the role of the middle fully connected layer in this larger DEERNet appears to be topological and group theoreticalthe layer is a bridge between two disconnected components of the orthogonal group.
The test of correctness of this hypothesis would be the sign of the determinant of the weight matrixa direct inspection confirms that the determinant is negative.

S5. Descrambling the input layer of an acoustic filter network
A fully connected neural network designed to remove additive noise from recordings of human voice  The immediate appearance of the weight matrix has no identifiable characteristics ( Figure S3). This is typical for fully connected layers. However, the input dimension is expected to have a block structure matching the eight blocks of the input vector. This is visible in the row element autocorrelation function ( Figure S4), which suggests a faint repeating pattern of 129 elements. The inspection of the weight matrix average over the eight correlated blocks reveals an elaborate frequency filter ( Figure S5): some input frequencies are mostly rejected, and linear combinations of others are sent to the output. This suggests that the function of the matrix is to attenuate undesired frequencies, meaning that a maximum diagonality descrambler (Section S2) would be appropriate here.  Thus, the descrambled weight matrix is multi-diagonal: the left panel of Figure S6 reveals that the main diagonal is applying a soft low pass filter to every STFT block, and the presence of other diagonals appearsat least algebraicallyto correspond to a weighed averaging and/or cancellation of individual frequencies between the blocks. The exact nature of what is happening here will be of interest to DSP