Volume-Preserving Feedforward Layer

The volume-preserving feedforward layers in GeometricMachineLearning are closely related to SympNet layers. We note that [50] proposes more complicated volume-preserving feedforward neural networks than the ones presented here. These are motivated by a classical theorem [51]; but for reasons of simplicity and their similarity to SympNet layers we resort to simpler ones.

Volume-preserving feedforward layers in GeometricMachineLearning are a special type of ResNet layer for which we restrict the weight matrices to be of a particular form. Each layer computes:

\[\mathtt{VPFF}_{A, b}: x \mapsto x + \sigma(Ax + b),\]

where $\sigma$ is a nonlinearity, $A$ is the weight and $b$ is the bias. The matrix $A$ is either a LowerTriangular matrix $L$ or an UpperTriangular matrix $U$. We demonstrate volume-preservation of these layers by considering the case $A = L$. The matrix looks as follows:

\[L = \begin{pmatrix} 0 & 0 & \cdots & 0 \\ a_{21} & \ddots & & \vdots \\ \vdots & \ddots & \ddots & \vdots \\ a_{n1} & \cdots & a_{n(n-1)} & 0 \end{pmatrix}.\]

For the jacobian we then have:

\[J = \nabla\mathtt{VPFF}_{L, b} = \begin{pmatrix} 1 & 0 & \cdots & 0 \\ b_{21} & \ddots & & \vdots \\ \vdots & \ddots & \ddots & \vdots \\ b_{n1} & \cdots & b_{n(n-1)} & 1 \end{pmatrix},\]

and the determinant of $J$ is 1, i.e. the map is volume-preserving. A similar statement holds if the matrix $A$ is UpperTriangular instead of LowerTriangular.

Library Functions

GeometricMachineLearning.VolumePreservingFeedForwardLayer — Type

VolumePreservingFeedForwardLayer <: AbstractExplicitLayer

Super-type of VolumePreservingLowerLayer and VolumePreservingUpperLayer. The layers do the following:

\[x \mapsto \begin{cases} \sigma(Lx + b) & \text{where $L$ is }\mathtt{LowerTriangular}, \\ \sigma(Ux + b) & \text{where $U$ is }\mathtt{UpperTriangular}. \end{cases}\]

The functor can be applied to a vector, a matrix or a tensor. The special matrices are implemented as LowerTriangular and UpperTriangular.

source

GeometricMachineLearning.VolumePreservingLowerLayer — Type

VolumePreservingLowerLayer(dim)

Make an instance of VolumePreservingLowerLayer for a specific system dimension.

See the documentation for VolumePreservingFeedForwardLayer.

Examples

We apply the VolumePreservingLowerLayer with matrix:

\[L = \begin{pmatrix} 0 & 0 \\ 1 & 0 \end{pmatrix}\]

to a vector:

\[x = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\]

using GeometricMachineLearning

l = VolumePreservingLowerLayer(2, identity; use_bias=false)
A = LowerTriangular([1,], 2)
ps = (weight = A, )
x = ones(eltype(A), 2)

l(x, ps)

# output

2×1 Matrix{Int64}:
 1
 2

Arguments

The constructor can be called with the optional arguments:

activation=tanh: the activation function.
use_bias::Bool=true (keyword argument): specifies whether a bias should be used.

source

GeometricMachineLearning.VolumePreservingUpperLayer — Type

VolumePreservingUpperLayer(dim)

Make an instance of VolumePreservingUpperLayer for a specific system dimension.

See the documentation for VolumePreservingFeedForwardLayer.

Examples

We apply the VolumePreservingUpperLayer with matrix:

\[U = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}\]

to a vector:

\[x = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\]

using GeometricMachineLearning

l = VolumePreservingUpperLayer(2, identity; use_bias=false)
A = UpperTriangular([1,], 2)
ps = (weight = A, )
x = ones(eltype(A), 2)

l(x, ps)

# output

2×1 Matrix{Int64}:
 2
 1

Arguments

The constructor can be called with the optional arguments:

activation=tanh: the activation function.
use_bias::Bool=true (keyword argument): specifies whether a bias should be used.

source