Volume-Preserving Feedforward Neural Network

The volume-preserving feedforward neural network presented here can be seen as an adaptation of an $LA$-SympNet to the setting when we deal with a divergence-free vector field. It also serves as the feedforward module in the volume-preserving transformer.

Neural network architecture

The constructor for VolumePreservingFeedForward produces the following architecture^[1]:

Visualization of how the keywords in the constructor are interpreted.

Here "LinearLowerLayer" performs

\[\mathrm{LinearLowerLayer}_{L}: x \mapsto x + Lx\]

and "NonLinearLowerLayer" performs

\[\mathrm{NonLinearLowerLayer}_{L}: x \mapsto x + \sigma(Lx + b). \]

The activation function $\sigma$ is the forth input argument to the constructor and tanh by default. We can make an instance of a VolumePreservingFeedForward neural network:

using GeometricMachineLearning

const d = 3

arch = VolumePreservingFeedForward(d)


for layer in Chain(arch)
    println(stdout, layer)
end

VolumePreservingLowerLayer{3, 3, :no_bias, typeof(identity)}(identity)
VolumePreservingUpperLayer{3, 3, :bias, typeof(identity)}(identity)
VolumePreservingLowerLayer{3, 3, :bias, typeof(tanh)}(tanh)
VolumePreservingUpperLayer{3, 3, :bias, typeof(tanh)}(tanh)
VolumePreservingLowerLayer{3, 3, :no_bias, typeof(identity)}(identity)
VolumePreservingUpperLayer{3, 3, :bias, typeof(identity)}(identity)

And we see that we get the same architecture as shown in the figure above, with the difference that the bias has been subsumed in the previous layers. Note that the nonlinear layers also contain a bias vector.

Note on Sympnets

In the general framework of feedforward neural networks SympNets are more restrictive than volume-preserving neural networks as symplecticity is a stronger property than volume-preservation:

Symplectic neural networks are a more restrictive class of architectures than volume-preserving ones. But by construction they only work in even dimensions.

Note however that SympNets rely on data in canonical form, i.e. data that is of $(q, p)$ type (called GeometricMachineLearning.QPT in GeometricMachineLearning), so those data need to come from a vector space $\mathbb{R}^{2n}$ of even dimension. Volume-preserving feedforward neural networks also work for odd-dimensional spaces. This is also true for transformers: the volume-preserving transformer works in spaces of arbitrary dimension $\mathbb{R}^{n\times{}T}$, whereas the linear symplectic transformer only works in even-dimensional spaces $\mathbb{R}^{2n\times{}T}$.

Library Functions

GeometricMachineLearning.VolumePreservingFeedForward — Type

VolumePreservingFeedForward(dim)

Make an instance of a volume-preserving feedforward neural network for a specific system dimension.

This architecture is a composition of VolumePreservingLowerLayer and VolumePreservingUpperLayer.

Arguments

You can provide the constructor with the following additional arguments:

n_blocks::Int = 1: The number of blocks in the neural network (containing linear layers and nonlinear layers).
n_linear::Int = 1: The number of linear VolumePreservingLowerLayers and VolumePreservingUpperLayers in one block.
activation = tanh: The activation function for the nonlinear layers in a block.

The following is a keyword argument:

init_upper::Bool = false: Specifies if the first layer is lower or upper.

source

1Based on the input arguments n_linear and n_blocks. In this example init_upper is set to false, which means that the first layer is of type lower followed by a layer of type upper.