Volume-Preserving Feedforward Neural Network

The volume-preserving feedforward neural network presented here can be seen as an adaptation of an $LA$-SympNet to the setting when we deal with a divergence-free vector field. It also serves as the feedforward module in the volume-preserving transformer.

Neural network architecture

The constructor for VolumePreservingFeedForward produces the following architecture^[1]:

Here "LinearLowerLayer" performs

\[\mathrm{LinearLowerLayer}_{L}: x \mapsto x + Lx\]

and "NonLinearLowerLayer" performs

\[\mathrm{NonLinearLowerLayer}_{L}: x \mapsto x + \sigma(Lx + b). \]

The activation function $\sigma$ is the forth input argument to the constructor and tanh by default. We can make an instance of a VolumePreservingFeedForward neural network:

using GeometricMachineLearning

const d = 3

arch = VolumePreservingFeedForward(d)


for layer in Chain(arch)
    println(stdout, layer)
end

VolumePreservingLowerLayer{3, 3, :no_bias, typeof(identity)}(identity)
VolumePreservingUpperLayer{3, 3, :bias, typeof(identity)}(identity)
VolumePreservingLowerLayer{3, 3, :bias, typeof(tanh)}(tanh)
VolumePreservingUpperLayer{3, 3, :bias, typeof(tanh)}(tanh)
VolumePreservingLowerLayer{3, 3, :no_bias, typeof(identity)}(identity)
VolumePreservingUpperLayer{3, 3, :bias, typeof(identity)}(identity)

And we see that we get the same architecture as shown in the figure above, with the difference that the bias has been subsumed in the previous layers. Note that the nonlinear layers also contain a bias vector.

Note on Sympnets

In the general framework of feedforward neural networks SympNets are more restrictive than volume-preserving neural networks as symplecticity is a stronger property than volume-preservation:

Note however that SympNets rely on data in canonical form, i.e. data that is of $(q, p)$ type (called GeometricMachineLearning.QPT in GeometricMachineLearning), so those data need to come from a vector space $\mathbb{R}^{2n}$ of even dimension. Volume-preserving feedforward neural networks also work for odd-dimensional spaces. This is also true for transformers: the volume-preserving transformer works in spaces of arbitrary dimension $\mathbb{R}^{n\times{}T}$, whereas the linear symplectic transformer only works in even-dimensional spaces $\mathbb{R}^{2n\times{}T}$.

Library Functions

VolumePreservingFeedForward(dim)