Neural Network Integrators

NeuralNetworkIntegrator is a super type of various neural network architectures such as SympNet and ResNet.

The purpose of such neural networks is to approximate the flow of an ordinary differential equation (ODE).

NeuralNetworkIntegrators can be seen as modeling traditional one-step methods with neural networks, i.e. for a fixed time step they perform:

\[ \mathtt{NeuralNetworkIntegrator}: z^{(t)} \mapsto z^{(t+1)},\]

to try to approximate the flow of some ODE:

\[ || \mathtt{Integrator}(z^{(t)}) - \varphi^h(z^{(t)}) || \approx \mathcal{O}(h),\]

where $\varphi^h$ is the flow map of the ODE for a time step $h$.

ResNet(dim, n_blocks, activation)

Make an instance of a ResNet.

A ResNet is a neural network that realizes a mapping of the form:

\[ x = \mathcal{NN}(x) + x,\]

so the input is again added to the output (a so-called add connection). In GeometricMachineLearning the specific ResNet that we use consists of a series of simple ResNetLayers.

Constructor

ResNet can also be called with the constructor:

ResNet(dl, n_blocks)

where dl is an instance of DataLoader.

See iterate for an example of this.

ResNetLayer(dim)

Make an instance of the resnet layer.

The ResNetLayer is a simple feedforward neural network to which we add the input after applying it, i.e. it realizes $x \mapsto x + \sigma(Ax + b)$.

Arguments

The ResNet layer takes the following arguments:

1. dim::Integer: the system dimension.
2. activation = identity: The activation function.

The following is a keyword argument:

• use_bias::Bool = true: This determines whether a bias $b$ is used.

iterate(nn, ics)

This function computes a trajectory for a NeuralNetworkIntegrator that has already been trained for valuation purposes.

It takes as input:

1. nn: a NeuralNetwork (that has been trained).
2. ics: initial conditions (a NamedTuple of two vectors)

Examples

To demonstrate iterate we use a simple ResNet that does:

\[\mathrm{ResNet}: x \mapsto \begin{pmatrix} 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 1\end{pmatrix}x + \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}\]

and we iterate three times with

\[ \mathtt{ics} = \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix}.\]

using GeometricMachineLearning

model = ResNet(3, 0, identity)
weight = [1 0 0; 0 2 0; 0 0 1]
bias = [0, 0, 1]
ps = NeuralNetworkParameters((L1 = (weight = weight, bias = bias), ))
nn = NeuralNetwork(model, Chain(model), ps, CPU())

ics = [1, 1, 1]
iterate(nn, ics; n_points = 4)

# output

3×4 Matrix{Int64}:
 1  2  4   8
 1  3  9  27
 1  3  7  15

Arguments

The optional keyword argument is

• n_points = 100

The number of integration steps that should be performed.

TransformerIntegrator <: Architecture

Encompasses various transformer architectures, such as the VolumePreservingTransformer and the LinearSymplecticTransformer.

The central idea behind this is to construct an explicit multi-step integrator:

\[ \mathtt{Integrator}: [ z^{(t - \mathtt{sl} + 1)}, z^{(t - \mathtt{sl} + 2)}, \ldots, z^{(t)} ] \mapsto [ z^{(t + 1)}, z^{(t + 2)}, \ldots, z^{(t + \mathtt{pw})} ],\]

where sl stands for sequence length and pw stands for prediction window, so the numbers of input and output vectors respectively.

iterate(nn, ics)

Iterate the neural network of type TransformerIntegrator for initial conditions ics.

The initial condition is a matrix $\in\mathbb{R}^{n\times\mathtt{seq\_length}}$ or NamedTuple of two matrices).

This function computes a trajectory for a Transformer that has already been trained for valuation purposes.

Parameters

The following are optional keyword arguments:

• n_points::Int=100: The number of time steps for which we run the prediction.
• prediction_window::Int=size(ics.q, 2): The prediction window (i.e. the number of steps we predict into the future) is equal to the sequence length (i.e. the number of input time steps) by default.