Losses and Errors
In general we distinguish between losses that are used during training of a neural network and errors that arise in the context of reduced order modeling.
Different Neural Network Losses
GeometricMachineLearning has a number of loss functions implemented that can be called standard losses. Those are the FeedForwardLoss, the TransformerLoss, the AutoEncoderLoss and the ReducedLoss. How to implement custom losses is shown in a tutorial.
A Note on Physics-Informed Neural Networks
A popular trend in recent years has been considering known physical properties of the differential equation, or the entire differential equation, through the loss function [71]. This is one way of considering physical properties, and GeometricMachineLearning allows for a flexible implementation of custom losses, but this is nonetheless discouraged. In general a neural networks consists of three ingredients:
Instead of considering certain properties through the loss function, we instead do so by enforcing them strongly through the network architecture and the optimizer; the latter pertains to manifold optimization. The advantages of this approach are the strong enforcement of properties that we know our network should have and much easier training because we do not have to tune hyperparameters.
Projection and Reduction Errors of Reduced Models
Two errors that are of very big importance in reduced order modeling are the projection and the reduction error. During training one typically aims at minimizing the projection error, but for the actual application of the model the reduction error is often more important.
Projection Error
The projection error computes how well a reduced basis, represented by the reduction $\mathcal{P}$ and the reconstruction $\mathcal{R}$, can represent the data with which it is build. In mathematical terms:
\[ e_\mathrm{proj}(\mu) := \frac{|| \mathcal{R}\circ\mathcal{P}(M) - M ||}{|| M ||},\]
where $||\cdot||$ is the Frobenius norm (one could also optimize for different norms). The corresponding function in GeometricMachineLearning is projection_error. The projection error is equivalent to AutoEncoderLoss and is used for training under that name.
Reduction Error
The reduction error measures how far the reduced system diverges from the full-order system during integration (online stage). In mathematical terms (and for a single initial condition):
\[e_\mathrm{red}(\mu) := \sqrt{ \frac{\sum_{t=0}^K|| \mathbf{x}^{(t)}(\mu) - \mathcal{R}(\mathbf{x}^{(t)}_r(\mu)) ||^2}{\sum_{t=0}^K|| \mathbf{x}^{(t)}(\mu) ||^2} },\]
where $\mathbf{x}^{(t)}$ is the solution of the FOM at point $t$ and $\mathbf{x}^{(t)}_r$ is the solution of the ROM (in the reduced basis) at point $t$. The reduction error, as opposed to the projection error, not only measures how well the solution manifold is represented by the reduced basis, but also measures how well the FOM dynamics are approximated by the ROM dynamics (via the induced vector field on the reduced basis). The corresponding function in GeometricMachineLearning is reduction_error. The reduction error is, in contract to the projection error, typically not used during training (even though some authors are using a similar error to do so [62]).
Library Functions
GeometricMachineLearning.FeedForwardLoss — TypeFeedForwardLoss()Make an instance of a loss for feedforward neural networks.
This should be used together with a neural network of type NeuralNetworkIntegrator.
Example
FeedForwardLoss applies a neural network to an input and compares it to the output via an $L_2$ norm:
using GeometricMachineLearning
using LinearAlgebra: norm
import Random
Random.seed!(123)
const d = 2
arch = GSympNet(d)
nn = NeuralNetwork(arch)
input_vec = [1., 2.]
output_vec = [3., 4.]
loss = FeedForwardLoss()
loss(nn, input_vec, output_vec) ≈ norm(output_vec - nn(input_vec)) / norm(output_vec)
# output
trueSo FeedForwardLoss simply does:
\[ \mathtt{loss}(\mathcal{NN}, \mathtt{input}, \mathtt{output}) = || \mathcal{NN}(\mathtt{input}) - \mathtt{output} || / || \mathtt{output}||,\]
where $||\cdot||$ is the $L_2$ norm.
Parameters
This loss does not have any parameters.
GeometricMachineLearning.TransformerLoss — TypeTransformerLoss(seq_length, prediction_window)Make an instance of the transformer loss.
This should be used together with a neural network of type TransformerIntegrator.
Example
TransformerLoss applies a neural network to an input and compares it to the output via an $L_2$ norm:
using GeometricMachineLearning
using LinearAlgebra: norm
import Random
const d = 2
const seq_length = 3
const prediction_window = 2
Random.seed!(123)
arch = StandardTransformerIntegrator(d)
nn = NeuralNetwork(arch)
input_mat = [1. 2. 3.; 4. 5. 6.]
output_mat = [1. 2.; 3. 4.]
loss = TransformerLoss(seq_length, prediction_window)
# start of prediction
const sop = seq_length - prediction_window + 1
loss(nn, input_mat, output_mat) ≈ norm(output_mat - nn(input_mat)[:, sop:end]) / norm(output_mat)
# output
trueSo TransformerLoss simply does:
\[ \mathtt{loss}(\mathcal{NN}, \mathtt{input}, \mathtt{output}) = || \mathcal{NN}(\mathtt{input})[(\mathtt{sl} - \mathtt{pw} + 1):\mathtt{end}] - \mathtt{output} || / || \mathtt{output} ||,\]
where $||\cdot||$ is the $L_2$ norm.
Parameters
The prediction_window specifies how many time steps are predicted into the future. It defaults to the value specified for seq_length.
GeometricMachineLearning.AutoEncoderLoss — TypeAutoEncoderLoss()Make an instance of AutoEncoderLoss.
This loss should always be used together with a neural network of type AutoEncoder (and it is also the default for training such a network).
Example
AutoEncoderLoss applies a neural network to an input and compares it to the output via an $L_2$ norm:
using GeometricMachineLearning
using LinearAlgebra: norm
import Random
Random.seed!(123)
const N = 4
const n = 1
arch = SymplecticAutoencoder(2*N, 2*n)
nn = NeuralNetwork(arch)
input_vec = [1., 2., 3., 4., 5., 6., 7., 8.]
loss = AutoEncoderLoss()
loss(nn, input_vec) ≈ norm(input_vec - nn(input_vec)) / norm(input_vec)
# output
trueSo AutoEncoderLoss simply does:
\[ \mathtt{loss}(\mathcal{NN}, \mathtt{input}) = || \mathcal{NN}(\mathtt{input}) - \mathtt{input} || / || \mathtt{input} ||,\]
where $||\cdot||$ is the $L_2$ norm.
Parameters
This loss does not have any parameters.
GeometricMachineLearning.ReducedLoss — TypeReducedLoss(encoder, decoder)Make an instance of ReducedLoss based on an Encoder and a Decoder.
This loss should be used together with a NeuralNetworkIntegrator or TransformerIntegrator.
Example
ReducedLoss applies the encoder, integrator and decoder neural networks in this order to an input and compares it to the output via an $L_2$ norm:
using GeometricMachineLearning
using LinearAlgebra: norm
import Random
Random.seed!(123)
const N = 4
const n = 1
Ψᵉ = NeuralNetwork(Chain(Dense(N, n), Dense(n, n))) |> encoder
Ψᵈ = NeuralNetwork(Chain(Dense(n, n), Dense(n, N))) |> decoder
transformer = NeuralNetwork(StandardTransformerIntegrator(n))
input_mat = [1. 2.; 3. 4.; 5. 6.; 7. 8.]
output_mat = [9. 10.; 11. 12.; 13. 14.; 15. 16.]
loss = ReducedLoss(Ψᵉ, Ψᵈ)
output_prediction = Ψᵈ(transformer(Ψᵉ(input_mat)))
loss(transformer, input_mat, output_mat) ≈ norm(output_mat - output_prediction) / norm(output_mat)
# output
trueSo the loss computes:
\[\mathrm{loss}_{\mathcal{E}, \mathcal{D}}(\mathcal{NN}, \mathrm{input}, \mathrm{output}) = ||\mathcal{D}(\mathcal{NN}(\mathcal{E}(\mathrm{input}))) - \mathrm{output}||,\]
where $\mathcal{E}$ is the Encoder, $\mathcal{D}$ is the Decoder. $\mathcal{NN}$ is the neural network we compute the loss of.
GeometricMachineLearning.projection_error — Functionprojection_error(rs)Compute the projection error for a HRedSys.
Arguments
If the full system has already been integrated, then the projection error can be computed quicker:
projection_error(rs, sol_full)This saves the cost of again integrating the systems.
GeometricMachineLearning.reduction_error — Functionreduction_error(rs)Compute the reduction error for a HRedSys.
Arguments
If the full system and the reduced system have already been integrated, then the reduction error can be computed quicker:
reduction_error(rs, sol_full, sol_reduced)This saves the cost of again integrating the respective systems.
References
- [62]
- K. Lee and K. T. Carlberg. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. Journal of Computational Physics 404, 108973 (2020).
- [71]
- M. Raissi, P. Perdikaris and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics 378, 686–707 (2019).