GitHub

NeuralNetworks in GeometricMachineLearning

GeometricMachineLearning inherits some functionality from another Julia package called AbstractNeuralNetworks. How these two packages interact is shown in the figure below for the example of the SympNet[1]:

Visualization of how the packages interact. Visualization of how the packages interact.

The red color indicates an abstract type, blue indicates a struct and orange indicates a const (derived from a struct). Solid black arrows indicate direct dependencies, i.e. we have

GradientLayer <: SympNetLayer <: AbstractExplicitLayer
GSympNet <: SympNet <: Architecture

Dashed black arrows indicate a derived neural network architecture. A GSympNet (which is an Architecture) is derived from GradientLayerQ and GradientLayerP (which are AbstractExplicitLayers) for example. An Architecture can be turned into a NeuralNetwork by calling the associated constructor

arch = GSympNet(3)
nn = NeuralNetwork(arch, CPU(), Float64)

Such a neural network has four fields:

  • architecture: the Architecture we supplied the constructor with,
  • model: a translation of the supplied architecture into specific neural network layers,
  • params: the neural network parameters,
  • backend: this indicates on which device we allocate the neural network parameters. In this case it is CPU().

We can get the associated model to GSympNet by calling:

nn.model.layers
(GradientLayerQ{3, 3, typeof(tanh)}(6, tanh), GradientLayerP{3, 3, typeof(tanh)}(6, tanh))

and we see that it consists of two layers: a GradientLayerQ and a GradientLayerP.

Saving and Loading

GeometricMachineLearning extends AbstractNeuralNetworks with HDF5-backed save and load methods for NeuralNetwork. These handle GML-specific parameter types (StiefelManifold, SymmetricMatrix, SkewSymMatrix) transparently.

AbstractNeuralNetworks.save — Method
save(h5::HDF5.H5DataStore, nn::NeuralNetwork)

Save the parameters of nn into an already-open HDF5 store.

Extends AbstractNeuralNetworks.save with a dispatch on NeuralNetwork. GML special array types (StiefelManifold, SymmetricMatrix, SkewSymMatrix, LowerTriangular, UpperTriangular) are tagged with a gml_type attribute so that load can reconstruct them faithfully.

source
AbstractNeuralNetworks.save — Method
save(filename::AbstractString, nn::NeuralNetwork)

Convenience overload: open filename for writing, then call save(h5, nn).

source
AbstractNeuralNetworks.load — Method
load(::Type{NeuralNetwork}, h5::HDF5.H5DataStore, arch::Architecture; backend=CPU())

Load network parameters from an already-open HDF5 store and return a NeuralNetwork for arch.

Extends AbstractNeuralNetworks.load with a dispatch on NeuralNetwork. GML special array types are reconstructed from their gml_type attribute. The element type is preserved as stored (Float32 files reload as Float32).

source
AbstractNeuralNetworks.load — Method
load(::Type{NeuralNetwork}, filename::AbstractString, arch::Architecture; backend=CPU())

Convenience overload: open filename for reading, then call load(NeuralNetwork, h5, arch; backend).

source
  • 1The section on SympNets also contains an explanation of all the structs and types described in this section here.