GitHub

Reading and writing HDF5

Loading HDF5 brings in the storage methods through a package extension, so the binary chain behind HDF5 is only paid for by users who ask for it. The four entry points are not exported — save and load are names that would collide with almost anything — so they are reached as NeuralNetworkParameters.save, or imported explicitly.

NeuralNetworkParameters.save — Function
save(h5, ps)
save(filename, ps)

Write a parameter set to an open HDF5 store or to a file. Implemented in the HDF5Ext extension.

source
NeuralNetworkParameters.load — Function
load(NetworkParameters, h5)
load(NetworkParameters, h5, prototype)

Read a parameter set back. Implemented in the HDF5Ext extension.

The three-argument form rebuilds the leaves against prototype, a parameter set of the right shape, and is the form that needs no registration — rebuild has something to work from. The two-argument form has to reconstruct structured leaves from what the file says about them, which is what register_parameter_type! is for.

source

Round trip

using NeuralNetworkParameters, HDF5
using NeuralNetworkParameters: save, load

ps = NetworkParameters((L1 = (W = [1.0 2.0], b = [3.0]),))
save("parameters.h5", ps)
load(NetworkParameters, "parameters.h5") == ps      # true

The element type is preserved: a Float32 network reloads as Float32.

Key order is recorded

HDF5 hands back the members of a group sorted, so a network with ten layers would read back as L1, L10, L2, … — and since key order is part of the identity of a parameter set, that set no longer compares equal to the one that was written. Guessing the order back from the names, by sorting on a trailing integer, only works for names that happen to look like L10.

The writer therefore records the order explicitly, as an attribute on each group, and the reader uses it. Files that predate this — see below — still fall back to the guess, because for them there is nothing better available.

Structured parameters

A structured leaf is written as a group holding its freeparameters and its parameter_metadata — the latter only when there is any, so a type that keeps everything in its storage, as a manifold element does, writes the storage alone. Reading it back needs to know how to reconstruct the type, and there are two ways to supply that.

Against a prototype. Pass a parameter set of the right shape and the leaves are rebuilt with rebuild, exactly as unflattening does. Nothing needs registering. Where the architecture is to hand — and when loading a network's parameters it usually is — this is the simpler path:

load(NetworkParameters, "parameters.h5", prototype)

From the registry. For a file that must be readable on its own, the package owning the type registers a reconstructor:

NeuralNetworkParameters.register_parameter_type! — Function
register_parameter_type!(name, reconstruct)

Teach load how to rebuild a structured leaf that was stored under name, when there is no prototype to rebuild against.

reconstruct(storage, metadata) receives what freeparameters produced and the parameter_metadata that went with it, and returns the leaf:

NeuralNetworkParameters.parameter_metadata(A::SymmetricMatrix) = (n = A.n,)
register_parameter_type!("SymmetricMatrix", (S, md) -> SymmetricMatrix(S, md.n))

A package that owns a parameter type registers it in its own __init__. Nothing here needs to know the type exists, which is the point: with the structured matrix types living upstream of the package that trains with them, the only way to serialise them without somebody committing type piracy is for the serialiser to be driven by a protocol rather than by a list.

The registry is only consulted by the two-argument load; the prototype form bypasses it.

source
NeuralNetworkParameters.parameter_metadata — Function
parameter_metadata(x)

The non-differentiable fields of a leaf, as a NamedTuple, for the benefit of storage formats that have no prototype to rebuild against.

Empty by default. rebuild recovers this information from its prototype, which is enough for flattening — but a parameter set read back from a file has no prototype, so a type whose storage does not determine it (the n of a SymmetricMatrix does follow from length(S); the N and n of a StiefelLieAlgHorMatrix do not) has to write it out. See register_parameter_type!.

source
NeuralNetworkParameters.parameter_metadata(A::SymmetricMatrix) = (n = A.n,)

register_parameter_type!("SymmetricMatrix", (S, md) -> SymmetricMatrix(S, md.n))

Loading a file with an unregistered type raises an error naming the type and both remedies, rather than failing obscurely.

Older files

Parameter files written before this package still load:

  • those written by AbstractNeuralNetworks, which are plain nested groups with no attributes;
  • those written by GeometricMachineLearning, whose structured matrices carry a gml_type attribute. Such a group holds the type's fields under their own names and no storage for rebuild to take, so the registry is the only way in: the type has to have been registered, and a prototype is no substitute. Passing one says so rather than failing obscurely.

The group's fields reach the registered reconstructor in both argument positions, storage and metadata alike, since a file in that layout records nothing that could tell them apart. A reconstructor that means to read these files sorts that out itself — and reaches for its fields by name, the layout having recorded no key order either.