The two representations

Structured

NeuralNetworkParameters.NetworkParameters — Type

NetworkParameters(params::NamedTuple)
NetworkParameters{T}(params::NamedTuple)
NetworkParameters{T, Keys, ValueTypes}(params::NamedTuple)

The parameters of a neural network: a NamedTuple whose entries follow the architecture, wrapped in a type of its own.

The wrapper exists so that the parameters of a network are a type somebody owns. A bare NamedTuple belongs to Base, so a package that wants to give the parameter set its own behaviour — saving it to file, flattening it, stepping an optimizer over it — has to write methods on a signature in which it owns nothing. Every such method is type piracy, and two packages doing it can silently disagree.

Entries are reached with getproperty or getindex; the underlying NamedTuple is params.

Examples

using NeuralNetworkParameters

ps = NetworkParameters((L1 = (W = [1.0 2.0], b = [3.0]), L2 = (W = [4.0;;],)))
ps.L1.W

# output

1×2 Matrix{Float64}:
 1.0  2.0

Both forms of access agree, and keys reports the layers:

using NeuralNetworkParameters

ps = NetworkParameters((L1 = (W = [1.0 2.0], b = [3.0]), L2 = (W = [4.0;;],)))
(ps[:L2] === ps.L2, keys(ps))

# output

(true, (:L1, :L2))

Element type

The first type parameter is the element type the leaves promote to, so that T binds in a method signature:

f(ps::NetworkParameters{T}) where {T} = ...
g(x::Union{AbstractVector{T}, NetworkParameters{T}}) where {T} = ...

That is the reason it is a type parameter and not only a function. GeometricOptimizers takes the element type from the type of the solution it is handed, and a parameter set could not join its OptimizerSolution{T} union while it carried no such parameter.

It is derived by parameter_eltype at construction and never chosen; naming it, as NetworkParameters{T}(params), asserts it and raises if the leaves say otherwise. Note that it is a promotion, not a guarantee of uniformity — a mixed set reports the type its leaves promote to while each leaf keeps its own:

using NeuralNetworkParameters

ps = NetworkParameters((L1 = (W = Float32[1 2], b = [3.0]),))
(ps isa NetworkParameters{Float64}, eltype(ps.L1.W))

# output

(true, Float32)

A set with nothing to promote — an empty one, or a gradient tree that is all gaps — reports Union{}.

Implementation

getproperty is overloaded to reach into the wrapped NamedTuple, so the field itself has to be read with getfield — which is what params does.

There is deliberately no Base.eltype: this type forwards the NamedTuple interface, for which eltype means the type of what iteration yields — a layer's NamedTuple — rather than the numeric element type. The numeric one is parameter_eltype.

source

NeuralNetworkParameters.params — Function

params(p::NetworkParameters)

The NamedTuple wrapped by p.

Also the accessor a NeuralNetwork uses for its own parameters in AbstractNeuralNetworks, hence the short name.

source

A whole set of parameters is a NetworkParameters and nothing else. A branch of one — a layer — is the plain NamedTuple it wraps, and that is a different question, answered by isparametertree: it is what the walks recurse into, and its domain also admits a Tuple, which is what freeparameters returns for a multi-block leaf.

There is deliberately no alias unioning the two. One would be a method on Base.NamedTuple wherever it were used — a type nobody owns, colliding with every other NamedTuple alias in the same method table — and it would say the same thing about a whole set and about a branch. Where a function genuinely takes both, it has a method for each.

NetworkParameters{T} wraps a NamedTuple and forwards getproperty, getindex, keys, values, length, iterate and pairs to it, so it reads like the NamedTuple it holds. NamedTuple(ps) unwraps it again, and is the same thing as params; the conversion is defined here rather than downstream because a package that owns neither Base.NamedTuple nor the type cannot define it without committing piracy on both counts.

The wrapper is not decoration. A bare NamedTuple belongs to Base, so any package wanting to give a parameter set its own behaviour — saving it, flattening it, stepping an optimizer over it — must write methods whose every argument type is somebody else's. That is type piracy, and two packages doing it can silently disagree about the same call. Owning the type removes the problem at the root.

The T is the element type the leaves promote to, carried on the type so that a method signature can bind it — f(ps::NetworkParameters{T}) where {T}, or a Union with AbstractVector{T}. A set is built from its keys and values with NetworkParameters(NamedTuple{keys}(vals)), since the braces name the element type rather than the keys, and it is derived rather than chosen: writing NetworkParameters{T}(nt) asserts T and raises if the leaves say otherwise. See parameter_eltype for what the promotion does and does not guarantee.

Note that key order is part of the identity of a parameter set:

using NeuralNetworkParameters

NetworkParameters((a = [1.0], b = [2.0])) == NetworkParameters((b = [2.0], a = [1.0]))

# output

false

which is why the HDF5 writer records it explicitly — see Reading and writing HDF5.

Flat

NeuralNetworkParameters.FlatParameters — Type

FlatParameters(data, layout)
FlatParameters(ps)

A parameter set in its flat form: an AbstractVector of every number in the set, carrying the ParameterLayout that puts it back together.

This is the representation to differentiate with respect to. It behaves as an ordinary vector, so ForwardDiff, a linear solver or an optimizer can work on it directly, while still knowing how to return an answer in the shape of the network. parent hands out the bare Vector for anything that would rather not see a wrapper.

similar keeps the layout, so scratch space derived from a flat parameter set — a gradient, a momentum buffer — stays self-describing.

Examples

using NeuralNetworkParameters

ps = NetworkParameters((L1 = (W = [1.0 2.0], b = [3.0]),))
fp = FlatParameters(ps)
(length(fp), fp[2])

# output

(3, 2.0)

Entries can be read back by layer, and the whole set converted:

using NeuralNetworkParameters

ps = NetworkParameters((L1 = (W = [1.0 2.0], b = [3.0]),))
fp = FlatParameters(ps)
(fp.L1.b, NetworkParameters(fp) == ps)

# output

([3.0], true)

source

NeuralNetworkParameters.flatlayout — Function

flatlayout(fp)

The ParameterLayout carried by fp.

source

FlatParameters is an AbstractVector, so it works with anything that takes one; parent gives the bare Vector for code that would rather not see a wrapper.

similar deliberately keeps the layout, so scratch space derived from a flat parameter set — a gradient, a momentum buffer — stays self-describing:

using NeuralNetworkParameters

ps = NetworkParameters((L1 = (W = [1.0 2.0], b = [3.0]),))
fp = FlatParameters(ps)
g = similar(fp)
flatlayout(g) == flatlayout(fp)

# output

true

Individual layers can be read back off the flat form, which is mostly useful interactively:

using NeuralNetworkParameters

fp = FlatParameters(NetworkParameters((L1 = (W = [1.0 2.0], b = [3.0]),)))
fp.L1.b

# output

1-element Vector{Float64}:
 3.0