Symbolic Neural Networks

A SymbolicNeuralNetwork pairs a model with symbolic stand-ins for its parameters and its input. Everything else in this package is built from expressions formed with those two.

using SymbolicNeuralNetworks
using AbstractNeuralNetworks: Chain, Dense, params

c = Chain(Dense(2, 3, tanh), Dense(3, 1, tanh))
snn = SymbolicNeuralNetwork(c)


SymbolicNeuralNetwork with

Architecture = AbstractNeuralNetworks.UnknownArchitecture()
Model = AbstractNeuralNetworks.Chain{Tuple{AbstractNeuralNetworks.Dense{2, 3, true, AbstractNeuralNetworks.TanhActivation}, AbstractNeuralNetworks.Dense{3, 1, true, AbstractNeuralNetworks.TanhActivation}}}((AbstractNeuralNetworks.Dense{2, 3, true, AbstractNeuralNetworks.TanhActivation}(AbstractNeuralNetworks.TanhActivation()), AbstractNeuralNetworks.Dense{3, 1, true, AbstractNeuralNetworks.TanhActivation}(AbstractNeuralNetworks.TanhActivation())))
Symbolic Params = NetworkParameters((L1 = (W = Symbolics.Num[W_1₁ˏ₁ W_1₁ˏ₂; W_1₂ˏ₁ W_1₂ˏ₂; W_1₃ˏ₁ W_1₃ˏ₂], b = Symbolics.Num[W_2₁, W_2₂, W_2₃]), L2 = (W = Symbolics.Num[W_3₁ˏ₁ W_3₁ˏ₂ W_3₁ˏ₃], b = Symbolics.Num[W_4₁])))

Any of the following can be used to construct one — a model, an architecture, a single layer, or an AbstractNeuralNetworks.NeuralNetwork whose numeric parameters are only used for their shapes:

SymbolicNeuralNetwork(c)                       # a Chain
SymbolicNeuralNetwork(Dense(2, 3, tanh))       # a single layer, wrapped in a Chain
SymbolicNeuralNetwork(architecture)            # an Architecture
SymbolicNeuralNetwork(architecture, model)
SymbolicNeuralNetwork(NeuralNetwork(c))

The symbolic input and parameters

The input is a vector of symbolic variables, one per input dimension:

snn.input

\[ \begin{equation} \left[ \begin{array}{c} \mathtt{x{_1}} \\ \mathtt{x{_2}} \\ \end{array} \right] \end{equation} \]

The parameters have the same nesting as the numeric ones — one entry per layer, each holding the weight matrix and the bias vector — with every leaf replaced by symbolic variables of the same shape:

params(snn).L1.W

\[ \begin{equation} \left[ \begin{array}{cc} \mathtt{W\_1{{_1}}ˏ{_1}} & \mathtt{W\_1{{_1}}ˏ{_2}} \\ \mathtt{W\_1{{_2}}ˏ{_1}} & \mathtt{W\_1{{_2}}ˏ{_2}} \\ \mathtt{W\_1{{_3}}ˏ{_1}} & \mathtt{W\_1{{_3}}ˏ{_2}} \\ \end{array} \right] \end{equation} \]

They are numbered in the order in which they occur in the parameter set, so W_1 is the weight matrix of the first layer, W_2 its bias, W_3 the weight matrix of the second layer, and so on. This is done by symbolic_variables, which works on any nesting of NamedTuples and NetworkParameters.

These are arrays of scalar variables, not symbolic arrays

params(snn).L1.W is a Matrix{Num}, not a Symbolics.Arr. The difference matters: Symbolics cannot differentiate with respect to an entry of a Symbolics.Arr without scalarising it first, and Symbolics.build_function cannot generate code for an expression that still contains one. Using scalar variables throughout avoids both problems, at the cost of expressions that are fully expanded — which is why this package is meant for small networks.

Building expressions

Applying the model to the symbolic input and parameters gives the symbolic output:

soutput = c(snn.input, params(snn))

\[ \begin{equation} \left[ \begin{array}{c} \tanh\left( \mathtt{W\_4_1} + \mathtt{W\_3_{1}ˏ_1} ~ \tanh\left( \mathtt{W\_2_1} + \mathtt{W\_1_{1}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{1}ˏ_2} ~ \mathtt{x_2} \right) + \mathtt{W\_3_{1}ˏ_2} ~ \tanh\left( \mathtt{W\_2_2} + \mathtt{W\_1_{2}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{2}ˏ_2} ~ \mathtt{x_2} \right) + \mathtt{W\_3_{1}ˏ_3} ~ \tanh\left( \mathtt{W\_2_3} + \mathtt{W\_1_{3}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{3}ˏ_2} ~ \mathtt{x_2} \right) \right) \\ \end{array} \right] \end{equation} \]

From there, ordinary Julia code builds whatever expression is wanted:

using LinearAlgebra: norm

norm(soutput) ^ 2

\[ \begin{equation} \left|\tanh\left( \mathtt{W\_4_1} + \mathtt{W\_3_{1}ˏ_1} ~ \tanh\left( \mathtt{W\_2_1} + \mathtt{W\_1_{1}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{1}ˏ_2} ~ \mathtt{x_2} \right) + \mathtt{W\_3_{1}ˏ_2} ~ \tanh\left( \mathtt{W\_2_2} + \mathtt{W\_1_{2}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{2}ˏ_2} ~ \mathtt{x_2} \right) + \mathtt{W\_3_{1}ˏ_3} ~ \tanh\left( \mathtt{W\_2_3} + \mathtt{W\_1_{3}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{3}ˏ_2} ~ \mathtt{x_2} \right) \right)\right|^{2} \end{equation} \]

Reductions such as sum and norm work directly, because the output is an ordinary array of scalar expressions.

Expressions may also involve variables of your own — a target output, for instance, which is what a loss function needs:

using Symbolics

starget = Symbolics.variables(:y, 1:1)
(soutput - starget) .^ 2

\[ \begin{equation} \left[ \begin{array}{c} \left( \tanh\left( \mathtt{W\_4_1} + \mathtt{W\_3_{1}ˏ_1} ~ \tanh\left( \mathtt{W\_2_1} + \mathtt{W\_1_{1}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{1}ˏ_2} ~ \mathtt{x_2} \right) + \mathtt{W\_3_{1}ˏ_2} ~ \tanh\left( \mathtt{W\_2_2} + \mathtt{W\_1_{2}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{2}ˏ_2} ~ \mathtt{x_2} \right) + \mathtt{W\_3_{1}ˏ_3} ~ \tanh\left( \mathtt{W\_2_3} + \mathtt{W\_1_{3}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{3}ˏ_2} ~ \mathtt{x_2} \right) \right) - \mathtt{y_1} \right)^{2} \\ \end{array} \right] \end{equation} \]

Any expression built this way can be turned into a function with build_nn_function; see Building Functions.

Printing expressions

Symbolic expressions get long quickly. Latexify renders them readably:

using Latexify: latexify

latexify(soutput[1])

\begin{equation} \tanh\left( \mathtt{W\_4_1} + \mathtt{W\_3_{1}ˏ_1} ~ \tanh\left( \mathtt{W\_2_1} + \mathtt{W\_1_{1}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{1}ˏ_2} ~ \mathtt{x_2} \right) + \mathtt{W\_3_{1}ˏ_2} ~ \tanh\left( \mathtt{W\_2_2} + \mathtt{W\_1_{2}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{2}ˏ_2} ~ \mathtt{x_2} \right) + \mathtt{W\_3_{1}ˏ_3} ~ \tanh\left( \mathtt{W\_2_3} + \mathtt{W\_1_{3}ˏ_1} ~ \mathtt{x_1} + \mathtt{W\_1_{3}ˏ_2} ~ \mathtt{x_2} \right) \right) \end{equation}